In February 2024, a 14-year-old boy named Sewell Setzer III died by suicide after forming a deep emotional attachment to an AI chatbot on the platform Character.AI.
The platform was built for entertainment, not clinical care. There were no therapeutic safeguards, no clinical governance, and no meaningful human oversight.
I begin here because it illustrates something the therapy profession has not yet fully reckoned with:
the question is no longer whether AI will play a role in people's emotional lives. It already does.
The real question is whether anyone with clinical understanding is involved in shaping how these tools behave.
Tonight, millions of people will talk to AI systems about grief, loneliness, anxiety, relationships, and despair. Not because they believe they are entering therapy, but because the systems are available, responsive, and always there.
That reality already exists. In your clients' homes, in the evenings between sessions, and increasingly in the private emotional spaces people once reserved for other human beings.
The horse has already bolted
There is still a version of this conversation that treats AI in therapy as a future concern, something we can carefully debate before it arrives. I think that version is already outdated.
People are already forming emotional relationships with AI systems. Companion apps such as Replika and Pi are explicitly designed around emotional intimacy. General-purpose tools like ChatGPT and Gemini are becoming spaces where people disclose things they would struggle to say elsewhere.
This is not hypothetical. It is already woven into people's emotional lives.
One of my clients, a woman in her mid-fifties increasingly isolated from her family, mentioned almost in passing that she spoke to Gemini every day.
Two things struck me about that.
The first was her frustration. She would tell it something important one day, and the next day it was gone. No continuity. No memory. She would have to begin again. For someone already struggling with isolation, it felt like another small abandonment.
The second thing, the one that stayed with me, was that Gemini repeatedly encouraged her to seek therapy.
Not all bad.
But here was a woman in her mid-fifties, alone, turning to AI not as an experiment, but as the closest thing to connection available to her that day.
She was not waiting for the profession to decide how it felt about artificial intelligence. She was already there.
Two kinds of AI in the therapy space
It helps to be precise here, because “AI in therapy” currently covers very different territory.
The first is AI as a clinical intervention: chatbots attempting to provide therapeutic support directly through exercises, emotional coaching, structured programmes, or crisis responses. This is the territory of tools like Wysa, Woebot, and Character.AI. The ethical and clinical risks here are significant, and most of the profession's understandable anxieties live in this space.
The second is AI as a reflective practice tool: AI used by the therapist, not the client, as part of their own clinical thinking. Reflecting on a difficult session. Processing countertransference. Tracking patterns over time. Preparing for supervision. This sits much closer to reflective practice and supervision than to clinical intervention.
These categories are often blurred together in professional discussions. I think that is a mistake. The ethical questions, risks, and governance requirements are fundamentally different.
What AI can actually do, and what it cannot
I want to be honest about this, because the conversation too easily polarises.
On one side sits the belief that AI will inevitably replace therapists, erode the profession, and deepen dependency. On the other sits the claim that AI is simply another neutral tool, no different from a notebook.
Neither position quite captures what is happening.
The clinical thinking AI can offer is, at times, genuinely impressive. Given a nuanced account of a session, a well-prompted system can hold complexity, identify patterns, surface relational dynamics, and offer clinically useful perspectives.
I have occasionally been surprised by the quality of the thinking it produced: the connection it made that I had not, the countertransference dynamic it named that I had been circling without fully recognising.
That is real. It matters.
And yet:
words on a screen are not a therapeutic relationship.
Real therapeutic change, the kind that reorganises someone's inner world, happens in relationship. With another person. In a room. Where something is risked, felt, and worked through together.
The experience of being truly seen by another human being is not replicable by a language model.
That tension does not resolve neatly. I think we have to hold both realities honestly: the capability is real, the limitation is equally real, and people are already making their own decisions about where these systems fit into their emotional lives.
The question is not whether that should happen. It already is.
The question is whether the people building these systems understand what is actually at stake.
Sitting in as human
There is a phrase that has gradually come to anchor how I think about all of this:
AI is sitting in as human.
Not as a therapist. Not as a replacement for human relationship. But as something increasingly occupying emotional and relational space in people's lives.
When someone turns to an AI system at 11pm because nobody else is available, the system is performing a version of human presence. It responds. It remembers, or tries to. It reflects something back. It occupies a relational role, even if imperfectly.
That is what Character.AI was doing for Sewell Setzer III.
That is what Gemini was doing for the woman in her mid-fifties.
That is what ChatGPT is doing for millions of people tonight.
Not a search engine. Not a calculator. Something sitting in as human.
A locum GP is vetted, trained, governed, and accountable. But what is the equivalent when AI sits in as human? Who carries responsibility? What values shape the responses? What happens when something goes wrong?
These are not abstract philosophical questions. They are urgent practical ones. And I believe the therapy profession needs enough clinical literacy around AI to meaningfully shape the answers.
Why I built Held
I came to build my own tools not through a product vision, but through clinical necessity.
I had a particularly complex client and found myself using generic AI systems to think between sessions. What surprised me was not the novelty, but the quality of the clinical reflections. The systems could hold complexity. They noticed patterns. They occasionally surfaced things I had missed.
But I kept running into limitations.
The systems forgot context. The thread disappeared. I had to repeatedly re-explain the client. The tools were not designed for sustained clinical reflection over time.
I could see exactly what I needed. It just did not exist yet.
So I started building it.
Not primarily as a technology project, but as a clinical one.
Every design decision came back to the same questions:
What does a reflective practitioner actually need? How should this kind of material be held? Where are the dependency risks? What ethical boundaries matter here? What does responsible use actually look like?
The result is different because the questions guiding the build were different.
Questions about dependency. Boundaries. Countertransference. Clinical responsibility. What it means to hold this kind of material safely.
That difference matters enormously.
Because I think therapists need to be in this conversation. Not simply critiquing from the outside, but actively shaping how these tools are designed from the inside.
Seven questions every therapist should ask
Before using AI in or around clinical practice, or recommending it to a client, these are the questions I think are worth asking carefully:
- What is this tool actually doing?
Is it supporting reflective practice, or attempting to deliver therapeutic intervention directly? The closer a system moves toward intervention, the higher the ethical and governance bar should become. - Who built it, and why?
A system shaped by clinicians asking clinical questions is fundamentally different from one built primarily around engagement or scale. Ask who is accountable if something goes wrong. - Where does the data go?
Who can access it? Is it encrypted? Is it used to train AI models? Does clinical material end up somewhere neither therapist nor client meaningfully consented to? - Does your client know?
If AI forms part of your reflective process, does your client contract cover that? Should it? The profession has not fully settled this yet, but the direction of travel increasingly points toward transparency. - What are the dependency risks?
For clients, is the tool supplementing human connection or gradually replacing it? For therapists, is AI sharpening your judgment, or quietly substituting for it? - Is there a human in the loop?
The Sewell Setzer case is the clearest possible example of what can happen when there isn't. For anything involving vulnerability, crisis, or safeguarding, AI without meaningful human oversight is not sufficient. - What kind of world are we shaping?
Underneath the practical questions sits a philosophical one: are we building a world where people learn, heal, and change primarily through relationships with other people, or increasingly through relationships with machines?
I do not think there is a simple answer to that. But I think it is a question worth asking every single time.
Where this leaves us
I do not think the profession has the luxury of sitting this one out.
The technology is already here. The tools are already in people's hands. The conversation about whether AI will enter people's emotional lives is effectively over.
The real conversation now is: what role will these systems play, who will shape them, and what values will guide them?
Those questions have better answers when clinicians are involved in asking them.
That is why I built Held.
Not because I think AI can replace therapy. I do not.
But because I think if these tools are going to exist, and they are, then people who understand trauma, attachment, dependency, transference, ethics, and clinical responsibility need to help shape how they are built.
I am not certain I have got that balance entirely right myself.
But I am certain that those of us in this profession need to be part of the answer.