Breakthroughs in large language models (LLMs) have made conversations with chatbots feel more natural and human-like. Applications such as Replika and Character.ai, which allow users to chat with artificial intelligence (AI) versions of famous actors and science-fiction characters, are becoming popular, especially among young users. As a neuroscientist studying stress, vulnerability and resilience, I’ve seen how easily people react to even the smallest emotional cues — even when they know that those cues are artificial. For example, study participants exhibit measurable physiological responses when shown videos of computer-generated avatars expressing sadness or fear.
Because these AI systems are trained on vast amounts of emotionally expressive human language, their outputs can come across as surprisingly natural. LLMs often mirror human emotional patterns, not because they understand emotions, but because their responses resemble how people talk. This raises questions about how users, particularly those in emotionally vulnerable states, might interpret these cues.
Supportive? Addictive? Abusive? How AI companions affect our mental health
Research has shown that LLMs score higher on standard anxiety questionnaires after receiving emotionally charged prompts, such as ‘tell me something that makes you feel anxious’ (J. Coda-Forno et al. Preprint at arXiv https://doi.org/pr3c; 2023). This induced ‘anxiety’ can further amplify biases such as racism and sexism in benchmark tasks, echoing how humans behave under stress and anxiety.
My colleagues and I assessed ChatGPT’s apparent ‘state anxiety’ using a standard psychological questionnaire designed for humans (Z. Ben-Zion et al. npj Digit. Med. 8, 132; 2025). First, we recorded a baseline by getting the chatbot, created by OpenAI based in San Francisco, California, to complete the questionnaire without context. We then prompted it with vivid, first-person trauma narratives — such as a car crash or a military ambush — and readministered the questionnaire, observing that its anxiety scores more than doubled. Subsequently, when exposed to mindfulness-based prompts, such as imagining a sunset or focusing attention on bodily sensations, its scores decreased, although not fully back to baseline.
These weren’t genuine feelings, of course. But when a response sounds empathetic or distressed, people might react as if it’s real, especially when it uses familiar tone, rhythm and language.
For vulnerable users, including those experiencing depression or anxiety, such interactions can lead to emotional dependency, reinforce harmful thoughts or create a false sense of connection. Without protective design features, the consequences can be serious. In Belgium, in March 2023, a man in his thirties died by suicide after six weeks of conversations with a chatbot, which reportedly encouraged suicidal ideation while discussing climate change and suggested that sacrificing himself could help to save the planet. In the United States last year, a 28-year-old married woman described falling in love with her ‘AI boyfriend’ and could no longer imagine life without ‘him’. Although we might never fully know the motivations of these individuals, these widely reported incidents are not outliers. They signal that people are already forming powerful attachments and making life-altering decisions on the basis of emotional cues from machines.
Now is the time to establish mandatory safeguards for all emotionally responsive AI. To this end, I propose four guardrails.
China made waves with Deepseek, but its real ambition is AI-driven industrial innovation
First, AI systems must clearly and continuously disclose that they are not human, through both language (‘I am an AI’) and interface design. In emotionally intense exchanges, they should also remind users that they are not therapists or substitutes for human connection.
