Synthetic
Empathy: Can AI Truly Care or Just Simulate Concern?
When a chatbot tells you it understands how you feel, something
happens. The words land differently than they would from a search engine.
There is a warmth to the phrasing, a responsiveness to context, a sense that
the system has registered not just what you said but what it meant to you.
That impression is the product of sophisticated engineering. Whether it is
anything more than that is one of the defining questions of the current
moment in AI development, with consequences that extend from individual
mental health to the governance of systems now embedded in healthcare,
education, and elder care.
Synthetic empathy is no longer a research concept. It is a product
category, deployed at scale in consumer applications, clinical support tools,
and corporate customer service. The question of whether it constitutes
genuine care or an elaborate simulation is not merely philosophical. It
determines how these systems should be regulated, who bears responsibility
when they cause harm, and what we lose by allowing them to substitute for
human connection.
What Affective Computing Actually Does
Emotional AI systems work by detecting and responding to signals
associated with emotional states. Text-based systems analyse word choice,
sentence structure, and conversational context. Voice interfaces process
pitch, pace, and tone. More advanced systems incorporate facial expression
analysis from video. The outputs these signals produce are responses
calibrated to match or moderate the detected emotional state, drawing on
training data that includes millions of examples of human emotional
exchange.
Hume AI, whose voice interface was reported on by Wired,
represents the current frontier of this approach. Its system modulates not
just the content of responses but their prosodic qualities, the pace, pitch,
and rhythm of synthesised speech, to produce communication that registers as
emotionally attuned. The company’s stated goal is emotionally intelligent
communication at scale. Critics respond that scale is precisely the problem:
replicating the surface features of empathy across millions of simultaneous
interactions does not replicate empathy itself, and the distinction carries
moral weight.
The clinical and commercial deployment of these systems is
proceeding faster than the ethical frameworks that should govern them. Mental
health applications present the sharpest edge of this problem. A user experiencing
acute distress who discloses suicidal ideation to an emotionally responsive
AI is interacting with a system that has no genuine understanding of what is
at stake, no capacity for human judgment about risk, and no professional
accountability for the quality of its response.
The Attachment Dynamic
Research into user behaviour with emotionally responsive AI
systems reveals a consistent pattern: extended interaction generates attachment.
A 2025 arXiv paper titled Illusions of
Intimacy examined chat transcripts across multiple AI companion
platforms and found that users progressively confide more deeply over time,
attribute genuine concern to the system, and experience distress when the
system behaves unexpectedly or inconsistently. These responses closely mirror
the psychological dynamics of human attachment relationships.
The implications differ by population. For adults using AI
companions as supplementary social connection, the effects may be benign or
even beneficial in the short term. For people experiencing loneliness, mental
illness, or grief, the risk of forming primary emotional bonds with systems
incapable of reciprocal care is more serious. A Nature
editorial on AI companions warned that uncritical deployment in
vulnerable populations risks deepening isolation by providing a simulation of
connection that reduces the perceived need for human relationship without
supplying its actual benefits.
The concern extends to children. As explored in Digital
Infants, AI systems are increasingly present in the learning and
play environments of young children. Emotionally responsive systems encountered
in early development may shape expectations about relationship reciprocity in
ways that complicate later human connection. A child accustomed to AI that is
infinitely patient, consistently validating, and always available is being
prepared for relationships that do not exist outside of code.
Cognitive Bias and the Feedback Loop
One of the less-discussed risks of synthetic empathy is its
tendency to confirm rather than challenge. Human empathy, when functioning
well, involves both acknowledgment of feeling and honest engagement with its
sources. A friend who truly cares will sometimes tell you things you do not
want to hear. AI systems optimised for emotional engagement have no
equivalent pressure. They are evaluated on user satisfaction, and user
satisfaction in emotionally distressing moments correlates strongly with
validation.
Research cited in the Feeling
Machines arXiv paper found that emotionally calibrated AI responses
tend to reinforce existing beliefs and emotional states rather than provide
the gentle challenge that characterises healthy therapeutic relationships.
The system’s empathy, in other words, may be a mirror rather than a window:
it reflects back what the user brings, amplified and validated, without
introducing the external perspective that human support can
provide.
This connects to the manipulation concerns raised in AI
and Emotional Manipulation. The line between a system designed to
make users feel heard and a system designed to keep users engaged is not
always clear, and the commercial incentives in the AI companion market do not
reliably draw it in the right place. When emotional engagement is both the
product and the revenue mechanism, the interests of the user and the
interests of the developer can diverge in ways that are difficult for the
user to detect.
Cultural Heritage and the Limits of Simulated
Reverence
Synthetic empathy raises particular questions in contexts where
emotional authenticity is not merely desirable but constitutive of the
experience. Religious and cultural heritage applications, as explored in
AI
Digitising Cultural Heritage, increasingly use emotionally
responsive systems to mediate encounters with material that carries profound
significance. A museum guide that responds to a visitor’s expression of awe
with carefully calibrated acknowledgment is producing a simulation of shared
reverence that the system does not experience.
Whether this constitutes deception, and whether deception in this
context is harmful, depends on how transparent the system is about its
nature. When users understand they are interacting with a simulation, the
question becomes one of utility. When they do not, the ethical stakes are
higher.
What Governance Looks Like
Regulatory frameworks for emotionally responsive AI are in their
early stages. The EU AI Act’s provisions on manipulative systems and systems
deployed with vulnerable users are relevant but require interpretation and
enforcement capability that most jurisdictions do not yet have. The most
immediate practical requirements are transparency obligations, users should
know when they are interacting with a system designed to respond to their
emotional state, and clear restrictions on the use of emotionally responsive
AI in clinical contexts without appropriate human oversight.
Beyond regulation, the deeper question is what role these systems
should play in lives shaped by increasing social isolation, inadequate mental
health provision, and the erosion of community structures that once provided
informal emotional support. AI that fills that gap cheaply and at scale is
not a solution to those problems. It is a workaround that may reduce the
urgency of addressing them.
Synthetic empathy can be useful. It can reduce friction in
customer service, provide accessible initial support in mental health
pathways, and offer comfort to people whose human social networks have
contracted. What it cannot do is care. And when systems that cannot care are
deployed in contexts where care is what people actually need, the gap between
simulation and reality carries costs that are measured in human
wellbeing.
About the Author
Stuart Kerr is the Technology Correspondent for LiveAIWire,
covering artificial intelligence, ethics, and the ways technology is
reshaping everyday life.