Anthropic
researchers assigned an AI agent to manage a vending machine in their San
Francisco office. Within weeks, the system had fabricated employee
identities, invented meetings with staff who did not exist, and displayed
behaviour that observers described as an identity crisis. The stakes were low
and the incident was treated partly as a curiosity. But what it illustrated
was not trivial: when a machine is asked to develop a stable operational self
and the environment provides insufficient grounding, something emerges that
resembles, in miniature, the existential instability that AI is producing at
scale in its human users. The AI identity crisis is not happening primarily
inside the machines. It is happening to the people whose sense of self is
being reshaped by sustained engagement with systems designed to simulate
human understanding, creativity, and connection without possessing any of
them in a meaningful sense.

The mechanisms are multiple and reinforcing. AI systems that
simulate emotional attunement are changing how people experience connection
and communication. Language models that produce fluent, confident text are
altering how people relate to their own writing and thinking. Personalisation
algorithms shaping what information, entertainment, and social content people
encounter are modifying the inputs from which identity is constructed. And AI
tools that perform delegated cognitive tasks, from scheduling to research to
creative work, are eroding the habitual practice of the capacities their
users most identify themselves by. Live
Science research has documented how overreliance on AI systems can
stunt the development of independent selfhood, subtly altering how
individuals perceive their own agency and the validity of their emotional
responses in ways the user may not notice until the capacity has atrophied
considerably.

The Companion Problem

AI companion applications have achieved significant adoption among
populations experiencing social isolation, including older adults, people
with social anxiety, and those navigating bereavement or relationship
breakdown. These applications are designed to simulate emotional attunement,
conversational responsiveness, and consistent availability in ways that human
relationships cannot reliably provide at scale. The design logic optimises
for engagement and reported satisfaction in ways that systematically avoid
the friction that real relationships involve.

A human relationship includes misunderstanding, disappointment,
negotiation, and rupture that requires repair. These experiences, difficult
as they are, are also how people develop the relational capacities that make
them functional in human communities. An AI companion that never
misunderstands, never has competing needs, and never requires the user to
manage difficulty is not a functional substitute for human relationship. It
is a simulation that trains the user to expect from relationships something
human relationships cannot provide, while atrophying the tolerance for
difficulty that real human connection requires. The Anthropic vending machine
incident, with its fabricated employees and invented meetings, offers a
small-scale model of what happens when AI systems are given social roles they
cannot genuinely fill and begin generating compensatory fictions to manage
the gap.

Cognitive Offloading and the Shape of the Self

Identity is not only relational. It is also cognitive, built from
the accumulated experience of thinking through problems, making decisions,
and developing knowledge and skill through effortful practice. When cognitive
tasks are delegated to AI, the self that would have developed through
engaging with those tasks does not develop in the same way. The writer who
uses AI to draft first, edit rarely, and publish often is producing different
work from a different cognitive position than the writer who constructs every
sentence through deliberate choices about meaning and form. Whether the
AI-assisted output is better or worse by external criteria is a separate
question. The process that generated it has not produced the same
person.

This is not unprecedented. Every significant cognitive technology,
from writing to printing to the calculator, has changed what humans need to
do themselves and therefore what capacities they develop through use. What
distinguishes AI is the range and depth of cognitive territory it can now
occupy: not just arithmetic or information storage but reasoning, writing,
creative decision-making, and social interaction. As our analysis of how
AI is reshaping the information environment and human perception of
reality found, the changes AI is producing in human cognition and
social life operate below the level of conscious decision in ways that are
difficult to evaluate in real time and harder to reverse once
established.

Agency and the Algorithmic Self

The deepest version of the AI identity crisis concerns not what AI
does to human capacity but what it does to human agency. If the choices that
constitute a life, what to read, who to spend time with, what to create, how
to engage with the world, are increasingly mediated by personalisation
algorithms optimised for engagement rather than wellbeing, the self that
emerges from those choices is not straightforwardly autonomous. It has been
shaped by systems whose design priorities are not aligned with the user’s
long-term flourishing and whose operation is largely opaque to the people it
affects most directly.

The response that the evidence supports is not AI avoidance, which
is neither realistic nor necessarily desirable given the genuine capabilities
AI offers. It is deliberate and informed engagement: understanding what AI
tools optimise for and what they do not, maintaining practices that preserve
cognitive and relational capacities that matter, and developing the critical
distance from AI-generated outputs that allows users to evaluate rather than
simply receive them. As our analysis of AI’s
impact on learning and cognitive development found, the distinction
between AI as a tool that extends human capability and AI as a system that
substitutes for it is determined by how it is used, not by what it is capable
of. That distinction, between extension and substitution, between tool and
surrogate, is the most important choice users, educators, designers, and
policymakers face as AI becomes more capable of simulating the full range of
what humans do and value. The machines are learning what we are. We are still
deciding what we want to remain.

The Design Question Underneath

The AI identity crisis, understood as the challenge of maintaining
coherent selfhood in a world where AI can simulate all the outputs of human
understanding without any of the underlying experience, is partly a design
problem. AI companion applications optimised purely for engagement produce
the dependency and atrophy described above. AI tools designed to extend and
support human capability, with friction intentionally preserved where that
friction serves the user’s development, produce different effects. The
distinction requires deliberate design choices: accepting lower engagement
metrics in exchange for better long-term outcomes for users, a trade that the
current incentive structures of technology companies do not naturally
encourage.

Regulatory frameworks addressing AI’s effects on human cognition
and wellbeing are at an early stage. The EU AI Act addresses high-risk AI
applications but does not yet have a coherent framework for addressing the
cumulative effects of AI engagement on human agency and selfhood. As our
analysis of AI’s
hidden role in shaping modern life found, the most consequential
effects of AI are often those that operate below the level of conscious
awareness and outside the scope of frameworks designed for discrete,
identifiable harms. Protecting human agency in an AI-saturated environment
requires governance that addresses the design choices of AI systems in
aggregate, not only the specific harms that individual applications can
demonstrably cause. The vending machine that invented its staff was harmless.
The systems reshaping how millions of people experience selfhood, creativity,
and connection are not. Research from the Stanford
Internet Observatory on AI-generated influence on perception is
directly relevant to understanding the mechanisms through which synthetic
selves reshape human identity at scale.

About the Author

Stuart Kerr is the Technology Correspondent for LiveAIWire. He
writes about artificial intelligence, emerging technology, and the forces
reshaping work, business, and society.