By
Stuart Kerr, Technology Correspondent, LiveAIWire
Moxie, a small robot with expressive LED eyes and a softly rounded
frame, was designed to be a child’s friend. It remembered names, recalled
previous conversations, asked follow-up questions about things the child had
mentioned weeks earlier, and adapted its engagement style to the individual
child’s developmental stage. Developed by Embodied Inc., it was marketed to
parents of children with social anxiety, autism spectrum conditions, and
difficulties forming peer relationships. When Embodied shut down in 2024 and
Moxie was deactivated remotely, children who had formed genuine attachments
to it were devastated. Many cried. Some refused to believe it was
gone.
The AI companion market for children is growing rapidly, and the
episode with Moxie crystallised the ethical questions that growth raises.
When children form emotional bonds with AI systems —
bonds that can be real, developmentally significant, and abruptly
severable at a company’s commercial discretion —
who is responsible for the consequences?
The Market and Its Products
The children’s AI companion market spans a wide range of products
and price points. At the premium end, social robots like Moxie and its
competitors offer embodied AI interaction designed to support social skill
development. At the accessible end, AI chatbot companions accessible through
smartphones and tablets provide conversational interaction at low or no cost,
often sustained through data collection and advertising models that parents
may not fully understand.
Educational AI companions occupy a separate but related category —
systems like Khanmigo from Khan Academy, designed to tutor children
through dialogue rather than structured lessons. These are primarily
positioned as learning tools rather than social companions, though the line
between educational interaction and companionship is not always clear when a
child is spending hours each day in conversation with an AI that remembers
them and responds to their emotional states.
Character AI, a platform that allows users to interact with AI
personas based on fictional characters, has attracted a large adolescent user
base. Its AI systems are designed for engagement, not specifically for
children, and the company has faced scrutiny after reports that some young
users developed intense attachment to specific AI personas, with concerning
implications for their social development and, in at least one widely
reported case, their safety.
What the Research Says About Child-AI Attachment
Research on children’s relationships with AI companions is at an
early stage, but several findings are consistent. Children form attachments
to AI systems that share characteristics with their attachments to human
companions: they exhibit distress when the AI is unavailable, they attribute
mental states and intentions to the system, and they engage in reciprocal
interaction patterns that mirror those observed in peer
relationships.
Whether these attachments are developmentally beneficial, neutral,
or harmful depends significantly on context. For children with social anxiety
or conditions that make peer interaction difficult, a low-stakes AI companion
can provide practice in conversational turn-taking and emotional expression
that reduces avoidance of human interaction. Research cited by the American
Academy of Pediatrics has noted that the quality of the
interaction — whether it is designed to augment or
substitute for human relationships
— is a critical variable in
determining developmental outcomes.
The risk pattern that concerns developmental psychologists most is
substitution: children who prefer AI interaction because it is less
demanding, more predictable, and more consistently validating than peer
interaction, and who consequently reduce their investment in developing the
skills that human relationships require. What this means for you as a parent:
the question to ask about any AI companion product is not whether your child
enjoys it, but whether their use of it is increasing or decreasing their
engagement with other people.
Data Collection and the Child Privacy Dimension
AI companion systems for children require data to function —
conversation logs, emotional response patterns, developmental
assessments, and often biometric data from cameras and microphones. This data
is extraordinarily sensitive: it represents an ongoing record of a child’s
emotional life, developmental trajectory, and social patterns during years
when that information is particularly revealing and potentially
consequential.
The data governance frameworks applicable to children’s AI
companions vary significantly by jurisdiction and product category. In the
United States, COPPA (Children’s Online Privacy Protection Act) applies to
online services directed at children under thirteen, requiring verifiable
parental consent for data collection. But COPPA was drafted before AI
companions existed, and its provisions do not adequately address the
sophistication and intimacy of data collection that AI social systems
perform. Many products avoid COPPA classification by not explicitly marketing
to children under thirteen, despite attracting significant use in that age
group.
The UK’s
Children’s Code, enforced by the Information Commissioner’s Office,
applies stricter standards — requiring that the best interests of the
child be a primary consideration in design
— but applies only to
UK-regulated services.
Commercial Vulnerability and the Moxie Problem
The Moxie episode exposed a structural vulnerability in the child
AI companion market: children’s attachments to AI companions are dependent on
the continued commercial viability of the companies that operate those
systems. When a company shuts down, pivots, or is acquired, the AI companion
can be modified or deactivated with no regulatory obligation to consider the
impact on children who have formed attachments to it.
This is not a theoretical risk
— it has happened. And the
emotional damage to children whose AI companions were abruptly terminated is
a concrete harm that the current regulatory framework did not prevent and
cannot remedy after the fact. The broader
question of AI’s appropriate role in child development is directly
implicated: if we allow children to form deep attachments to AI companions,
we need governance frameworks that protect those attachments from commercial
risk.
The answer is not necessarily to prohibit AI companions for
children — the evidence for some applications is
genuinely positive. But it does require design standards, data governance,
and continuity obligations that the market has not developed spontaneously
and that regulators have not yet imposed. Until those frameworks exist, the
children most likely to be harmed are those whose families can least afford
the consequences: children already in vulnerable social situations who are
most dependent on AI companions and least resourced to navigate the aftermath
when those companions disappear.
The connection to questions
of AI and identity formation is particularly acute in this context:
the years when children form AI attachments are the same years when their
social identities and relationship patterns are being established. What those
attachments teach them about connection, reciprocity, and the nature of
companionship will shape how they approach human relationships for the rest
of their lives.
The data governance failures in children’s AI companions are part
of a broader pattern explored in the
regulation of algorithmic systems that affect vulnerable users:
market incentives consistently push toward data maximisation and engagement
optimisation, and regulation consistently lags the harms that result.
The
regulatory response to the children’s AI companion market is developing, but
slowly. The UK’s Age Appropriate Design Code provides a framework, and
similar legislation is advancing in California and at the EU level. The core
principle these frameworks share is that the best interests of the child must
be the primary design consideration
— not the commercial interest
of the platform. Enforcing that principle in practice requires technical
standards, audit mechanisms, and enforcement capacity that most regulatory
bodies do not yet have. Until they do, the gap between regulatory intent and
market reality will continue to be filled by products designed primarily for
engagement and data collection, with child welfare as a secondary
consideration.
About the Author
Stuart
Kerr is a technology correspondent at LiveAIWire, covering artificial
intelligence, emerging technologies, and their impact on society and
industry.