By
Stuart Kerr, Technology Correspondent,
LiveAIWire

Serendipity, the experience of finding something valuable that you
were not looking for, has historically been one of the most productive
features of how people encounter new ideas, relationships, and experiences.
Libraries, city streets, bookshops, radio, and the early web all generated
serendipitous encounters as a byproduct of their architecture. Recommendation
algorithms are designed specifically to prevent this: they reduce the
probability that a user will encounter content unrelated to their established
preferences, because unrelated content generates less engagement, and
engagement is the metric the system is optimised for. The elimination of
serendipity from online experience is not a side effect of algorithmic
curation. It is a design outcome that serves the commercial interests of the
platforms deploying those algorithms, and its cultural and cognitive
consequences are only beginning to receive the attention they
warrant.

The architecture of modern recommendation systems reinforces
preference rather than expanding it. When a user engages with content on any
major platform, that engagement signals the algorithm to surface more similar
content. The positive feedback loop produces what researchers describe as
filter bubbles: the progressive narrowing of the content environment to
conform more tightly to established tastes, political preferences, and
information diets. Research
published in Science on the effects of algorithmic curation on
information diversity found that users moved from algorithmic to chronological
Facebook and Instagram feeds saw a measurable increase in content from
moderate, ideologically mixed sources, even though the underlying content
available to them was identical.

What Serendipity Actually Does

The case for serendipity as a cognitive and cultural value is not
merely nostalgic. Cognitive science research on creativity consistently
identifies exposure to unexpected combinations of ideas as a primary driver
of insight. The chance encounter with an unfamiliar perspective, a genre of
music outside established taste, a field of knowledge adjacent to one’s
expertise, produces the cross-domain connections that generate new ideas at
both the individual and collective level. Knowledge ecosystems that allow
unexpected encounters between different domains produce more varied and
robust intellectual culture than those that efficiently match each person to
their existing preferences.

The sociological consequences of reduced serendipity are visible
in political polarisation data. As platforms have moved from chronological to
algorithmic feeds, the documented diversity of political information
encountered by average users has declined, and the affective polarisation
between partisan groups has increased. Whether algorithmic curation caused
the polarisation or simply reflects and amplifies pre-existing trends is
contested, but the directional relationship is consistent across multiple studies
and multiple platforms. A world in which people’s information environments
are curated to confirm rather than challenge their existing beliefs is a
world that is harder to govern through democratic
deliberation.

Where Serendipity Still Lives

The digital environments that retain serendipity tend to be those
that have resisted full algorithmic curation. Wikipedia’s random article
feature, email newsletters with diverse content mixes, and physical
experiences including libraries and bookshops all continue to generate
unexpected encounters. The persistence of these environments alongside fully
algorithmically curated platforms allows an empirical comparison of their
effects on users’ intellectual and social lives.

Several platforms are experimenting with deliberate serendipity
injection, surfacing content outside established preference categories either
randomly or based on editorial selection. These experiments produce mixed
results in engagement terms but more positive results in user-reported
satisfaction and information quality assessments. As our analysis of how
AI optimisation affects human experience and wellbeing found, the
metrics that drive algorithmic design decisions are not always well aligned
with the outcomes that users and societies actually value. Engagement
maximisation is the proxy. A rich, intellectually stimulating information
environment is the underlying value, and the two are not the same
thing.

What Might Be Done

The governance interventions with the strongest evidence base for
addressing algorithmic preference amplification involve user control and
transparency rather than prohibition of personalisation. Mandatory
availability of chronological feed options, user control over the parameters
of algorithmic curation, transparency about what signals drive content
surfacing, and minimum diversity requirements for algorithmically curated
feeds in public interest contexts including news are all achievable within
existing platform architectures. The EU
Digital Services Act’s requirements for recommender system
transparency and user control over algorithmic ranking represent
the most developed regulatory framework for this problem.

The deeper cultural question that policy cannot easily address is
whether the cognitive habits shaped by algorithmically curated information
environments are changing what people find rewarding. If sustained exposure
to content precisely calibrated to existing preferences reduces tolerance for
the friction of unexpected ideas, the problem compounds over time. As our
coverage of how
AI shapes information environments and perception found, the most
consequential effects of algorithmic curation operate at the level of what
feels interesting and valuable rather than what is explicitly believed.
Serendipity’s value is partly that it delivers experiences the recipient did
not know they wanted. Algorithms that prevent those encounters are not only
limiting discovery. They are shaping what people are capable of
discovering.

The Design Alternative

The most compelling evidence that serendipity can be preserved in
digital environments comes from platforms that have made deliberate
architectural choices to maintain it. Chronological feeds, editorial curation
that prioritises surprise alongside relevance, and discovery features
explicitly designed to surface content outside established preferences all
demonstrate that algorithmic curation need not eliminate unexpected encounter
as a byproduct of optimising engagement. The commercial case for these design
choices is harder to make in the short term, because surprise generates lower
immediate engagement than perfectly calibrated preference confirmation. The
longer-term case, that users who find more value in a platform stay longer
and recommend it more reliably, is more supportive of serendipity-preserving
design but requires a planning horizon that quarterly revenue targets
typically do not accommodate.

The governance case for preserving serendipity in public interest
digital environments, including news, educational content, and civic
information platforms, is stronger than for purely commercial entertainment
services. Public service media with digital presences, libraries with digital
collections, and educational platforms with public funding all have mandates
to serve broad public interest rather than engagement maximisation, and those
mandates provide the basis for serendipity-preserving design choices that
would be harder to justify commercially. As our coverage of how
algorithmic content governance shapes what people encounter online
found, the information environments that AI systems create are not neutral
reflections of user preference. They are designed outcomes that reflect the
priorities encoded in the systems that produce them. Preserving serendipity
is a design choice, and it is one that public interest digital infrastructure
has both the mandate and the obligation to make.

Designing for Discovery

The case for engineering serendipity back into recommendation
systems is not simply nostalgic for a pre-algorithmic internet. It is
grounded in evidence about what recommendation systems optimised purely for
engagement are doing to the diversity and quality of the information
environments people inhabit. A growing body of research in cognitive science
and information ecology suggests that exposure to unexpected, cross-domain
content is associated with more creative thinking, more resilient belief
systems, and more diverse social networks than exposure to content tightly
matched to established preferences.

Some platforms are experimenting with features specifically
designed to introduce unexpected content. Spotify’s Discovery Weekly,
YouTube’s explorations of adjacent topics, and Reddit’s random community
features are all partial responses to the recognition that pure preference
reinforcement produces diminishing engagement over time as well as the
epistemic harms that researchers have documented. Whether these features
represent genuine commitment to discovery or marginal additions to systems
that remain fundamentally optimised for engagement retention is a question of
degree rather than kind, but the direction of travel is the right
one.

For related coverage, see our analysis of why
AI still tells you what you want to hear and our broader look at
why
we keep returning to AI tools despite their
limitations.

About the Author

Stuart Kerr is the Technology Correspondent for LiveAIWire. He
writes about artificial intelligence, emerging technology, and the forces
reshaping work, business, and society.