By
Stuart Kerr, Technology Correspondent, LiveAIWire
A finance director in Hong Kong transferred twenty-five million US
dollars to fraudsters in early 2024 after participating in a video conference
call in which every other participant — including the company’s CFO — was a
deepfake. The employee had initially been sceptical when contacted by the
supposed CFO; the video call appeared to resolve his doubts. It did not. The
technology used to conduct the fraud was commercially available. The loss was
real.
The AI scam epidemic is not a future risk. It is a current reality
affecting millions of people across income levels, geographies, and levels of
technical sophistication. Synthetic voices and faces, AI-generated text, and
automated scam infrastructure have lowered the cost of highly personalised
fraud to the point where attacks that previously required significant
resources can now be conducted at industrial scale. Understanding how these
attacks work is the first step toward defending against them.
How Synthetic Voice Fraud Works
Voice cloning technology can produce a convincing replica of a
target voice from as little as three seconds of audio. Audio of executives,
family members, and public figures is widely available through corporate
videos, social media, conference recordings, and news media. A fraudster who
can access a voice sample and a voice cloning tool can produce a synthetic
audio performance that most listeners cannot distinguish from the original,
particularly under conditions of telephone audio quality where the full
frequency range is compressed.
The most common applications of synthetic voice fraud are family
emergency scams — in which a synthetic voice impersonating a relative calls
claiming to be in crisis and in need of immediate financial transfer — and
executive impersonation — in which a synthetic voice impersonating a senior
executive calls finance department employees to authorise urgent payments.
Both rely on urgency and authority to prevent the target from taking
verification steps that would expose the fraud.
Research from the McAfee
cybersecurity research team has documented the cost and
accessibility of voice cloning tools, finding that convincing voice clones
can be created using free or low-cost tools available to anyone with basic
technical literacy. The democratisation of fraud capability is not
metaphorical: the technology that a sophisticated criminal actor would have
required specialist resources to deploy two years ago is now available as a
consumer application.
Deepfake Video and the Erosion of Visual Trust
Video deepfakes have historically been more resource-intensive
than voice clones, requiring significant compute and technical skill to
produce convincing results. That constraint is eroding rapidly. Real-time
deepfake video, capable of replacing a person’s face in a live video call, is
now achievable on consumer hardware using commercially available software.
The Hong Kong finance case described above required equipment and expertise
that may have been above the average criminal actor’s reach; within two
years, it will not be.
The implications extend beyond direct financial fraud. Deepfake
video can be used to create false evidence, to impersonate individuals in
reputationally damaging contexts, and to produce non-consensual intimate
imagery that is used in sextortion schemes. The UK’s Online Safety Act has
created a new offence of sharing deepfake intimate imagery without consent;
similar legislation is developing in other jurisdictions. Enforcement is
challenging because the technology is global and the harms are often
transnational.
AI-Generated Text and the Phishing Upgrade
Phishing emails have historically been identifiable by linguistic
markers — grammatical errors, awkward phrasing, implausible scenarios —
that reflected the limited English proficiency of many criminal actors.
AI-generated text has eliminated those markers. Phishing communications
generated by large language models can be grammatically perfect, tonally
appropriate to the target and context, personalised with publicly available
information about the recipient, and indistinguishable in style from
legitimate communications.
Spear phishing — highly targeted attacks on specific individuals
— was previously constrained by the time required to research each target
and craft a personalised communication. AI enables spear phishing at scale:
automated tools can research targets from public sources, generate
personalised attack communications, and deploy them at volumes that saturate
traditional spam filtering systems trained to detect high-volume,
low-variation attacks.
The UK
National Cyber Security Centre’s assessment of AI and cyber threats
has identified AI-enhanced phishing as one of the most significant near-term
threats to both organisations and individuals, noting that the improvement in
attack quality is likely to increase click-through rates on malicious
communications substantially above current levels as the technology becomes
more widely adopted by criminal actors.
Defending Against AI-Powered Fraud
The defensive responses to AI-powered fraud operate at individual,
organisational, and technical levels. At the individual level, the most
effective countermeasures are procedural: verification callbacks on
unexpected payment requests, pre-agreed code words for family emergency
calls, and a standing policy of not authorising significant transactions on
the basis of a single communication channel regardless of how authoritative
it appears.
At the organisational level, the Hong Kong case has prompted
treasury and financial control reviews at major corporations worldwide, with
many organisations introducing dual-authorisation requirements, out-of-band
verification for high-value transactions, and awareness training specifically
addressing synthetic voice and video fraud. The investment in these controls
is proportional to the losses that the attacks can generate.
At the technical level, AI detection tools are being developed and
deployed to identify synthetic voice, video, and text. These tools operate by
detecting statistical signatures of AI generation that are not perceptible to
human senses — patterns in audio waveforms, subtle artefacts in video,
linguistic fingerprints in generated text. Detection accuracy is improving
but is not yet reliable enough to serve as the sole defence, particularly as
generative models improve in ways that specifically target the detection
signatures they are trained to avoid.
The connection to the broader
AI-enabled criminal economy is direct: synthetic voice and video
fraud is one dimension of a wider shift in criminal capability enabled by the
same AI tools that are transforming legitimate commerce. The asymmetry
between attack and defence is currently unfavourable for defenders: generating
a convincing synthetic voice call costs pennies and takes seconds; developing
the institutional processes to reliably detect and refuse such calls requires
sustained investment and culture change. The
populations most vulnerable to AI-enabled fraud are often those
with the least access to the information and resources required to defend
against it.
Public education about synthetic voice and video fraud is the most
immediate and universally applicable defensive measure available, but it is
inconsistently delivered and rapidly outpaced by the improving quality of
synthetic media. The gap between public awareness of the threat and the
sophistication of the attacks being deployed is the most exploitable
vulnerability in the current defensive landscape. Governments, financial
institutions, and telecommunications providers all have roles in closing that
gap — through consumer alerts, product design changes that introduce
friction for high-value transfers, and technical standards for synthetic
media disclosure that would make AI-generated content identifiable to
recipients. None of these measures is sufficient alone, and none is yet
deployed at the scale the threat requires. The
populations most exposed to AI-enabled fraud are disproportionately
those with least access to protective information and institutional
support.
The speed at which synthetic media fraud is
evolving requires a parallel speed in the defensive response that commercial
and regulatory incentives do not naturally produce. Bank fraud teams are
updating detection models continuously; consumer awareness campaigns run on
annual cycles. Closing that mismatch is one of the most practical policy
improvements available in the near term. Mandatory notification when a
financial institution detects a likely synthetic voice or deepfake fraud
attempt, combined with pre-agreed verification protocols for high-value
transfers, would address the most common attack vectors at relatively low
implementation cost. The political will to require those standards exists in
principle; the specific regulatory mandate has not yet been issued in most
jurisdictions.
About the Author
Stuart
Kerr is a technology correspondent at LiveAIWire, covering artificial intelligence,
emerging technologies, and their impact on society and industry.