By
Stuart Kerr, Technology Correspondent, LiveAIWire
A software engineer at a major financial institution recently
described her working day: she writes a specification, feeds it to an AI
coding assistant, reviews the output, edits it, asks the AI to fix the parts
that are wrong, reviews again, and eventually integrates code she has not
written line by line into a system that processes millions of transactions.
She is faster than she was two years ago. She is also less certain, at each
moment, that she fully understands what the system does. That uncertainty is
the central risk of AI-driven programming, and it is arriving faster than the
engineering culture and tooling required to manage it.
AI coding assistants have genuinely transformed software
development productivity. The question is whether the productivity gain is
being accompanied by the quality, security, and comprehensibility standards
that software systems — particularly those running critical infrastructure
— require. The early evidence is mixed in ways that the enthusiastic
adoption narrative does not always acknowledge.
What AI Coding Tools Actually Do
Tools like GitHub Copilot, Cursor, and Claude Code operate
primarily through a combination of autocomplete and generation: they suggest
code completions based on context, generate functions from natural language
descriptions, explain existing code, and propose fixes for identified bugs.
The underlying models are trained on large corpora of publicly available
code, which means their suggestions reflect patterns that are common in that
training data — including patterns that are insecure, deprecated, or simply
wrong.
Research from Stanford University published in 2022 found that
developers using AI coding assistants wrote code with significantly more
security vulnerabilities than those who did not, partly because the AI tools
suggested code that looked correct but contained subtle flaws that the
developer, working faster and reviewing less carefully, failed to catch. The
finding prompted methodological debate, but subsequent research has confirmed
the core concern: AI-generated code tends to be plausible-looking, which
reduces the scrutiny it receives relative to code that is obviously
incomplete or syntactically incorrect.
What this means for software teams adopting AI coding tools: the
productivity gains are real and the quality risks are real. Managing both
requires deliberate process adaptation — code review standards that account
for AI-generated code, security scanning integrated into development
pipelines, and a culture that treats AI-generated code as a starting point
requiring verification rather than a finished product requiring only
approval.
The Security Vulnerability Problem
AI coding models trained on public repositories inherit the
security practices of those repositories, which are variable. Stack Overflow
answers, GitHub projects, and tutorial code — all well-represented in
training data — frequently contain security vulnerabilities that have not
been corrected because the primary purpose of the code is illustration, not
production deployment. An AI model that has learned from millions of such
examples will generate code that reflects those practices.
Common vulnerability categories appear in AI-generated code at
rates that security researchers find concerning: SQL injection
vulnerabilities, insecure random number generation, improper error handling
that leaks sensitive information, and hardcoded credentials. These are not
obscure edge cases; they are among the most frequently exploited
vulnerability classes in production software. Research from the cybersecurity
research community has documented the presence of these patterns in
code generated by major AI coding assistants across multiple programming
languages.
The software supply chain dimension is particularly significant.
AI-generated code that is integrated into open-source libraries and
frameworks inherits the trust placed in those components by downstream users.
A vulnerability introduced by AI generation into a widely used library can
affect thousands of dependent applications. The mechanisms for identifying
and remediating such vulnerabilities are the same as for human-written code,
but the rate at which AI-generated code is entering shared repositories is
increasing the surface area that requires monitoring.
Comprehensibility and the Understanding Gap
Software that developers do not fully understand is software that
cannot be reliably maintained, debugged, or extended. The concern that AI
coding tools are producing code that works but that its nominal authors do
not fully comprehend is not hypothetical: it is reported consistently by
experienced developers working with AI assistants, and it has implications
for long-term software quality that go beyond the immediate productivity
gains.
Code comprehensibility is not just a quality concern; it is a
safety concern in systems where the consequences of failures are severe. A
financial system, a medical device controller, or critical infrastructure
software whose maintenance team does not fully understand its codebase is a
system with a hidden reliability risk that will not surface until something
goes wrong. The history of software failures in critical systems frequently
features this pattern: code that passed all tests and performed correctly
until an edge case that the development team did not fully understand caused
a failure with significant consequences.
The explainability
problem in AI medical systems has a direct parallel in AI-generated
code: in both cases, a system that performs well on observed inputs but whose
internal logic is opaque to the humans responsible for it presents risks that
are difficult to quantify until they materialise.
AI-Generated Code in Critical Infrastructure
The adoption of AI coding tools is not uniform across software
domains. Consumer application development, where the cost of errors is
moderate and iteration is fast, has adopted AI assistance most extensively.
Safety-critical domains — aerospace, medical devices, nuclear control
systems, financial infrastructure — have been more cautious, partly because
their existing engineering standards and regulatory requirements create
friction for novel tools.
That caution is appropriate and should be maintained. The IEC
61508 functional safety standard, the DO-178C aviation software standard, and
equivalent frameworks in other safety-critical domains require evidence of
software correctness that AI-generated code cannot easily provide under
current tooling. Formal verification, static analysis at a level of rigour
that AI coding assistants do not yet support, and the documentation standards
required for regulatory approval all represent friction points where
AI-generated code faces genuine barriers to adoption in the most critical
applications.
The research community working on AI code verification is
addressing these barriers, developing tools for automatically checking
properties of AI-generated code and generating formal proofs of correctness
for specific classes of programs. Progress is real, but the gap between
current AI coding tools and the standards required for safety-critical
deployment remains significant.
What Good Practice Looks Like
The software engineering community is developing norms for
AI-assisted coding that balance productivity with quality. Mandatory security
scanning of all AI-generated code before integration, peer review processes
that specifically check AI contributions, and the maintenance of human
expertise in core system components that are not delegated to AI generation
are among the practices that forward-thinking engineering teams are adopting.
The UK
National Cyber Security Centre has published guidance on the use of
AI coding tools that addresses the security implications specifically,
recommending that organisations treat AI-generated code with the same
scrutiny as third-party library code — a standard that has well-established
practices for security review and dependency management. Applying those
practices to AI-generated code does not eliminate the productivity gains; it
ensures that those gains do not come at the cost of security posture that is
harder to recover than the time saved in development.
The connection to the broader question of who
governs the most powerful AI systems applies at the code level: the
software that runs critical infrastructure is increasingly written with AI
assistance, and the governance of that process — who is responsible when
AI-generated code fails, and what standards it must meet — is being
established through practice rather than policy, in real systems, before the
consequences of poor choices are fully understood.
The economic incentives driving AI coding adoption are powerful
enough to override caution in most commercial development environments.
Developers who use AI assistants produce more code faster; teams that adopt
AI coding tools complete projects ahead of teams that do not; organisations
that embrace AI coding tools can serve more clients with the same headcount.
These incentives are real and are driving adoption faster than the quality
and security frameworks required to manage the risks. The result is a
situation in which the software industry is knowingly accumulating technical
debt of a new kind: AI-generated code whose security properties are not fully
understood, integrated into systems that will be expensive to audit and
correct later. Managing that accumulation requires deliberate investment in
quality infrastructure that most organisations are not yet making at the
required scale. The broader context of AI
systems managing consequential processes with insufficient
transparency applies directly here.
About the
Author
Stuart Kerr is a technology correspondent at
LiveAIWire, covering artificial intelligence, emerging technologies, and
their impact on society and industry.