By
Stuart Kerr, Technology Correspondent, LiveAIWire

In 2018, a deep learning system developed at Google demonstrated
that it could detect diabetic retinopathy from fundus photographs with
accuracy matching that of experienced ophthalmologists. When researchers
asked the model to explain why it had flagged a particular image as
high-risk, it could not. The performance was real; the reasoning was absent.
That tension  —  between demonstrable clinical accuracy and
opaque decision-making  —  defines the central challenge of AI in
medicine.

Healthcare is now one of the most active domains for AI
deployment, with systems making or influencing decisions across radiology,
pathology, drug discovery, treatment planning, and clinical risk assessment.
The clinical performance of many of these systems is well-evidenced. The
question of whether clinicians, patients, and regulators should trust them
when their internal reasoning is fundamentally inaccessible is less
settled.

Where AI Is Already Making Medical Decisions

Radiology has been the most heavily penetrated domain. FDA-cleared
AI systems now assist with the detection of pneumonia on chest X-rays,
pulmonary embolism on CT scans, breast cancer on mammograms, and stroke on
brain MRI. The evidence base for many of these applications is strong: in
controlled studies, AI-assisted radiology has demonstrated sensitivity and
specificity that matches or exceeds specialist radiologist performance, and
in some studies has reduced missed diagnoses in conditions where early
detection is critical.

Sepsis prediction is another high-stakes application. Systems
trained on electronic health record data 
—  vital signs, laboratory
results, nursing observations, medication changes  — 
can identify patients at risk of septic shock hours before clinical
deterioration becomes apparent. Several large hospital systems have deployed
these tools; the evidence on whether they reduce mortality in real-world
settings is mixed, partly because the alert systems can generate alarm
fatigue when false-positive rates are high.

In pathology, AI systems trained on digitised slide images are
being used to grade tumour severity, identify cancer subtypes, and predict
treatment response. These are decisions with direct implications for the
aggressiveness of treatment a patient receives. The opacity of the models
making those assessments  —  and the difficulty of challenging them when
a patient believes a decision was wrong 
—  is a live concern among
clinical ethicists.

The Explainability Problem

Most high-performing medical AI systems use deep learning
architectures  —  neural networks with millions of
parameters  —  that produce accurate outputs through
computations that cannot be straightforwardly translated into
human-interpretable rules. When such a model classifies a scan as malignant,
it has processed pixel patterns at multiple scales in ways that do not reduce
to the kind of reasoning a radiologist could articulate: “this shadow
has these characteristics, which I associate with this condition, because of
these prior cases.”

Explainable AI (XAI) research aims to address this through
techniques that generate post-hoc explanations  — 
heat maps showing which image regions influenced the classification, or
feature importance scores indicating which variables drove a risk prediction.
These explanations are often clinically useful, but they are also
approximations of the model’s actual computation rather than faithful
representations of its internal reasoning. A heat map that highlights the
tumour region in a cancer-positive scan may look like confirmation that the
model is reasoning correctly, but does not guarantee that the highlighted
region is causally responsible for the classification.

What this means for you as a patient: when an AI system
contributes to a clinical decision about your care, you are entitled to ask
how that contribution was made. In practice, most clinical AI systems are
presented to patients as tools that support clinical judgment, with limited
transparency about their specific role in the decision. That framing protects
the institution but may not serve the patient’s interest in understanding the
basis of their care.

Do Clinicians Trust the Black Box?

Clinical adoption of AI tools has been uneven, and trust is a
significant factor. Radiologists who have worked alongside AI
decision-support systems report a spectrum of responses: some find the tools
useful as a second check on decisions they have already made independently;
others report over-reliance, deferring to AI outputs on cases where their own
assessment diverged; and some report systematic distrust, especially when AI
alerts do not match their clinical intuition.

Research published in the
New England Journal of Medicine on AI in clinical decision-making
has noted that the introduction of AI tools can alter diagnostic behaviour in
ways that are not uniformly beneficial 
—  reducing the thoroughness of
independent assessment among clinicians who assume the AI is a reliable
backstop, while simultaneously failing to catch errors that the AI
introduces.

Regulatory Frameworks and Their Limits

Medical AI is regulated as a medical device in most jurisdictions.
In the United States, the FDA has cleared hundreds of AI-based medical
software applications, primarily in radiology and cardiology. Clearance
typically requires evidence of clinical performance in a test population, but
not a demonstration of why the system performs as it does. The regulatory bar
for explainability is currently low relative to what many clinical ethicists
believe the stakes require.

The EU’s Medical Device Regulation and the AI Act together create
a more demanding framework for high-risk AI medical devices in Europe,
requiring transparency, clinical evidence, and post-market surveillance.
Neither framework requires that AI systems be fully interpretable; they
require that their performance be monitored and that significant failures be
reported and acted upon.

The growing
role of AI in medical decision-making at every scale  — 
from nanobot therapeutic agents to population-level risk
prediction  —  makes the question of trust and
transparency increasingly urgent. The framework for answering it is still
being constructed, and patients are already living inside the
experiment.

The parallel with algorithmic
decision-making in other high-stakes contexts is clear: the pattern
of deployment outpacing governance, and of efficiency arguments overriding
transparency demands, is consistent across domains. In medicine, the stakes
are measured in lives. That ought to change the pace of the governance
response, even if it has not yet done so.

The FDA’s
framework for AI-enabled medical devices is evolving but remains
focused primarily on performance evidence rather than explainability
requirements, reflecting a regulatory philosophy that prioritises
demonstrated outcomes over interpretable reasoning  —  a
trade-off that not all clinical ethicists accept.

The
governance challenge  —  deploying high-performing AI in clinical
settings without adequate transparency or accountability frameworks  — 
connects to broader patterns identified in AI
systems making consequential decisions without adequate oversight
across multiple high-stakes domains.

The question of when
AI performance evidence is sufficient to justify clinical adoption without
full explainability is ultimately a values question, not a technical one. If
a model demonstrably reduces missed cancer diagnoses by thirty percent, the
case for deployment is strong even if the mechanism of that improvement
cannot be fully articulated. If the same model performs well on average but
fails systematically for a specific patient subgroup, the aggregate
performance figure conceals a distributional harm that aggregate performance
metrics do not reveal. The evaluative framework for medical AI needs to
account for both dimensions, and most current frameworks do
not.

The trust deficit in black-box medical AI is not
primarily a technical problem, though technical solutions exist. It is a
governance problem: institutions have adopted AI tools whose performance
characteristics meet regulatory thresholds while their transparency
characteristics fall well short of what clinicians, patients, and ethicists
believe the stakes require. Resolving it requires either raising the regulatory
transparency bar or developing a robust framework for when performance
evidence alone is sufficient to justify deployment. Neither conversation has
yet reached a conclusion, and patients are living inside the experiment that
the absence of that conclusion creates.

About the
Author

Stuart Kerr is a technology correspondent at
LiveAIWire, covering artificial intelligence, emerging technologies, and
their impact on society and industry.