By Stuart Kerr, Technology Correspondent, LiveAIWire

Black box medicine has a new legal test attached to it. On January 6, 2026, the FDA issued updated guidance on clinical decision support software that, for the first time, ties a product’s regulatory status directly to how well clinicians can see inside it. According to a summary published by law firm Faegre Drinker, the agency’s new position is blunt: the greater the extent to which software is a black box to healthcare professionals, the greater the risk the FDA will classify it as a regulated medical device rather than a lighter-touch decision-support tool.

That single sentence reframes a debate that has run for years in radiology, pathology, and diagnostics. AI models that sift tumor scans, triage emergency patients, and flag early-stage cancers have kept outperforming humans on raw accuracy while remaining unable to explain, in terms a clinician can act on, why they reached a given conclusion. The FDA’s 2026 guidance does not ban that opacity. It makes opacity itself a regulatory trigger.

What Black Box Medicine Actually Means in Practice

Black box medicine refers to AI models, typically deep learning networks used in imaging and diagnostics, whose internal logic is too complex or too proprietary for a treating clinician to meaningfully interrogate. The model produces a recommendation; it does not produce a chain of reasoning that a doctor can check against clinical judgment. For software to legally qualify as clinical decision support rather than a regulated device, the FDA’s updated guidance requires it to support, not drive, a clinician’s decision, and to let that clinician independently review the basis for its recommendation. A tool that cannot explain itself in accessible documentation no longer meets that bar automatically.

The practical effect on developers is significant. Companies building AI-driven clinical decision support tools must now provide clear, accessible documentation of what data the software uses, the logic behind its recommendations, and how outputs are generated, delivered through labeling, user manuals, or explainability features built directly into the interface. Software that analyzes continuous physiological signals, medical images, or genomic data in ways that effectively drive a diagnosis, rather than merely informing one, generally remains subject to full FDA device oversight regardless of how the explanation is packaged.

The Liability Gap Nobody Has Closed

Explainability requirements address one half of the underlying problem. The other half, who is responsible when an opaque system gets it wrong, remains largely unresolved. A May 2026 analysis published by the International Multispeciality Journal of Health walks through the mechanics of the problem: diagnostic responsibility that once rested with a single clinician is now distributed across data curators, model developers, device manufacturers, hospitals, and the treating physician who ultimately acts on the AI’s output. When a system’s reasoning cannot be reconstructed, courts and regulators struggle to isolate which link in that chain actually failed.

The same analysis describes clinicians in this position as “moral crumple zones,” a term borrowed from human-automation research to describe how legal and reputational responsibility collapses onto the most visible human in a system, even when that person had limited ability to override or understand the tool they relied on. A radiologist who defers to an AI model that meets regulatory approval, follows institutional protocol, and still misses a malignant nodule is left legally exposed in a way that existing negligence law was not built to resolve cleanly.

Explainable AI: Progress, Not a Solution

Explainable AI, often shortened to XAI, is the industry’s proposed fix, and it has made real progress since the early “black box or nothing” era. Techniques including heatmaps overlaid on medical images, simplified decision trees, and feature-attribution scores now let developers show which parts of an input most influenced a model’s output. Coverage of the FDA’s shift toward what one physician commentary called “glass box” transparency, published on the physician platform KevinMD in January 2026, frames the change as a meaningful improvement in regulatory posture, while also cautioning that the guidance rests on an assumption that time-pressed clinicians will actually engage with layered explanations rather than clicking past them during a busy shift.

The deeper limitation is structural rather than a matter of interface design. Most explainability methods work by approximating a model’s reasoning after the fact, not by exposing the reasoning itself, since the actual computation is distributed across millions of weighted parameters that do not map onto any single human-readable rule. An explanation generated this way can be genuinely useful for spotting an anomaly and still fail to meet the standard of justification a court or a skeptical patient would expect from a human diagnosis.

Bias Compounds the Trust Problem

Opacity and bias reinforce each other in ways that make this kind of AI riskier than either problem alone. A diagnostic model trained on data that underrepresents women or minority patients can learn to systematically underestimate risk for those groups, and because the model cannot explain its own reasoning, that systematic error can persist for months or years before anyone notices the pattern. Our own reporting on why bias mitigation in AI systems is harder than it looks found that even well-resourced organizations struggle to agree on which mathematical definition of fairness to optimize for, let alone verify that a deployed model meets it. In medicine, that ambiguity translates directly into which patients get flagged early and which do not.

What This Means for Patients and Clinicians

For patients, the practical shift is that AI is already involved in more of their care than most realize, often through consumer devices rather than hospital equipment. Our coverage of what your smartphone actually knows about your health found that the same regulatory logic, lighter oversight for low-risk wellness tools and stricter standards for anything closer to diagnosis, now shapes everything from a phone’s heart rhythm alerts to hospital-grade imaging AI, and that most people cannot reliably tell which category a given app falls into just from its marketing description.

For clinicians, the honest position is neither uncritical adoption nor blanket rejection. A model that outperforms unaided human review on early cancer detection is worth using even when its full internal logic resists explanation, provided it is treated as one input reviewed alongside clinical judgment rather than as a final answer. That same caution about treating AI output as more definitive than it has earned applies well beyond diagnostics, including in AI mental health tools now showing genuine clinical trial evidence alongside real limits around crisis detection and long-term safety that no amount of interface polish resolves on its own.

Black box medicine is not going away, and it should not. The evidence that these models catch what human clinicians miss is too strong to discard over a discomfort with opacity alone. What the FDA’s 2026 guidance signals, and what the unresolved liability questions underscore, is that the industry is finally being asked to prove interpretability rather than assume it will be forgiven. Whether that pressure produces genuinely transparent systems or simply better-documented black boxes is the question that will define clinical AI adoption for the rest of the decade.

About the Author
Stuart Kerr is Technology Correspondent at LiveAIWire, covering artificial intelligence, emerging technology, and their impact on business, society, and everyday life. LiveAIWire publishes original AI journalism every weekday at liveaiwire.com.