By Stuart Kerr, Technology Correspondent, LiveAIWire
A public database maintained by legal researcher Damien Charlotin has logged 1,731 court decisions worldwide involving AI hallucinated citations, invented case law, fake quotations, or misstated legal authority that a party relied on and a court caught. Most were filed by self-represented litigants using consumer chatbots. A growing share were filed by qualified lawyers who should have known better. In December, a federal magistrate judge in Oregon fined two attorneys a combined 110,000 dollars after they filed briefs containing 15 references to nonexistent cases and eight fabricated quotations, then tried to quietly remove the fake material rather than disclose it. The judge called it “a notorious outlier in both degree and volume.” It will not stay an outlier for long.
Can AI draft your legal defence? Technically, increasingly, yes. Whether it should do so without a qualified lawyer checking every line is the question courts, regulators, and bar associations spent the past year trying to answer, with uneven results.
Table of Contents
The AI Hallucinated Citations Crisis Courts Are Now Tracking
Charlotin’s database, which tracks cases where a court has explicitly found that hallucinated AI content was relied upon, shows 1,193 of its cases originating in the United States alone, with lawyers responsible for 669 of the roughly 1,731 incidents tracked globally and self-represented litigants responsible for most of the rest. The nature of the fabrications varies: invented case law accounts for the largest share, followed by misquoted or misrepresented real cases and outright false quotations attributed to judges who never wrote them. Sanctions range from a written warning to referral for disciplinary action to, in the Oregon case, a six-figure financial penalty and dismissal of the underlying claim.
The Oregon case is instructive because of what it reveals about verification, not just generation. Attorney Stephen Brigandi filed three briefs over five months containing the fabricated material. When opposing counsel flagged the errors, Brigandi removed them and refiled rather than explaining how they got there. Judge Mark Clarke found no evidence he had verified any of the legal arguments in his amended filing either. The lesson courts keep drawing is not that lawyers used AI. It is that they filed its output without checking it, which existing professional conduct rules already prohibit regardless of how the error was produced.
What This Means for You
If you are representing yourself in a legal matter and considering a chatbot to help draft filings, treat every case name, quotation, and statute citation it produces as unverified until you have located the actual source. General-purpose AI tools are not connected to a live legal database and will generate plausible-sounding citations that do not exist, a failure mode distinct from ordinary mistakes because the output reads as confident and precise. If you are working with a lawyer, it is reasonable to ask directly whether AI was used to draft your documents and what verification process was applied before filing. Courts increasingly expect that disclosure, and some now require it explicitly.
How Legitimate Legal AI Tools Actually Work
The tools built specifically for legal work are architecturally different from a general chatbot, which is why the hallucination pattern looks different in practice. Harvey, the legal AI startup valued at 11 billion dollars after its March 2026 funding round, now works with more than 100,000 lawyers and the majority of the AmLaw 100, the ranking of the largest US law firms by revenue. Thomson Reuters built its CoCounsel product on top of Westlaw, a licensed legal database, so outputs are grounded in verified source material rather than generated freely from a general training set. Neither company claims its tools eliminate the need for a lawyer to review the output.
That distinction, a licensed database with citation grounding versus an open-ended chatbot, is the one missing from most of the hallucination cases in Charlotin’s database. It is also the one Thomson Reuters found most legal professionals have not fully internalised. In its 2026 AI in Professional Services Report, based on more than 1,500 respondents, the company found generative AI use nearly doubled in a year, with 40 percent of organisations now using it, up from 22 percent. Yet only 18 percent of professionals say their organisation actually tracks the return on that investment, meaning many are deploying AI in high-stakes work without a clear picture of where it succeeds or fails.
Judges Are Using AI Too, Quietly and Cautiously
Courts are not immune from the same pressures facing the lawyers appearing before them. The UK judiciary issued updated AI guidance in October 2025 that permits judges, clerks, and legal advisers to use AI tools but places full personal responsibility on the judicial office holder for anything that goes out under their name, with an explicit warning that AI-generated information may be inaccurate and must be independently verified before use. The guidance also flags a newer risk: hidden prompt text embedded in documents that is invisible to a human reader but readable by an AI system reviewing the file.
England and Wales went further in February 2026, when the Civil Justice Council opened an eight-week consultation, chaired by Lord Justice Birss, on whether formal rules are needed to govern AI use by legal representatives preparing pleadings, witness statements, and expert reports. Early responses suggested existing professional conduct rules are adequate for routine legal drafting, but that witness statements may need a declaration confirming AI was not used to alter a witness’s actual evidence.
Judges Weigh In on AI Evidence Too
In the United States, the Judicial Conference’s rules committee has approved a draft Federal Rule of Evidence 707, which would require AI-generated evidence offered without a supporting human expert witness to clear the same reliability bar already applied to expert testimony under Rule 702. Public comment on the proposal closed in February 2026, and if the rule survives further review it would take effect no earlier than December 2027. Supporters argue federal courts need one consistent standard rather than each judge improvising an approach case by case. Critics, including several public defender groups, warn that requiring a technical expert to challenge AI-generated evidence could put under-resourced defendants at a further disadvantage against prosecutors who can afford one.
Who Gets Left Behind When Only Big Firms Can Verify
Charlotin’s database breaks its cases down by who was using the AI tool that produced the hallucination, and the pattern is stark. Self-represented litigants account for the largest single category, well over half of all logged cases, followed by lawyers, then a smaller number involving experts, paralegals, and even judges. Harvey’s 100,000 lawyers and 500 in-house legal teams sit at one end of a widening gap. A person representing themselves in a landlord dispute or a small claims matter, with no institutional subscription to a database-grounded tool and no colleague to check their filing, sits at the other.
This is the access to justice paradox at the centre of the AI legal drafting debate. The same technology that could, in principle, help someone without the money for a lawyer draft a competent filing is also the technology most likely to get that same person sanctioned, because the free consumer tools available to them carry none of the citation grounding built into the products large firms pay for. Legal aid organisations and court self-help centres are the obvious place to close that gap, but few currently have the budget or technical staff to evaluate which AI tools are safe to recommend to the public they serve.
The Adoption Numbers Nobody Has Fully Reckoned With
The scale of legal AI adoption now underway makes the verification gap more consequential, not less. Regulatory frameworks written before generative AI existed are being asked to govern a technology now embedded in document review, legal research, and first-draft contract generation across a majority of large firms. Harvey’s own figures put its footprint at over 500 in-house legal teams and 60 countries, evidence that the shift from experimentation to standard practice happened faster than most firms updated their internal verification procedures to match.
The result is a two-speed profession. Firms with licensed, database-grounded tools and mandatory citation-checking workflows are seeing genuine productivity gains without the hallucination problem showing up in their filings. Self-represented litigants and under-resourced practices reaching for free consumer chatbots, with no citation grounding and no institutional review layer, account for the overwhelming majority of the cases in Charlotin’s database. The technology is not uniformly risky. The verification infrastructure around it is what varies, and that infrastructure costs money most solo practitioners and pro se litigants do not have.
The Ethical Fault Lines That Remain
The same structural bias documented in AI systems used for hiring and healthcare applies to legal AI trained on historical case law, which reflects the outcomes of a justice system with its own well-documented disparities. A tool trained to predict case outcomes from precedent will reproduce the biases embedded in that precedent unless specifically corrected for, and few commercial legal AI products publish the kind of independent bias audit that would let a client assess that risk. This is a harder problem than hallucination, because a fabricated citation is usually detectable on inspection while inherited bias is not. LiveAIWire’s reporting on predictive sentencing tools found standard debiasing techniques applied to the COMPAS recidivism algorithm produced only modest fairness gains.
Accountability is the second unresolved fault line. When an AI-assisted filing causes harm, whether through a fabricated citation that wastes a court’s time or a biased risk score that influences an outcome, professional responsibility currently sits entirely with the human lawyer who signed and filed it. Every guidance document issued so far, from the UK judiciary to the Civil Justice Council to individual state bar associations, affirms that principle rather than shifting any responsibility onto the AI vendor. That is likely to remain the position for the foreseeable future, which makes independent verification, not AI avoidance, the actual professional standard being enforced.
Where the Rules Are Heading
None of the frameworks currently in development, the UK consultation, the proposed Federal Rule of Evidence 707, or the patchwork of state bar guidance across the US, treat AI-assisted drafting as prohibited. All of them converge on the same requirement: verification before filing, disclosure when asked, and full professional accountability for whatever ends up in front of a judge. The Oregon sanction, the Charlotin database’s 1,731 logged cases, and the UK’s hidden-prompt warning all point at the same underlying fact. The technology that can draft a competent legal argument in seconds has not changed what a lawyer, or a self-represented litigant, is required to do before signing their name to it.
About the Author
Stuart Kerr is Technology Correspondent at LiveAIWire, covering artificial intelligence, emerging technology, and their impact on business, society, and everyday life. LiveAIWire publishes original AI journalism every weekday at liveaiwire.com.