By Stuart Kerr, Technology Correspondent, LiveAIWire

Black defendants who did not reoffend were nearly twice as likely as white defendants to be wrongly flagged as high risk by COMPAS, the algorithmic tool used across American courts to help decide bail and sentencing. That finding comes from a 2016 ProPublica investigation into the Correctional Offender Management Profiling for Alternative Sanctions system, and it became one of the founding case studies in AI sentencing bias, triggering a decade of legal and statistical argument over what fairness means when a machine scores a person’s likelihood of committing a future crime. The company behind COMPAS confirmed the underlying numbers while disputing that they amounted to bias, arguing that the tool predicted recidivism at similar rates across racial groups.

A 2026 peer reviewed study in the journal AI and Ethics revisited the original COMPAS dataset using newer machine learning methods, including gradient boosting, and found that standard debiasing techniques produced only modest gains in fairness. That result is evidence the disparity ProPublica identified in 2016 has proven resistant to a decade of technical refinement. It reframes the COMPAS debate: this was never simply a bug that better code could fix.

AI in criminal justice has expanded well beyond COMPAS. Predictive policing tools, pretrial risk assessment tools, parole decision support systems, and prison classification algorithms are now deployed across multiple US states, UK police forces, and criminal justice systems in several European countries. The evidence base for their effectiveness has grown alongside the evidence base for their harms, and both remain contested. That contestation is not primarily a technical question. It is a question about what the criminal justice system is for, what fairness requires in a system with centuries of documented racial disparity, and who bears accountability when an algorithm gets it wrong.

How These Systems Work and Why Bias Is Structural

Risk assessment tools generate scores by comparing defendants to historical populations and weighting factors statistically associated with recidivism in that data. The inputs include prior criminal history, employment status, education level, residential stability, substance use history, and peer associations, variables that correlate with socioeconomic status and, through socioeconomic status, with race in the United States and other historically stratified societies. A defendant’s ZIP code, for example, functions as a strong proxy for race in most US cities because of decades of residential segregation, so a score that never asks about race directly can still sort defendants along racial lines with real fidelity.

A 2025 Annual Review of Criminology paper by Roland Neil and Michael Zanger-Tishler identifies the mechanism precisely: racial bias in arrest records, itself a documented consequence of differential policing intensity across communities, produces algorithmic bias through several distinct pathways, including label bias in what the model is trained to predict and sample bias in whose data it learns from. The tool learns from data that reflects enforcement rates rather than crime rates, and enforcement rates are shaped by policing decisions that carry their own historical biases. The algorithm cannot distinguish between the two. It sees only the outcomes of past decisions and learns to predict future outcomes that resemble them.

The Mathematical Proof That AI Sentencing Bias Can’t Be Engineered Away

White defendants who did reoffend were, by contrast, more often mislabeled as low risk than black reoffenders, the mirror image of COMPAS’s error against black defendants. In a 2016 proof, computer scientists Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan demonstrated that these two errors, along with a third common definition of fairness called calibration, are mathematically incompatible whenever base rates differ across groups. This is not a limitation specific to COMPAS. It is a constraint on any classification system applied to groups with different prior probabilities of the outcome being predicted, and no engineering fix can satisfy all three definitions of fairness at once when those base rates diverge.

That result matters because historical recidivism rates differ across racial groups, and those rates differ in part because of differences in arrest rates, prosecution rates, and sentencing patterns that themselves reflect decades of documented racial disparity. The fairness problem in criminal justice AI is not an engineering problem that better algorithms can solve. It is a consequence of applying algorithmic tools to a domain where the historical data encodes structural injustice, and any algorithm trained on that data will tend to reproduce the injustice in a more systematic and less visible form than a human decision maker would.

The 2026 AI and Ethics study illustrates why. Researchers Kienzle, Velarde, and Gannon reproduced earlier results using logistic regression and support vector machines on the COMPAS dataset, then added a gradient boosting classifier and tested two standard debiasing techniques, hyperparameter optimization and a correlation remover method that strips out correlated features. The more sophisticated model performed better on raw accuracy, but the debiasing techniques produced only modest improvement in fairness metrics. Ten years and multiple generations of machine learning methods after ProPublica’s original analysis, the underlying disparity has not been engineered away, which is consistent with Kleinberg, Mullainathan, and Raghavan’s proof that it cannot be.

What This Means for You

If you are ever arrested in a jurisdiction that uses an actuarial risk tool, a number generated from historical data, not from anything you have personally done beyond your own record, may influence whether you wait for trial in a cell or at home, and later whether you are paroled or classified as high security. Judges are told the score is advisory. Research on automation bias in human-AI interaction, documented across sectors well beyond criminal justice, suggests that a number presented with the appearance of statistical authority shapes decisions even when it is formally treated as one factor among many.

The practical asymmetry is stark. Prosecutors and courts have access to the score, while defendants and their lawyers, in most jurisdictions, do not have access to the methodology that produced it. Knowing that this asymmetry exists, and asking your own state or local courts whether risk assessment tools are in use and what oversight applies to them, is one of the few points of leverage available to the public in a system that has moved faster than the legal protections meant to govern it.

The Trade Secret That Courts Can’t Examine

The Harvard Law School Center on the Legal Profession’s research on AI and racial bias in legal decision-making identifies the problem that distinguishes AI bias in criminal justice from AI bias in other domains: bias in legal decision-making is harder to detect. When COMPAS was used in state courts for sentencing, its racial bias in predicting recidivism went largely unchallenged for years, even though empirical studies later exposed the flaws. If AI models used elsewhere in legal practice inherit similar biases, they could produce unfair outcomes without raising the immediate red flags that would prompt scrutiny.

That difficulty of detection compounds a second problem: proprietary secrecy. COMPAS is owned by Equivant, formerly Northpointe Inc., which treats the algorithm as a trade secret and does not disclose the specific variables, weights, or methodology used to generate risk scores. When the Wisconsin case State v. Loomis reached the state Supreme Court, the justices ruled that judges could use COMPAS scores but had to treat them as one factor among many rather than as determinative. The court reached that decision without ever having access to COMPAS’s methodology, because the vendor declined to disclose it, and the due process implications of that gap remain unresolved by any US court or legislature.

The defense counsel challenging a COMPAS score in court faces a system whose methodology the vendor will not produce, leaving the defendant without the information needed to contest the basis of a decision that may determine years of their liberty. The European Union’s approach points to an alternative. The EU AI Act classifies criminal justice risk assessment as high risk and requires transparency about training data and methodology before such a system can be deployed. No equivalent federal requirement exists in the United States, and the political dynamics of criminal justice policy make that kind of transparency legislation harder to pass than in other domains.

How States Are Trying to Force Disclosure

Reform is proceeding through legal challenge, legislation, and academic advocacy at once, mostly at the state level. California’s Racial Justice Act allows defendants to challenge sentences using evidence of racial bias in the charging and sentencing process, and it has generated legal work applying AI tools to analyze sentencing data for evidence of bias, a case study in using AI to challenge AI. New York’s Assembly Bill A7172 would require law enforcement agencies to disclose the AI and facial recognition systems they use in investigations, including error rates and known biases, and to submit to independent audits with publicly accessible results.

The academic consensus, reflected in the Harvard and Annual Review of Criminology research cited above, is that risk assessment tools should not be opposed in principle but should be held to a standard requiring demonstrated improvement over human decision-making, measured across the full distribution of errors and their consequences, not just overall accuracy. The argument that AI is better than the biased human decision-making it replaces is not, by itself, sufficient justification for deploying tools whose biases are structural and whose harms concentrate on communities that are already disadvantaged. For related coverage of algorithmic policing, see LiveAIWire’s analysis of AI in law enforcement.

Why One Flawed Score Follows You for Years

Beyond sentencing, AI risk scores are used in parole decisions and prison classification, determining where an individual is housed, what programs they can access, and when they are considered for release. The same structural bias affecting pretrial and sentencing tools affects these downstream applications, and it compounds over time. A risk score generated before trial that elevated a defendant’s perceived risk will influence the prison classification decision and the parole recommendation for the same person years later, so each subsequent AI application builds on the initial error rather than correcting it.

Prison classification determines whether someone is housed in a minimum, medium, or maximum security facility, whether they have access to education, vocational training, and mental health services, and how they are treated by staff who see a risk score before they see the person it describes. LiveAIWire’s earlier reporting on AI in prisons documented how predictive surveillance and risk scoring inside correctional facilities operate with weaker oversight than the sentencing decisions that precede them. The cumulative effect of AI errors across a person’s entire path through the justice system, from arrest to sentencing to parole, is a documented harm that no single disclosure requirement addresses, because it is a property of the system rather than of any one tool.

Why This Is a Political Choice, Not a Technical One

The mathematical impossibility of satisfying every reasonable fairness definition at once means the choice of which definition to prioritize in criminal justice AI is a political and moral choice, not a technical one. It requires democratic deliberation about whose interests the criminal justice system serves and whose it burdens, deliberation that AI deployment has largely proceeded without. The wider pattern, in which AI systems trained on historical data reproduce the discrimination embedded in that data, is not unique to criminal justice. LiveAIWire has covered the same dynamic in algorithmic bias across hiring, healthcare, and lending, and in AI at the border, where similar accountability gaps affect migrants and asylum seekers.

The institutions best positioned to force that deliberation are the courts, through constitutional challenge to the use of proprietary algorithmic tools in sentencing and parole, and the legislatures, through transparency and audit requirements that treat algorithmic decisions about human liberty with the same seriousness as human decisions about human liberty. Both pathways are slow. The deployment of AI in criminal justice is not.

About the Author

Stuart Kerr is Technology Correspondent at LiveAIWire, covering artificial intelligence, emerging technology, and their impact on business, society, and everyday life. LiveAIWire publishes original AI journalism every weekday at liveaiwire.com.