In 2016, ProPublica published an investigation into COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — a risk-assessment tool used by judges across the United States to inform bail and sentencing decisions. The investigation found that the algorithm was twice as likely to falsely flag Black defendants as future criminals compared to white defendants. Northpointe, the company behind COMPAS, refused to release the algorithm's weights, citing trade-secret protections. Defendants had no way to challenge scores they could not see.
The case crystallised a fundamental tension: opacity is often not an oversight but a deliberate product choice, and that choice carries severe distributional consequences.
Transparency in AI decision making is not a single toggle. It is a spectrum of disclosures — from revealing that an algorithm exists, to showing which inputs it uses, to exposing the mathematical relationships between those inputs and outputs. Each layer of disclosure has different legal, commercial, and ethical implications.
Researchers distinguish three levels: algorithmic transparency (knowing the system exists and its general logic), data transparency (knowing what training data shaped it), and outcome transparency (knowing why a specific decision was reached for a specific person). Most public controversy focuses on the third level, because it is the one that affects individual rights.
Commercial incentives reward opacity. Algorithms are expensive to build and easy to copy; trade-secret law protects them. Regulators have historically lacked the technical capacity to audit what they cannot read. And most affected individuals — loan applicants, criminal defendants, welfare recipients — lack both the legal standing and the technical vocabulary to demand explanations.
Most black-box decisions in high-stakes domains share a recognisable structure. A proprietary model is trained on historical data, deployed via an API or embedded in government software, and its outputs are presented to decision-makers as objective scores. The decision-maker — a judge, a loan officer, a benefits caseworker — often treats the score as authoritative precisely because they cannot interrogate it.
The UK's A-level algorithm controversy of 2020 illustrates this clearly. Ofqual's standardisation model downgraded 39% of teacher-predicted grades, disproportionately harming students from state schools and disadvantaged backgrounds. The model was never published in advance. Students who received degraded grades had no access to the weighting formula that had effectively determined their university places.
Used in at least 10 US states. Scores range 1–10 on recidivism risk. Northpointe maintained trade-secret protection throughout litigation. A New Jersey court ruled in 2017 that defendants had no constitutional right to examine the source code.
Ofqual's model used school-level historical performance to adjust individual grades. Students at small schools were most affected because of insufficient historical data. Model was reversed within nine days after mass protests.
Opacity is not neutral. When a system cannot be examined, its errors cannot be identified, its biases cannot be corrected, and its accountability cannot be assigned. The COMPAS case shows that the absence of transparency is itself a policy choice — one that systematically disadvantages those who most need recourse.
You are examining three real-world scenarios where AI systems were used in high-stakes decisions. Use the chat below to work through each scenario with your AI tutor. Discuss: what information was withheld, who was harmed by that withholding, and what a minimally transparent alternative would look like.
When the European Union's General Data Protection Regulation came into force in May 2018, Article 22 gave individuals the right not to be subject to solely automated decisions with significant effects — and Article 13/14 required that individuals be given "meaningful information about the logic involved." Within weeks of GDPR enforcement, privacy advocates filed complaints against companies ranging from credit bureaus to social media platforms, testing whether "meaningful information" could be satisfied by boilerplate disclosures.
Meanwhile in the United States, Eric Loomis had appealed his Wisconsin sentence all the way to the state Supreme Court, arguing that using a proprietary COMPAS score violated his due-process rights. In 2016, the Wisconsin Supreme Court upheld the sentence, ruling that the score was one factor among many and that sufficient safeguards existed. The US Supreme Court declined to hear the case in 2017.
GDPR Article 22 is frequently cited as creating a "right to explanation," but its scope is narrower than commonly understood. It applies only to solely automated decisions — the moment a human reviews (even superficially) an algorithmic output, the protection may not apply. Recourse consists of requesting human review and contesting the decision, not necessarily of receiving a full algorithmic disclosure.
The EU AI Act, provisionally agreed in December 2023, goes further. High-risk AI systems — those used in credit scoring, employment, education, and criminal justice — must provide documentation sufficient for a "meaningful explanation" of outputs, and affected individuals must be notified that a high-risk AI system was used in decisions affecting them.
Wisconsin Supreme Court upholds COMPAS-informed sentencing. Rules proprietary protection of algorithm does not violate due process when the score is one factor among many.
Article 22 creates qualified right against solely automated decisions with significant effects. "Meaningful information about the logic involved" remains legally undefined.
UK Information Commissioner's Office publishes guidance stating that explanations must be "meaningful, contextual and tailored to the individual." Generic disclosures found insufficient.
High-risk AI systems must provide documentation and individual notification. Penalty regime includes fines up to 3% of global annual turnover for compliance failures.
The phrase "meaningful information about the logic involved" appears in GDPR but is not defined. Courts and regulators have struggled to specify what it requires in practice. A 2019 case involving credit insurer Allianz found that a statement saying "your application was assessed using an automated credit-scoring system" was not sufficient; affected individuals were entitled to know which factors were weighted and in what direction.
But full model disclosure creates its own problems. Publishing exact weights enables gaming: loan applicants who know that credit utilisation has a higher weight than account age can optimise superficially without improving creditworthiness. Regulators are therefore caught between the transparency needed for accountability and the opacity needed to prevent manipulation.
Research by the Alan Turing Institute (2019) found that the majority of GDPR Article 22 requests in the financial sector received responses that courts would likely consider inadequate — usually short paragraphs describing model type and input categories, without factor-level weighting or any individualised explanation.
GDPR Article 22 and the EU AI Act represent the strongest transparency mandates in the world — yet researchers still find them widely circumvented in practice. Use the chat to work through the gaps: What makes a transparency requirement meaningful? What makes it hollow? How would you write the law differently?
When Scott Lundberg and Su-In Lee published the SHAP (SHapley Additive exPlanations) paper at NeurIPS 2017, it rapidly became the dominant interpretability tool in industry. By 2020, the major US credit bureaus — Equifax, Experian, and TransUnion — had adopted SHAP or LIME to generate the "adverse action notices" required by the Fair Credit Reporting Act: the four reasons a credit application was declined.
In healthcare, researchers at the University of Washington used SHAP values to audit a commercial sepsis-prediction algorithm deployed at a major hospital network. They found that the model's top-weighted feature was the number of prior nursing notes — a proxy for administrative documentation intensity rather than clinical severity. The model was predicting which patients had more nurses, not which patients were sicker. SHAP made the problem visible; without it, the system would have continued operating invisibly.
LIME (Local Interpretable Model-agnostic Explanations), introduced by Ribeiro, Singh, and Guestrin in 2016, generates explanations by perturbing a specific input and observing how the model's output changes. It fits a simple interpretable model (typically a sparse linear model) in the local neighbourhood of the prediction, producing feature-importance values for that particular instance.
The key limitation is locality: LIME explains behaviour around a single point and can produce contradictory explanations for nearby points. A loan applicant with income of $45,000 and one of $46,000 may receive meaningfully different LIME explanations despite near-identical underlying model logic.
SHAP grounds feature importance in Shapley values from cooperative game theory: each feature is credited with the marginal contribution it makes to the prediction across all possible orderings of features. This produces globally consistent attributions — a feature's SHAP value means the same thing across the entire dataset.
SHAP satisfies three desirable axioms: efficiency (attributions sum to the prediction), symmetry (features with identical contributions receive identical values), and dummy (features with no effect receive zero attribution). TreeSHAP, the variant for gradient-boosted trees, runs in polynomial time and is fast enough for production use.
Strengths: Model-agnostic, fast, intuitive output. Limits: Locally inconsistent across nearby inputs; instability documented by Alvarez-Melis & Jaakkola (2018); perturbation distribution may not reflect real data manifold.
Strengths: Theoretically grounded, globally consistent, additive. Limits: Computationally expensive for neural networks; assumes feature independence in KernelSHAP; Shapley values can be counterintuitive for correlated features.
The University of Washington's audit found that "number of nursing notes" ranked as the model's highest SHAP-value feature — not clinical vitals or lab results. This reflected that sicker patients accumulate more documentation, making note-count a statistical proxy for severity. SHAP exposed a correlation the model was exploiting but that had no causal clinical meaning.
Post-hoc methods explain model behaviour, not model correctness. If a model has learned a spurious correlation — as in the sepsis case — SHAP faithfully reports that the model relies on it. The explanation is accurate but the underlying decision process is flawed. This distinction is critical: explainability and fairness are not the same property.
Rudin (2019) argues in "Stop Explaining Black Box Machine Learning Models for High Stakes Decisions" that post-hoc explanations are inherently untrustworthy for high-stakes use because they are approximations of behaviour that may diverge in precisely the cases that matter most. She advocates instead for inherently interpretable models — decision trees, logistic regression, scorecard models — whose logic is the prediction process itself.
You are a data scientist asked to add explainability to a credit-scoring model at a bank subject to GDPR. The model is a gradient-boosted tree. You must choose an explanation method, generate adverse action notices, and defend your choices to a regulator. Use the chat to work through the technical and ethical trade-offs.
In November 2021, New York City passed Local Law 144, the first US law requiring bias audits of automated employment decision tools. Companies using AI in hiring or promotion must commission independent bias audits and publish summary results before deploying the system. By January 2023, compliance had exposed a practical gap: several major vendors refused to publish audit results, claiming the requirement was preempted by federal law. The enforcement debate crystallised what transparency legislation alone cannot solve — you can mandate disclosure but not candour.
Amsterdam took a different approach. In 2020, the city government published an Algorithm Register — a public database of every algorithm used by municipal services, including its purpose, data sources, decision impact, and a risk classification. By 2023, over 90 algorithms had been documented. Rotterdam and Helsinki followed with similar registers. The European Commission cited Amsterdam's approach as a model in its AI Act implementation guidance.
Transparency is most tractable when it is designed in from the start, not retrofitted after deployment. Several principles have emerged from the research literature and from organisations that have done this well:
1. Use the simplest model that meets performance requirements. Logistic regression with calibrated probabilities is fully transparent. A gradient-boosted tree requires SHAP. A deep neural network may be practically inexplicable at the individual level. The choice of model is a transparency choice.
2. Maintain a model card. Introduced by Mitchell et al. at Google (2019), model cards document intended use, evaluation metrics disaggregated by subgroup, known limitations, and recommended deployment conditions. Several major organisations now publish them as standard.
3. Conduct pre-deployment disparate impact analysis. Before deployment, measure whether the model produces statistically significant differences in outcomes across protected groups. The legal standard in US employment law is the 4/5ths rule: if the selection rate for any group is less than 80% of the highest group's rate, disparate impact is indicated.
Amsterdam's Algorithm Register works because it treats algorithmic transparency as a public infrastructure problem, not a disclosure compliance exercise. Each entry includes not just what the algorithm does but who commissioned it, what decision it informs, whether humans review outputs, what appeals process exists, and when it was last audited. The register is searchable and machine-readable.
The practical lesson is that transparency requires standardised vocabulary. When every organisation can define "meaningful explanation" however it chooses, comparisons are impossible and accountability is undermined. Registers force a common schema.
Employers using automated employment decision tools in NYC hiring or promotion must: (1) commission an independent bias audit within one year of deployment, (2) make a summary of results publicly available on their website, (3) notify candidates and employees that an AEDT was used, and (4) provide, on request, the data categories the tool relies on. First civil penalties were issued in 2023.
Audits of deployed systems face a fundamental limitation: they measure the model's behaviour on the data it has seen. If the model is deployed on a population that differs from the audit sample — in demographic composition, economic conditions, or the tasks they perform — the audit findings may not apply. This is the distribution shift problem, and it is particularly acute in credit and hiring systems that operate in dynamic labour and financial markets.
A hiring algorithm audited against 2019 applicant data may not reflect the model's behaviour against 2024 applicants who have optimised their applications using AI-assisted tools. Transparency obligations should therefore include ongoing monitoring, not just pre-deployment audit — a requirement the EU AI Act addresses through its post-market monitoring obligations for high-risk systems.
Transparency is not a single act — it is a system of practices: choosing interpretable models where possible, documenting them with model cards, registering them publicly, auditing them before deployment, monitoring them in production, and providing individuals with meaningful recourse. Each element compensates for the gaps in the others.
A city government wants to deploy an algorithm to prioritise housing maintenance inspections in low-income neighbourhoods. The algorithm uses publicly available data: complaint history, building age, inspection records, and census demographics. You are the responsible AI lead. Design the transparency architecture for this system — audit, register entry, explanation approach, and appeals process.