At 9:58 p.m. on March 18, 2018, a self-driving Uber Volvo SUV struck and killed Elaine Herzberg as she walked her bicycle across Mill Avenue. The vehicle's perception system had detected her 5.6 seconds before impact and had reclassified her as a vehicle, then a bicycle, then a vehicle again — cycling through labels without triggering an emergency stop. The safety driver, Rafaela Vasquez, was monitoring a video stream on her personal phone. She looked up less than a second before impact.
The National Transportation Safety Board's 2019 report identified the root cause not as a sensor failure but as a trust calibration failure. Uber's system had disabled Volvo's built-in emergency braking. Vasquez had been trained to trust the automation. The company had deployed the vehicle in a mode that produced frequent spurious braking alerts — so engineers had suppressed the alert system entirely to reduce false positives. Every choice made trust easier. None of them made the system safer.
Trust in AI systems is not a single variable. Researchers distinguish at minimum three separable dimensions: competence trust (does the system do what it claims?), integrity trust (does it operate within agreed constraints?), and benevolence trust (does it act in the user's interest rather than against it?). These dimensions can diverge sharply — a system can be highly competent while having low integrity, as when a recommendation algorithm maximizes engagement at the expense of user wellbeing.
A fourth dimension, increasingly important in deployed systems, is predictability: users extend more trust to systems whose failures they can anticipate. This is why pilots who understand the failure modes of autopilot systems maintain safer manual reversion than pilots who merely trust the system to work.
In May 2016, ProPublica published an analysis of COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), a risk-scoring tool used by judges in Broward County, Florida and hundreds of other jurisdictions. The algorithm assigned defendants a "recidivism risk" score of 1–10. ProPublica found that Black defendants were nearly twice as likely as white defendants to be falsely flagged as high risk, while white defendants were more likely to be incorrectly labeled low risk when they later reoffended.
The failure was not purely algorithmic — it was a trust design failure. Judges were given a numeric score without adequate disclosure of the model's training data, its error rate distribution across demographic groups, or the fact that the score's predictive accuracy for violent recidivism was only marginally better than chance. The interface communicated false precision. Judicial users had no mechanism for appropriate skepticism. The system had been designed to be trusted rather than designed to be understood.
Interfaces that present AI outputs as crisp numbers or binary recommendations without surfacing uncertainty, training limitations, or known failure modes are not neutral — they actively manufacture over-trust. Designing for calibrated trust requires building skepticism affordances directly into the interface.
Lee and See's foundational 2004 framework in Human Factors described trust calibration as requiring three things: that the system's actual reliability be knowable, that the interface convey that reliability accurately, and that users have sufficient cognitive resources to integrate this information. In practice, AI systems routinely fail all three conditions. Reliability is difficult to measure because it varies across input distributions. Interfaces typically display confidence scores calibrated on training data that may not match deployment contexts. And users operating under time pressure default to heuristic trust — they trust whatever worked last time.
Contemporary research by Bansal et al. (2021, "Does the Whole Exceed Its Parts?") demonstrated that AI explanations improve human-AI team accuracy only when they help users identify when the AI is wrong — not when they simply increase understanding of how the AI works. This is a crucial distinction: transparency that explains correct outputs adds no value; transparency that reveals error signatures is what enables calibration.
The goal of trust design is not maximum trust — it is accurate trust. A user who trusts an AI system at 85% reliability and rejects its outputs 15% of the time in the right situations outperforms both the user who always trusts and the user who never trusts.
You're a design reviewer evaluating AI systems for trust calibration problems. Discuss real or hypothetical AI interface scenarios with the assistant — identify whether each case represents over-trust, under-trust, or miscalibration, and propose specific design interventions.
Article 22 of the EU's General Data Protection Regulation, which took effect in May 2018, granted individuals a right "not to be subject to a decision based solely on automated processing" that significantly affects them — and, in contested readings, a right to "meaningful information about the logic involved." Within months, legal scholars and computer scientists were publicly disagreeing about what this meant. Could a GDPR-compliant explanation be a post-hoc rationalization of a black-box decision? Did "meaningful information" require technical accuracy — or was it enough to give a reason that felt meaningful to the user, even if it didn't reflect the model's actual computation?
The resulting implementations diverged wildly. Some companies provided genuine feature importance scores. Others produced what Wachter, Mittelstadt and Russell called "legally compliant but informationally vacuous" outputs: sentences like "You were declined because your financial profile did not meet our criteria." The regulation created a market for the appearance of explainability without creating a standard for its substance.
The most consequential finding from a decade of explainable AI (XAI) research is that explanations do not straightforwardly transfer understanding. Chromik and Butz (2021) surveyed 54 XAI user studies and found that the majority measured user satisfaction with explanations, not whether users had gained actionable understanding of the model's behavior. User satisfaction is a weak proxy — users rate confident, fluent explanations as more helpful regardless of whether those explanations are accurate.
The phenomenon of explanation-induced over-trust is particularly well-documented in medical AI. Cai et al. (2019) found that physicians shown AI diagnostic outputs with accompanying explanations accepted more AI errors than physicians shown outputs without explanations. The explanation created a sense of auditability that suppressed independent clinical judgment. The researchers called this the "explanation paradox": the explanation was doing the opposite of what it was designed to do.
A 2019 study by Alvarez-Melis and Jaakkola demonstrated that LIME-generated explanations for nearly identical inputs could differ substantially — producing different feature rankings for inputs that differed by a single pixel. If a user could generate a different explanation by slightly rephrasing their query, what does the explanation actually represent? Ghassemi, Oakden-Rayner and Beam (2021) made this point forcefully in The Lancet Digital Health, arguing that current XAI methods are insufficient for clinical deployment not because they are technically flawed but because they provide a false sense of auditability — they perform transparency without producing it.
This has direct interface design consequences. An interface that presents a LIME explanation without conveying its inherent instability is making an implicit claim about the explanation's reliability that the underlying method cannot support. Designing responsibly requires either not presenting such explanations, or designing interfaces that make their limitations legible — for example, showing confidence ranges around feature importance scores or providing comparison explanations across similar inputs.
Multiple studies of deep learning skin cancer classifiers (notably Narla et al., 2018, and Winkler et al., 2019) found that models achieved high accuracy partly by learning to associate surgical ruler markings and colored calibration patches with malignancy — a spurious correlation present in training data. When saliency map explanations were generated, they highlighted clinically irrelevant regions. Dermatologists shown these explanations could not detect the spurious correlation. The explanation system was functioning correctly; it was accurately depicting what the model had learned. The problem was that users had no reason to distrust it.
The research suggests several design principles that diverge from common practice. First, explanations should target decision support, not comprehension — users need to know what to do with an AI output, not how the model computed it. Second, explanations should be contrastive: "the system predicted X rather than Y because of Z" is more actionable than "the system predicted X because of Z." Third, explanations should be honest about their own limitations — an explanation that says "this feature was identified as important, but this explanation has a 30% probability of being unstable for similar inputs" is more trustworthy than a confident feature list.
The EU AI Act of 2024, which goes further than GDPR in addressing transparency requirements for high-risk AI systems, requires that AI systems be designed with "appropriate human oversight measures" rather than simply providing explanations. This framing — oversight rather than explanation — reflects a growing consensus that what users need is not understanding of the model but the ability to catch and correct model errors.
The question to ask of any AI explanation is not "does this help the user understand the model?" but "does this help the user catch the model's mistakes?" These are different design targets, and most existing XAI approaches optimize for the former while claiming to achieve the latter.
You are critiquing explanation designs for high-stakes AI systems. For each scenario you discuss, analyze whether the explanation approach enables users to catch errors — or merely creates the appearance of transparency. Propose specific redesigns.
IBM's Watson for Oncology was deployed in hospitals across the United States, India, South Korea, Thailand, and elsewhere between 2015 and 2018, promoted as a tool that could recommend cancer treatments aligned with leading oncologists' judgment. In 2018, STAT News reported on internal IBM documents showing that Watson had recommended "unsafe and incorrect" treatment options, including suggesting a patient with severe bleeding be given a drug that could worsen hemorrhaging. An IBM slide deck from 2017 described Watson's recommendations as "sometimes wrong" and noted that the system had been trained on a limited set of hypothetical patient cases rather than real clinical data.
Multiple hospitals quietly discontinued use. The University of Texas MD Anderson Cancer Center ended its Watson contract after spending $62 million. The Memorial Sloan Kettering collaboration, which had trained Watson's recommendations, was revealed to have an undisclosed financial relationship with IBM. What the case illustrated was not simply bad AI — it was the deployment of AI in an interface that provided no affordances for appropriate skepticism. Physicians were given recommendations without access to the reasoning, the training data's limitations, or the confidence distribution across different cancer types where Watson's accuracy varied enormously.
The theoretical basis for human-AI teaming is complementarity: humans and AI systems have different error profiles, and a well-designed collaboration should outperform either alone. AI systems tend to be consistent across high-volume repetitive judgments but fail on distributional shifts, rare cases, and inputs far from training data. Humans are inconsistent across repetition (a well-documented finding in clinical and legal judgment studies) but often better at reasoning about novel situations, ethical edge cases, and contextual factors that fall outside training distributions.
The practical challenge is building interfaces that activate this complementarity. Dietvorst, Logg and colleagues showed in a series of studies (2015–2018) that humans frequently abandon algorithms after observing a single error — even when the algorithm significantly outperforms human judgment in aggregate. This is called algorithm aversion. Counterintuitively, when users were allowed to slightly modify algorithm outputs, aversion decreased dramatically — not because the modifications improved accuracy but because users' sense of agency reduced their psychological need to distance themselves from the system after errors.
Aviation's transition from manual flight to increasing automation over 50 years provides the most thoroughly studied natural experiment in appropriate reliance design. Following a series of accidents in the 1970s and 1980s caused by automation over-reliance — notably Air France Flight 447 (2009), where pilots responded incorrectly to an autopilot disconnect because they had lost situational awareness — aviation human factors researchers developed a set of design principles that have broadly influenced AI interface design.
The key principle is mode awareness: the interface must make the current automation state, its constraints, and the conditions under which it will disengage legible to the operator at all times. Mode confusion — where pilots were unaware of which automation mode was active — was a contributing factor in multiple fatal accidents. Applied to AI systems, this principle argues that interfaces should continuously communicate not just what the AI recommends but what kind of task the AI is currently performing, what its confidence level is, and under what conditions it would expect to be wrong.
Google Maps' navigation system displays different visual treatments for high-confidence routing (a solid blue line) and lower-confidence estimates (a dotted or grayed display during signal loss). This is a simple implementation of confidence-qualified display — the interface communicates not just a recommendation but the epistemic status of that recommendation. The same principle, applied to medical AI or legal risk tools, would require displaying not just a risk score but a clear indication of the confidence range and the conditions under which the score is most likely to be inaccurate.
One of the most consequential design decisions in human-AI systems is how the interface handles human disagreement with AI recommendations. Systems that make override easy tend toward under-reliance; systems that make override difficult or invisible tend toward over-reliance. The optimal design, empirically, appears to be structured override: the ability to override is prominent and requires minimal friction, but the interface asks users to briefly categorize their reason for overriding.
This approach has been tested in radiology AI deployment at Massachusetts General Hospital and in ICU alarm systems at Emory University. In both cases, requiring clinicians to briefly indicate why they were overriding the AI served two functions: it modestly slowed impulsive overrides without preventing deliberate ones, and it generated a dataset of override reasons that could be used to improve the model's failure modes. The override mechanism was simultaneously a user support tool and a system improvement mechanism.
The interface should make it slightly easier to scrutinize and slightly harder to blindly accept an AI recommendation than it is to blindly reject it. Most current interfaces invert this: they present AI outputs in ways that require active effort to question, while making acceptance the path of least resistance.
You're a UX designer asked to redesign an AI-assisted decision-making interface to better support appropriate reliance. Work through specific design choices with the assistant — what information to show, how to display confidence, how to structure overrides, how to prevent both over-trust and algorithm aversion.
Between 2018 and 2022, Facebook's automated content moderation systems made approximately 100 billion content decisions per year. The individual transparency design was consistent: when a post was removed, users received a notification explaining the violated policy category ("This post goes against our Community Standards on hate speech"). Appeals were possible through an automated review queue. This satisfied the surface requirements of individual transparency.
What it concealed was the system's aggregate behavior. Frances Haugen, a former Facebook product manager, released internal documents in October 2021 showing that the company's own research found its systems consistently under-moderated harmful content in non-English languages, disproportionately flagged content from political conservatives in the United States (producing political controversy over the system's intent), and that the appeals process resolved fewer than 1% of appeals in favor of users. Individual transparency — telling each user why their specific post was removed — coexisted with near-complete opacity about how the system behaved across populations, which communities it failed, and what error rates were acceptable.
Zerilli et al. (2019) distinguish between local transparency (explanation of a specific decision to the individual affected) and global transparency (disclosure of how a system behaves across populations). These require different design approaches. Local transparency is an interface design problem: how do you communicate this decision to this person? Global transparency is an institutional and regulatory design problem: what aggregate performance data should be disclosed, to whom, in what form, and how often?
Current practice is heavily weighted toward local transparency. Privacy laws like GDPR and CCPA create rights to individual explanation. But individual explanations can be accurate about a specific case while concealing systemic failures — COMPAS could accurately tell an individual defendant "your score was 7 because of factors X, Y, Z" while the system's differential false positive rates across racial groups remained undisclosed. The explanation is locally accurate but systemically misleading.
The EU AI Act (2024) creates a tiered transparency regime based on risk level. High-risk AI systems — defined to include systems used in employment, credit, education, criminal justice, and critical infrastructure — must maintain technical documentation, keep logs of system operation, provide information sufficient for human oversight, and in some categories undergo conformity assessment before deployment. General-purpose AI systems above a compute threshold must publish summaries of training data and comply with copyright obligations.
Notably, the Act's transparency requirements are largely directed at providers and deployers — not at the affected individuals. The Act creates obligations to document and disclose to regulators and downstream users (businesses deploying the AI), while individual transparency remains governed by GDPR. This creates a layered transparency architecture in which regulatory transparency (disclosure to oversight bodies), market transparency (disclosure to business customers), and user transparency (disclosure to individuals) are legally distinct with different requirements.
Palantir Technologies' predictive policing software was deployed in New Orleans from 2012 to 2018 under a secret contract disclosed publicly only through a 2018 Veritas investigation. The city's residents — and their elected representatives — had no knowledge of the system's existence during this period. This represents a transparency failure at the institutional level: neither the individual affected (a person placed on a watch list) nor the community governed by the system had access to information about it. When transparency operates only at the individual level, it is possible to maintain complete opacity about whether a system exists at all.
The practical design challenge of institutional transparency involves three components that are distinct from individual explanation design. First, audit trails: systems must log decisions in forms that support retrospective analysis of aggregate behavior. This is a data architecture problem that must be built into systems from the start, not added later. Second, disclosure mechanisms: there must be channels through which aggregate performance data reaches appropriate audiences — regulators, civil society, affected communities — not just the organizations that deploy the system. Third, contestability structures: there must be mechanisms for affected communities to challenge not just individual decisions but the deployment of the system itself.
Amsterdam and Helsinki's AI registers — public databases of AI systems used in municipal governance — represent one approach to institutional transparency design. Published since 2020, these registers require city departments to disclose the purpose, data sources, vendor, and known limitations of AI systems they deploy. The registers are publicly accessible, creating a form of systemic transparency that individual explanation mechanisms cannot provide. By 2023, Amsterdam's register listed over 50 active AI applications used in municipal services.
Individual transparency is necessary but not sufficient. Systems that affect large populations create obligations of institutional transparency that require audit trail architecture, disclosure mechanisms, and contestability structures — none of which can be satisfied by improving individual explanation interfaces alone.
You're advising a city, healthcare system, or large organization on designing institutional transparency for an AI system that will affect large numbers of people. Go beyond individual explanation interfaces to design the full transparency architecture: what gets logged, what gets disclosed, to whom, in what form, and how affected communities can contest the system.