In 2014, Amazon's machine learning team built a résumé-screening engine trained on a decade of successful hires. The historical data reflected a tech industry that had long skewed male. By 2015, internal audits revealed the model was systematically downgrading CVs that included the word "women's" — as in "women's chess club" — and penalising graduates of all-women colleges. The engineers who discovered this were staff inside the same company that deployed the tool. Amazon disbanded the team in 2018 and scrapped the system. The tool had been used, in some form, by recruiters in the interim.
No external regulator mandated the shutdown. An internal professional decision — by engineers willing to surface an uncomfortable audit result — ended the system's use. That decision carried career risk. It also prevented ongoing harm to an unknown number of candidates.
Machine learning systems trained on historical hiring data learn to replicate past decisions, including discriminatory ones. If an industry hired mostly men for engineering roles over a 10-year window, the training signal treats "maleness" as a latent predictor of success — not because of any explicit instruction, but because correlation is extracted regardless of causation.
This is not a hypothetical edge case. A 2019 audit by HireVue's own external reviewers found that facial-expression analysis in video interviews correlated candidate scores with lighting quality and webcam resolution — proxies for socioeconomic background, not competence. HireVue removed facial analysis from its products in 2021 following sustained pressure from the Electronic Privacy Information Center (EPIC) and a coalition of civil-rights organisations.
The pattern is consistent: automated hiring tools trained on biased historical data amplify past discrimination unless actively audited and corrected. The professional question is not whether this happens — it does — but what obligations a developer, deployer, or HR professional carries when they suspect or confirm it.
A 2020 study published in Science Advances (Lambrecht & Tucker) found that Facebook's ad-delivery algorithm showed software engineering job ads to 20% fewer women than men — not because of any advertiser instruction, but because the platform's optimisation engine found female users more expensive to reach in competitive advertising auctions. The discrimination emerged from the algorithm's cost-minimisation logic, not explicit bias.
Responsibility in automated hiring is distributed across a long chain: the data scientist who selects training data, the engineer who builds the scoring model, the product manager who decides which outputs to surface, the HR director who chooses to deploy the vendor's tool, and the recruiter who acts on the score. Each link in this chain can either amplify or interrupt discriminatory outcomes.
The key professional obligations that have emerged from documented cases include:
Audit before deployment. The Illinois Artificial Intelligence Video Interview Act (2019) requires employers using AI video-interview tools to notify candidates and conduct annual audits for racial and gender bias before expanding use — the first US law imposing pre-deployment bias auditing on hiring AI.
Document decisions. When the New York City Local Law 144 (2023) required bias audits of automated employment decision tools, it also required employers to post summary results publicly — making professional accountability visible rather than internal.
Maintain a human override. Best-practice frameworks from the EEOC, the UK's Equality and Human Rights Commission, and the EU AI Act all converge on one point: high-stakes employment decisions must retain a meaningful human review path that cannot be blocked by algorithm output alone.
The Amazon and HireVue cases share a structure: internal professionals identified the problem before external regulators did. The ethical moment is not the deployment decision — it is the point at which someone with knowledge of the problem chooses whether to act on it. Silence at that moment is itself a professional choice.
LinkedIn's 2019 Fairness-Aware Ranking paper, authored by its engineering team, describes how the platform adjusted its job-recommendation algorithm after discovering it over-represented male candidates for certain roles. The team introduced a fairness constraint into the ranking objective — accepting a slight reduction in click-through rate in exchange for more equitable exposure across genders. The paper was published openly, making the trade-off visible.
This represents the positive case: professionals who identify bias, document it, implement a correction with known trade-offs, and publish their reasoning. The approach was not perfect — fairness constraints in ranking remain technically contested — but it demonstrated that commercial pressure and ethical obligation are not always mutually exclusive when professionals choose to treat them as joint constraints.
You are an HR data analyst at a mid-size technology company. Your team has just deployed a third-party résumé-screening tool. After two months, a junior recruiter notices that only 9% of the candidates advancing past the first screen are women, despite women making up 38% of applicants. You have been asked to investigate and recommend next steps.
In October 2019, researchers Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan published findings in Science that a widely deployed commercial health risk algorithm — built by Optum and used by hospitals and insurers to identify patients needing complex care management — was systematically underestimating illness severity in Black patients relative to equally sick white patients. The model used health-care cost as a proxy for health need. Because structural racism in the US health system means Black patients historically incur lower costs for the same level of illness — due to documented barriers to access — the algorithm interpreted lower cost as lower need. The result: Black patients were assigned lower risk scores and were less likely to be enrolled in care-management programs. The disparity was large: at the same risk-score threshold, Black patients were significantly sicker than white patients on objective clinical measures.
Optum acknowledged the findings and said it was working to address them. But the tool had been used, in various versions, for years before the academic audit surfaced the problem. Hospitals and payers who licensed the tool had made care-allocation decisions based on its output without knowing — or without disclosing — that its proxy variable embedded structural inequity.
The Optum case is a textbook example of what researchers now call "outcome proxy failure": when a model is trained to predict a measurable proxy (cost, readmission, appointment compliance) rather than the underlying health need, it inherits all the structural biases embedded in that proxy. Health-care cost is shaped by access, insurance status, geography, historical discrimination in medical treatment, and patient trust — none of which reflect need. Training on cost as a proxy for need encodes all of those structural inequities into the scoring logic.
The problem is not unique to this vendor. A 2021 review in The Lancet Digital Health analysed 130 clinical AI studies and found that only 56% reported model performance by any demographic subgroup. Without subgroup analysis, disparate performance in minority populations is invisible until deployment reveals it — often after harm has already occurred at scale.
Obermeyer et al. estimated the Optum algorithm was influencing care decisions for approximately 200 million people across US hospitals and insurers at the time of publication. This is not a small pilot study — it represents routine algorithmic deployment at a scale that dwarfs most clinical trials of new drugs, which are held to strict bias-reporting requirements under FDA regulation.
The clinical AI context is unusual because it intersects two sets of professional duties: data-science ethics and medical ethics. Physicians and nurses who use algorithmic recommendations carry duties of non-maleficence and beneficence that do not disappear because the recommendation came from software. The question "should I act on this score?" remains a clinical judgment, and clinical judgment carries professional accountability.
Clinician disclosure obligations: The American Medical Association's 2019 policy on augmented intelligence states that physicians retain responsibility for patient care even when using AI tools, and that they should understand the limitations of tools they rely on. Relying on a tool known (or suspected) to perform differently across demographic groups, without disclosure to the patient, may constitute a violation of informed consent.
Vendor disclosure obligations: The FDA's 2021 Action Plan for AI/ML-Based Software as a Medical Device introduced the concept of a "predetermined change control plan" — vendors must pre-specify how their models will be monitored and updated post-deployment. This implicitly requires subgroup monitoring as part of post-market surveillance.
Institutional obligations: A 2021 joint statement by the Association of American Medical Colleges and the National Academy of Medicine called for hospitals deploying clinical AI to maintain standing bias-audit committees with authority to suspend tools pending investigation — analogous to the data-safety monitoring boards used in clinical trials.
In the Optum case, the ethical moment was not deployment — it was the point at which hospital administrators and vendor data scientists had sufficient data to suspect the disparity existed. The 2019 Science paper used data that vendors and hospital partners could, in principle, have analysed themselves. The professional question is: when does an obligation to look become an obligation to act on what you find?
One response to documented clinical AI bias is to treat it as a vendor problem — the tool was wrong, the vendor must fix it. But the Optum case illustrates that deploying institutions bear independent responsibility. The hospitals and payers that licensed the tool made a professional choice to rely on vendor-provided risk scores for high-stakes care allocation decisions. Outsourcing the algorithm does not outsource the obligation to understand it.
The parallel to financial services is instructive: when banks used third-party credit-scoring algorithms that produced discriminatorily disparate outcomes, regulators held the banks — not just the scoring vendors — liable under the Equal Credit Opportunity Act. The principle — that an institution cannot disclaim responsibility for a tool it has chosen to embed in a consequential decision process — is increasingly being applied to health care through the emerging regulatory frameworks described above.
You are a clinical informatics director at a regional hospital network. Your data team has just completed a subgroup analysis of your sepsis-prediction AI — a tool that has been in production for 18 months. The analysis shows the model's false-negative rate (missed sepsis cases) is 31% higher for Black patients than for white patients. The vendor says this is within acceptable aggregate performance benchmarks. Your CMO is asking whether you need to disclose this to affected patients, suspend the tool, or simply flag it for the next software update cycle.
PredPol — later rebranded Geolitica — was deployed by the Los Angeles Police Department beginning in 2012, and subsequently by dozens of US police departments. The system used historical crime report data to generate 500-square-foot "hotspot" boxes where officers were directed to patrol. Its proponents claimed it was race-neutral, using only crime type, crime location, and time as inputs — not demographic data.
A 2021 investigation by the Los Angeles Times and the Human Rights Data Analysis Group (HRDAG), using internal LAPD data obtained via public-records requests, found that PredPol directed patrols disproportionately to low-income communities of colour — not because those areas had objectively higher underlying crime rates, but because they had historically higher rates of police contact. More patrols produced more arrests, which produced more data, which produced more hotspot predictions. The feedback loop was self-reinforcing. The HRDAG analysis also found that drug-related offences — highly sensitive to where police actually patrol — were the strongest driver of hotspot designation in several precincts. LAPD announced it was ending its contract with PredPol in April 2020, several months before the formal analysis was published.
The central structural problem with predictive policing is that it creates a closed loop between prediction and data collection. In most machine-learning contexts, training data is collected independently of the model's predictions — a medical AI does not cause patients to develop the conditions it predicts. Policing is fundamentally different: deploying officers to a location based on an algorithmic prediction increases the probability of recording a crime in that location, because crimes go undetected without police presence.
This means the historical crime data used to train PredPol-style tools is not a neutral record of where crime occurs — it is a record of where police were previously deployed. Training a new model on that data does not learn where crime is; it learns where prior policing concentrated. The model then concentrates policing further in those areas, generating confirmatory data, and the loop tightens.
Santa Cruz, California became the first US city to ban predictive policing tools in June 2020, citing this structural feedback problem alongside civil-liberties concerns. The Santa Cruz resolution explicitly used the phrase "discriminatory feedback loop" in its legislative findings — a legal document formally acknowledging the algorithmic mechanism described above.
PredPol's developers consistently described it as race-neutral because demographic data was not a direct input. The HRDAG analysis demonstrated why this argument fails: historical police contact data is not race-neutral data. It is the product of decades of racially disparate enforcement policy. Using it as training data does not neutralise that history — it computes with it. Excluding race as a variable does not exclude racial disparity as an outcome when the input data encodes that disparity structurally.
Law enforcement officers and department administrators face a specific version of the professional dilemma: using an algorithm that may constitute a discriminatory practice, in a profession where explicit racial discrimination is illegal under the Equal Protection Clause and the pattern-or-practice provisions of 42 U.S.C. § 14141.
The officer's dilemma: An individual officer directed by PredPol to patrol a specific block faces a choice about how to conduct that patrol. Discretionary enforcement in that block — whom to stop, whom to question, whom to cite — carries its own bias risks. The algorithm amplifies the stakes of that individual-level discretion rather than eliminating it.
The administrator's dilemma: Police chiefs and city officials who procure predictive policing tools bear responsibility for understanding what "race-neutral inputs" actually means given the structure of the training data. The LAPD's 2020 termination of PredPol came after sustained pressure from civil-rights organisations and city council members — not from internal data review alone.
The data scientist's dilemma: PredPol's founders were academics who published their methodology and believed it would reduce crime. The 2021 investigative findings confronted them with evidence that their system was operating differently in practice than their model assumed. The professional obligation to engage with that evidence — rather than defend the original model design — is a central test of engineering ethics.
Chicago's Strategic Subject List (SSL), also known as the "heat list," assigned risk scores to individuals based on criminal history and social network analysis, purporting to predict who would be involved in gun violence. A 2017 RAND Corporation evaluation found no evidence the SSL reduced gun violence. An ACLU investigation found that Black and Latino men were disproportionately listed. Chicago terminated the program in 2020. This case illustrates a second failure mode beyond the feedback loop: predictive tools that generate racially disparate outputs without demonstrable efficacy — causing harm with no corresponding benefit.
Santa Cruz (2020), LAPD (2020), and Chicago (2020) all terminated predictive policing tools in the same year. The mechanism differed: Santa Cruz acted on principled grounds before a full local audit; LAPD acted under external pressure with a contract termination; Chicago acted after both a RAND efficacy evaluation and an ACLU investigation documented failure. Each path reached the same professional conclusion through a different accountability mechanism.
The lesson for professionals in adjacent fields — social services, child protective services, parole systems — is that the structure of feedback-loop bias is not unique to policing. Any algorithmic system deployed to allocate investigative or supervisory resources, trained on data reflecting historical over-surveillance of disadvantaged communities, risks replicating and amplifying those patterns. The obligation to ask "what does this training data actually represent?" precedes deployment, not follows it.
You are a senior analyst at a county child protective services agency. Your director wants to implement an algorithmic risk-scoring tool to prioritise which families receive home visits after a maltreatment report. The vendor claims the tool is validated on national data and race-neutral because it does not use race as an input. Your team's preliminary review suggests the tool's training data comes largely from counties with historically high CPS contact rates in low-income neighbourhoods.
In 2017, Google entered a contract with the US Department of Defense to help develop object-recognition algorithms for drone footage analysis under Project Maven. In early 2018, a group of Google employees learned the full scope of the project and began circulating an internal petition that eventually gathered over 4,000 signatures. The petition called on CEO Sundar Pichai to cancel the contract, arguing that "Google should not be in the weapons business." Several senior AI researchers resigned rather than continue work on the project.
Google did not renew the Maven contract when it expired in 2019. The company subsequently published its AI Principles, which explicitly prohibit applications "whose purpose contravenes widely accepted principles of international law and human rights" and weapons systems "that could cause or directly facilitate injury to people." The principles were a direct institutional response to the internal dissent. Whether the employees' action was whistleblowing, organised labour action, or principled professional refusal depends on how one categorises it — but its documented outcome was an institutional policy change driven by employee professional objection.
Professional dissent in AI development takes several forms, each with distinct legal protections and organisational consequences:
Internal objection: Raising concerns through recognised channels — manager, ethics review board, legal team. The lowest risk form of dissent. At Google, the initial petition was internal. This form is protected under general employment law if it relates to legal compliance but carries no specific whistleblower protection unless it involves a regulated domain (securities, health and safety, discrimination).
Collective action: Coordinating with colleagues to escalate objections — as in the Maven petition or the subsequent Google Walkout (2018, over sexual harassment policy). This is legally protected concerted activity under the National Labor Relations Act in the US, regardless of union membership. Employers cannot lawfully retaliate against employees organising collectively over workplace policy.
Public disclosure: Going to press or publishing externally. This carries the highest professional risk and the most variable legal protection. The Whistleblower Protection Act covers federal employees reporting government contract violations; private-sector whistleblower protections are sector-specific (Dodd-Frank for securities, OSHA for safety, etc.) and often have narrow scope.
The ethical framework underlying each form is distinct. Internal objection is a duty-based argument: I am obligated not to participate in a project I believe is unethical. Collective action is a rights-based argument: employees have rights to shape the conditions and purpose of their labour. Public disclosure is a consequentialist argument: the public harm from silence exceeds the personal and institutional harm from disclosure.
In December 2020, Google AI researcher Timnit Gebru was dismissed following a dispute over a research paper she co-authored examining risks of large language models, including environmental costs and bias in training data. Gebru said she was fired after pushing back on a demand to retract the paper before publication. Google said she resigned. The dispute prompted a second wave of employee protest and the resignation of several colleagues, including Margaret Mitchell, who led Google's AI ethics team. The episode documented the specific professional risk faced by AI researchers who produce findings that conflict with their employer's commercial interests — not a hypothetical, but a specific 2020 career outcome.
One lesson from both the Maven and Gebru cases is that the institutional placement of an ethics function determines how much power it carries. At Google in 2020, the AI ethics team lacked authority to block publication decisions or product deployments — it operated as an advisory function within an engineering organisation whose primary accountability was to product and revenue goals.
By contrast, the FDA's Center for Devices and Radiological Health has statutory authority to block the deployment of medical AI that fails pre-market review. The difference is not ethical seriousness — it is institutional design. Ethics functions without authority to say no are, structurally, communication exercises rather than governance mechanisms.
The EU AI Act (effective 2024–2026) attempts to impose this structural design requirement from outside organisations: for high-risk AI systems (employment, credit, law enforcement, migration, critical infrastructure), it requires a conformity assessment with a notified body, a human oversight mechanism with authority to intervene, and post-market monitoring — creating governance obligations that cannot be met by advisory-only ethics teams.
Each case in this module — Amazon's hiring AI, Optum's health risk scores, PredPol's patrol algorithm, Google Project Maven, and Timnit Gebru's dismissal — involves a professional who had knowledge of a potential harm before external regulators or the public did. In each case, the ethical moment was not the system's deployment. It was the moment a professional with relevant knowledge had to decide whether to act on it, stay silent, or actively suppress it. The question this module asks you to carry forward is simple: what is your default, and is it defensible?
Waiting for individual professionals to risk their careers is not a governance system — it is an externalisation of institutional responsibility onto individuals. Organisations that produce defensible AI outcomes typically have several structural features that reduce the individual cost of raising concerns:
Anonymous ethics reporting channels with a documented response protocol and non-retaliation policy (not just a stated commitment).
Pre-deployment review gates with documented criteria for what triggers additional ethical review — including fairness testing, subgroup analysis, and human-override requirements — before a product ships.
Cross-functional review authority, where an ethics or legal team can issue a formal hold that pauses deployment pending review, rather than offering an opinion that product managers can ignore.
Post-deployment monitoring obligations assigned to specific named roles, with defined escalation criteria and a board-level reporting line for AI risk — analogous to how financial institutions treat compliance risk.
None of these structures guarantee ethical outcomes. But they create the organisational conditions under which individual professionals can raise concerns without bearing the full personal cost of doing so — which is the prerequisite for consistent, rather than exceptional, ethical behaviour.
You are a senior engineer at a financial services company. Your team is six weeks from launching a credit-scoring model that uses rental payment history, utility bill data, and social graph signals as "alternative credit data" for applicants without traditional credit histories. Internal testing shows the model increases approvals for underserved populations — but also shows a 19% higher false-positive rate (incorrectly denied credit) for applicants in majority-Black zip codes. Your manager wants to launch on schedule. Legal has said the model passes disparate-impact testing under current regulatory thresholds. You believe the documented disparity is ethically unacceptable even if technically legal.