In November 2019, Apple launched its Goldman Sachs–backed credit card. Within weeks, software engineer David Heinemeier Hansson posted on Twitter that his Apple Card credit limit was 20× higher than his wife's — despite her having a better credit score and sharing assets. Steve Wozniak, co-founder of Apple, reported the same pattern with his wife. The New York Department of Financial Services opened a formal investigation. Goldman Sachs stated the algorithm did not use gender — but regulators found it used proxies that correlated with gender, producing discriminatory outcomes anyway.
The FICO score — introduced by Fair Isaac Corporation in 1989 — turned creditworthiness into a three-digit number computed entirely by an algorithm. Today approximately 90% of top U.S. lenders use FICO scores. The score weighs five categories: payment history (35%), amounts owed (30%), length of credit history (15%), new credit inquiries (10%), and credit mix (10%). These weights were set by analyzing historical lending data — which itself encoded decades of discriminatory lending practices.
In 2007, researchers at the National Consumer Law Center documented that African-American and Latino borrowers were systematically steered into subprime mortgages even when they qualified for prime rates. The algorithm was trained on patterns produced by human discrimination, so it reproduced those patterns at scale and with the veneer of mathematical objectivity.
Modern lenders go far beyond FICO. Upstart, a lending platform, incorporates over 1,600 variables including your education, your job title, and how long you spent filling out the application. ZestFinance has used social-media behavior as a proxy for creditworthiness. These systems are largely unregulated and their exact inputs are trade secrets.
In 2021, the Consumer Financial Protection Bureau (CFPB) warned that algorithmic credit models using "alternative data" — shopping patterns, phone usage, location — may systematically disadvantage protected classes. A 2022 University of California Berkeley study found that fintech mortgage algorithms still charged Black and Latino borrowers 11 basis points more than white borrowers with identical risk profiles.
The Equal Credit Opportunity Act (ECOA) requires lenders to give "specific reasons" for adverse actions — but regulators have struggled to apply this to black-box models. When Upstart's algorithm denies your loan, it may cite a factor like "insufficient credit experience" — a description that tells you nothing about which of its 1,600+ variables actually drove the decision.
In 2023, the CFPB issued guidance stating that "I relied on a complex algorithm" is not an acceptable reason to deny a loan applicant specific, actionable explanations. The guidance put lenders on notice that opacity is not a legal defense. But enforcement remains uneven, and the underlying models are still proprietary.
The European Union's AI Act, which began phasing in during 2024, classifies credit scoring as "high-risk AI" and requires human oversight, transparency, and the right to explanation. The United States has no equivalent federal law as of this writing.
Credit algorithms do not eliminate bias — they automate it, scale it, and make it harder to challenge. When the training data reflects historical discrimination, the model learns to discriminate. The mathematical form of the output doesn't make it neutral.
You're going to interrogate how credit scoring algorithms work, where their biases come from, and what "proxy discrimination" looks like in practice. Ask the AI about the Apple Card case, FICO's history, alternative data risks, or anything else from Lesson 1.
In 2013, Eric Loomis was arrested in La Crosse, Wisconsin, after being found driving a car used in a shooting. He pleaded guilty to charges and was sentenced to six years in prison. At sentencing, the judge cited his COMPAS risk score — a proprietary algorithm that had classified Loomis as "high risk." Loomis argued he had a constitutional right to know how the algorithm worked so he could challenge it. In 2016, the Wisconsin Supreme Court ruled against him, holding that COMPAS could be used at sentencing even though its formula was a trade secret and he could not inspect it.
COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) was developed by Equivant (formerly Northpointe). It generates risk scores based on a 137-question survey covering criminal history, age, education, employment, and — critically — neighborhood characteristics and family criminal history. The algorithm does not ask about race directly, but a 2016 investigation by ProPublica found that COMPAS was twice as likely to falsely flag Black defendants as high-risk compared to white defendants, and twice as likely to falsely label white defendants as low-risk.
ProPublica analyzed 7,000 criminal defendants in Broward County, Florida. Of defendants who were labeled high-risk but did not reoffend, 44.9% were Black and only 23.5% were white. The company disputed the methodology, and academics debated the statistical approach — but the underlying disparate outcomes were not refuted.
COMPAS is used in at least 17 U.S. states to inform decisions about bail, sentencing, and parole. In most jurisdictions, defendants are not told their score and have no formal mechanism to challenge the algorithm's output.
PredPol (now Geolitica) sold "predictive policing" software to over 150 U.S. police departments, claiming its algorithm could predict where crimes would occur. A 2021 investigation by The Markup found that the algorithm disproportionately directed police to low-income Black and Latino neighborhoods — not because those areas had more crime, but because they had historically received more police surveillance, creating a feedback loop. More policing produced more arrests, which produced more data pointing to more policing. Santa Cruz, California banned predictive policing in 2020; Los Angeles stopped using PredPol in 2020 after community protests.
Criminal justice algorithms share a structural flaw with predictive policing: they are trained on arrest and conviction records, not on actual crime rates. Because policing has historically been concentrated in certain communities, those communities have higher arrest records — not necessarily higher crime rates. An algorithm trained on arrests will predict more arrests in those communities, directing more police there, creating more arrests, and reinforcing its own predictions in an endless loop.
This was documented in a 2019 paper by researchers at Rutgers University, who found that predictive policing models amplify historical racial biases by approximately 2× to 3× in communities that were historically over-policed. The models are not predicting where crime will occur — they are predicting where police will find it if they look.
The Human Rights Watch, in a 2021 report, documented that COMPAS and similar tools are used in juvenile justice in states including Pennsylvania and Ohio, meaning teenagers as young as 12 may have their futures shaped by algorithmic risk assessments they cannot see or challenge.
Criminal justice algorithms make decisions about freedom and incarceration based on data that encodes historical policing patterns. They are shielded from challenge by trade secret protections. The people most affected — often the poorest and least powerful — have the fewest tools to contest them.
You're going to work through the ethical and legal problems with using algorithms in criminal justice decisions. Consider the COMPAS case, predictive policing feedback loops, or the trade secret problem. Push the AI on the hardest questions.
Between 2014 and 2017, Amazon built and then quietly abandoned an AI recruiting tool it had hoped would automate resume screening. The system was trained on 10 years of Amazon's hiring decisions. Researchers discovered it had learned to penalize resumes that contained the word "women's" — as in "women's chess club" or "women's college" — and downgraded graduates of all-women's colleges. Because Amazon's historical hiring had been male-dominated, the algorithm concluded that male was the correct profile for a successful candidate. Amazon disbanded the project in 2017 without ever deploying it in final hiring decisions, but the episode became one of the most cited examples of algorithmic bias in hiring.
Applicant Tracking Systems (ATS) are used by an estimated 98% of Fortune 500 companies and up to 75% of all employers. These systems parse and rank resumes before any human sees them. Studies by Harvard Business School and Accenture (2021) found that ATS systems were filtering out millions of qualified candidates — people with non-linear career paths, those who had taken time off, or those with skills described in non-standard terminology. The report called this the "hidden workers" problem: an estimated 27 million Americans were systematically screened out by algorithms before any human could evaluate them.
A 2019 study from the National Bureau of Economic Research found that when companies adopted ATS software, the probability that a job posting received an application from an older worker dropped significantly. The study found that ATS keyword optimization created what amounted to indirect age discrimination: job descriptions written for ATS parsing used tech-industry vocabulary that was systematically less familiar to workers over 50.
HireVue serves over 700 companies including Unilever, Delta, and Goldman Sachs. Its platform records one-way video interviews and, until 2021, used AI to analyze facial expressions, word choice, and tone of voice to generate a hirability score. In 2019, the Electronic Privacy Information Center (EPIC) filed a complaint with the FTC arguing the practice was unfair and deceptive. HireVue announced in 2021 that it would stop using facial analysis — but continued analyzing voice patterns and linguistic content. Critics note that "voice analysis" for personality inference has no validated scientific basis.
Amazon's warehouse workers are managed largely by algorithm. The system tracks "time off task" — any period when a worker is not scanning items — and can automatically generate termination notices without human review. A 2019 investigation by The Verge obtained internal Amazon documents showing that hundreds of workers had been fired by algorithm for productivity failures, often without any human ever reviewing their cases. Workers described being fired for taking bathroom breaks or for slowing down when they were injured.
Gig economy platforms operate similarly. Uber drivers' accounts can be deactivated — effectively terminating their income — by algorithm, based on low passenger ratings. A 2021 UK Supreme Court ruling found that Uber drivers were legally "workers" and entitled to labor protections. The ruling noted that the algorithm effectively controlled the terms of employment, substituting for the management decisions a traditional employer would make.
The Equal Employment Opportunity Commission (EEOC) issued guidance in 2023 warning employers that algorithmic employment tools can violate the Americans with Disabilities Act if they screen out qualified candidates with disabilities or penalize reasonable accommodations like non-standard work patterns.
Algorithms now control both entry to employment and conditions within it. They filter who gets interviews, score how you perform in them, monitor you on the job, and can terminate you — all with limited human oversight and limited recourse for affected workers.
You're going to investigate the specific ways employment algorithms create bias and limit opportunity. Ask about Amazon's recruiting AI, ATS hidden worker problems, HireVue's video analysis, or algorithmic management of warehouse and gig workers.
In November 2023, a class-action lawsuit was filed against UnitedHealth Group alleging that its subsidiary NaviMedix used an AI algorithm called nH Predict to systematically deny post-acute care claims for elderly patients. The lawsuit alleged the algorithm had a documented 90% error rate — meaning 9 out of 10 patients who appealed their AI-generated denials won on appeal. The plaintiffs alleged that UnitedHealth knew of the error rate but continued using the algorithm because most denied patients did not appeal, making algorithmic denial financially profitable. At least two named plaintiffs died shortly after their care was cut off.
The UnitedHealth case was not isolated. A 2023 investigation by STAT News and ProPublica found that multiple major insurers — including Cigna, Humana, and Aetna — were using AI systems to automate denial of prior authorization requests. In one documented case, Cigna's PXDX system allowed physicians to deny claims at a rate of 50 claims per day per doctor, spending an average of 1.2 seconds per claim, without opening any patient file. The process was described as "assembly-line medical care" driven by algorithmic recommendations that physicians rubber-stamped.
A U.S. Senate investigation published in October 2023 found that three of the largest Medicare Advantage insurers — UnitedHealthcare, Humana, and CVS/Aetna — had denial rates as high as 18%, with AI playing a significant role. The Centers for Medicare & Medicaid Services (CMS) issued a memo warning insurers that coverage decisions must be made on an individual basis and that algorithmic denial of entire categories of care was potentially illegal under Medicare Advantage rules.
In 2016, Arkansas replaced its manual Medicaid home care assessments with an algorithm. Thousands of patients immediately had their hours of personal care reduced — often by 30–50% — without explanation. When patients and advocates investigated, they found the algorithm used an illogical formula: patients with diabetes or ulcers actually received fewer care hours than before because the formula's inputs were misconfigured. In Ledgerwood v. Jeger (8th Circuit, 2019), courts found patients' due process rights had been violated because they were never told how the algorithm worked or how to challenge it. Idaho faced a nearly identical problem with its Medicaid algorithm and similar legal consequences.
At least 30 U.S. states use algorithmic risk assessment tools in child welfare decisions — to predict which families are at risk of child abuse or neglect, and to guide decisions about intervention, removal of children, and family services allocation. The most studied is the Allegheny Family Screening Tool (AFST) used in Allegheny County, Pennsylvania.
A 2019 study by Virginia Eubanks, published in her book Automating Inequality, found that AFST and similar tools systematically score poor families and families receiving public assistance as higher-risk — not because poverty causes abuse, but because poor families are more visible to government systems. Families receiving food stamps, Medicaid, or housing assistance generate more data points in government databases, making them algorithmically legible in ways that wealthier families who access private services are not. The algorithm scores what it can measure, not what actually predicts harm.
An independent audit of the AFST commissioned by Allegheny County in 2022 found that Black children were referred to the tool at 2.3× the rate of white children, and that the tool had limited ability to predict actual harm — performing only marginally better than a random baseline in some outcome categories.
Algorithms are now rationing healthcare, cutting benefits, and making child welfare interventions — decisions where errors cost lives. The financial incentive structure rewards algorithmic denial: most people don't appeal, making mistakes profitable. Without accountability, transparency, and meaningful appeal rights, algorithmic decision-making in high-stakes human services is a mechanism for harm at scale.
You'll examine the real-world consequences of algorithms making healthcare and benefits decisions. Ask about the UnitedHealth case, insurance prior authorization AI, Arkansas's Medicaid algorithm, or child welfare tools. Consider accountability, incentives, and what reform would actually look like.