Horizon IT
Faulty accounting data, prosecutions, financial ruin, and years of disputed warnings involving sub-postmasters. The case demonstrates the danger of treating the system's output as more credible than repeated human reports.
Horizon IT · Robodebt · Toeslagenaffaire
When Systems Harm
Three major scandals show why technical performance, institutional oversight, and human welfare cannot be considered separately.
Three warnings from the real world
Chapter 3 of Algorithmic Duty of Care examines three cases in which automated or computer-supported systems operated within institutions that failed to protect people from severe consequences. The authors emphasize the human decisions surrounding the technology, not just the technology itself.
Robodebt
Automated welfare-debt practices led to wrongful demands and devastating consequences. The case exposes what can happen when administrative efficiency outruns legality, fairness, and meaningful review.
Childcare benefits scandal
Families were wrongly targeted in a benefits enforcement system. The scandal illustrates how institutional decisions, data practices, and inadequate safeguards can compound injustice.
People reported problems. The harm continued.
These stories are not simply tales of machines making mistakes. They are also about complaints, warnings, institutional denial, delayed correction, and the long human cost of leaving systems running after serious concerns arise.
A complaint should not be the end of a help-line interaction. It may be the beginning of evidence that something is wrong with the system itself.
A care-oriented design could help connect similar complaints across locations, preserve records, alert independent reviewers, and require documented responses. These are illustrative safeguards inspired by the book's preventive argument—not a claim that they were implemented in the three cases.
Why oversight alone is not enough
Mason and Hall describe the limitations of relying solely on “human guardrails.” Humans can fail to notice harm, refuse to act, or delay action. The authors therefore argue for care built into algorithms and data systems, working alongside human oversight and law.
That does not mean replacing accountability with automation. It means building an additional means of detecting and interrupting harm before it grows.