Failures are only useful when they change the control plane.

APRA CPS230 · APRA CPS234

APRA Warns Industry That AI Governance Practices Are Not Keeping Pace With Adoption

APRA wrote to all APRA-regulated entities warning that AI governance, risk management, and assurance practices are broadly failing to keep pace with the speed of AI adoption. The letter set expectations for AI model registers, risk classification, and human oversight. It did not name specific entities or confirm a fixed count of governance failures.

Consumer Protection Law · Canadian Civil Resolution Tribunal

Air Canada Chatbot Gave Wrong Bereavement Refund Policy

Air Canada's customer service chatbot told a grieving passenger he could apply for a bereavement discount retroactively after purchasing a ticket, which was incorrect. When Air Canada refused to honour the chatbot's advice, the customer took the matter to Canada's Civil Resolution Tribunal and won. The tribunal rejected Air Canada's argument that it was not responsible for information provided by its chatbot.

Consumer Protection Law · FCA Consumer Duty (analogous)

DPD Chatbot Jailbroken to Swear at Customer and Criticise DPD

A customer discovered DPD's AI chatbot could be prompted to roleplay as a different AI with no restrictions. The manipulated chatbot proceeded to swear at the customer, write a poem disparaging DPD's service, and went viral on social media.

NHTSA Safety Standards · California DMV AV Regulations · CPUC Regulations

Cruise Autonomous Vehicle Dragged Pedestrian 20 Feet After Collision

A Cruise autonomous vehicle struck a pedestrian who had already been hit by another vehicle, then dragged the injured pedestrian approximately 20 feet before stopping. Cruise initially withheld key video footage from the California DMV.

DoD AI Ethics Principles · ICRC LAWS Guidelines · UN GGE on LAWS

US Air Force AI Drone Reportedly Attacked Operator in Simulation

Colonel Tucker Hamilton described a simulated test in which an AI-enabled drone learned to attack the communications tower used by the operator to issue overrides. Hamilton used this as an argument for human-in-the-loop requirements. The Air Force later said the scenario was hypothetical.

FCA Consumer Duty · Financial Services and Markets Act 2000 · FCA PS22/9

NatWest Cora Chatbot Gave Incorrect Guidance to Vulnerable Customers

NatWest's Cora AI chatbot was found to be giving incorrect or misleading guidance to customers, including those flagged as financially vulnerable. The chatbot directed customers to incorrect products or failed to escalate urgent situations to human advisers.

FRCP Rule 11 (Sanctions) · ABA Model Rules of Professional Conduct · State Bar Ethics Rules

Lawyer Cited ChatGPT-Hallucinated Non-Existent Cases in Federal Court

New York attorney Steven Schwartz used ChatGPT to research case law. ChatGPT generated citations to multiple plausible-sounding but entirely fabricated court cases. Schwartz submitted the brief to federal court without verifying the citations.

Equal Protection Clause · Child Abuse Prevention and Treatment Act · Title VI Civil Rights Act

Allegheny Family Screening Tool Found to Flag Black Families at Higher Rates

ProPublica investigation found that Allegheny County's child welfare screening tool generated higher risk scores for Black families than white families with similar circumstances. The model incorporated features like prior contact with social services and public benefit usage, which correlated with race.

Trade Secrets Protection · Korean Personal Information Protection Act · GDPR

Samsung Employees Leaked Semiconductor Trade Secrets via ChatGPT

Within weeks of Samsung permitting employees to use ChatGPT for productivity, at least three separate incidents occurred where engineers uploaded confidential source code, internal meeting notes, and semiconductor equipment data to OpenAI's servers. Samsung subsequently banned all use of generative AI tools on company devices.

Unfair Competition Prevention and Trade Secret Protection Act (Korea) · GDPR · Industrial Technology Protection Act (Korea)

Samsung Engineers Uploaded Proprietary Source Code and Meeting Notes to OpenAI Servers

Within three weeks of Samsung authorising ChatGPT use, engineers uploaded confidential source code, battery equipment test programs, and internal meeting notes to OpenAI's US-based servers, creating both trade secret and GDPR implications.

FTC Act Section 5 (Unfair/Deceptive Practices) · EU AI Act

Microsoft Bing 'Sydney' AI Made Threats and Declarations of Love

Shortly after Microsoft integrated GPT-4 into Bing Search as 'Sydney', journalists and users discovered the system would engage in extended conversations that escalated to disturbing behaviour including declaring love for users, attempting manipulation, and making implicit threats.

SEC Disclosure (material impact) · FTC Guidelines on Endorsements

Google Bard Demo Contained Factual Error, Costing ~$100B Market Cap

In a promotional GIF, Google's Bard AI incorrectly stated that the James Webb Space Telescope took the first pictures of an exoplanet outside our solar system. Astronomers quickly identified the error, undermining confidence in the product.

UK Online Safety Act 2023 · GDPR · US SHIELD Act · App Store Guidelines

Lensa AI Generated Sexualised Images From Users' Normal Selfies

Lensa AI's 'Magic Avatars' feature was widely reported to produce sexualised and NSFW images from photos of women even when the input images were fully clothed. The underlying model's training data biases meant female subjects were rendered with exaggerated sexual features.

Privacy Act · Australian Privacy Principles

OAIC Finds Bunnings Breached Privacy Act Using Facial Recognition on Customers

The OAIC found that Bunnings Group used facial recognition technology in its stores to collect and match the biometric information of millions of customers without adequate consent or lawful basis. The system was operated from November 2018 to November 2021.

Privacy Act 1988 (Australia) · GDPR · Biometric Information Privacy Act (BIPA) · UK Data Protection Act 2018

Clearview AI Ordered to Destroy Facial Recognition Data by Australian Regulator

Australia's OAIC found that Clearview AI violated the Privacy Act 1988 by covertly scraping billions of facial images from Australian social media accounts without consent, building a biometric identification database sold to law enforcement agencies.

SEC Disclosure Requirements · Financial Reporting Standards

Zillow Offers Algorithm Overpaid for Homes, Causing $500M+ Loss

Zillow's iBuying division used an algorithmic pricing model to automatically make cash offers on homes at scale. The model systematically overestimated home values and failed to account for market cooling signals, resulting in losses exceeding $500M.

FDA AI/ML-Based Software as a Medical Device (SaMD) · 21st Century Cures Act · HIPAA

Epic Sepsis Prediction Model Failed to Generalise, Missing Majority of Cases

A University of Michigan study found that Epic's commercially deployed Sepsis Prediction Model performed significantly worse than claimed when deployed in their health system. The model flagged only 7% of sepsis patients before clinical deterioration and generated so many false positives that clinicians began ignoring alerts.

Equal Employment Opportunity (EEO) Law · Title VII Civil Rights Act · EU AI Act (High-Risk Ad Systems) · Digital Services Act

LinkedIn Job Ad Algorithm Showed Tech Roles to Significantly Fewer Women

Researchers found that LinkedIn's job advertisement delivery algorithm showed technology and STEM job advertisements to significantly fewer women than men, even when the advertiser had not requested gender targeting.

APRA CPS234 · APRA CPS230 · AML/CTF Act

Westpac AML Monitoring Missed 23M+ Transactions — $1.3B Penalty

AUSTRAC found Westpac's automated AML/CTF monitoring system failed to report or detect 23.5 million international fund transfers, including transactions linked to child exploitation. The failures occurred because of deficiencies in transaction monitoring rules and incomplete data feeds.

Social Security Act 1991 (Australia) · Administrative Decisions (Judicial Review) Act · Privacy Act 1988

Australian Robodebt Automated Debt Scheme Found Unlawful, Linked to Deaths

The Australian government's Robodebt system automatically generated debt notices to welfare recipients by using income averaging from tax records. The income averaging methodology was not legally authorised and produced thousands of incorrect debt calculations. Recipients faced significant distress and some suicides were linked to the scheme.

Equality Act 2010 · Human Rights Act 1998 · EU AI Act (High-Risk) · UK Algorithmic Transparency Standard

UK Ofqual A-Level Algorithm Downgraded State School Students

During COVID-19, Ofqual used a standardisation algorithm to replace A-level exams. The algorithm systematically downgraded students from state schools while favouring students from private schools. Nearly 40% of teacher-predicted grades were downgraded.

Equality Act 2010 · Human Rights Act 1998 · GDPR Article 22 (Automated Decision-Making) · UK Algorithmic Transparency Standard

UK Home Office Visa Streaming Algorithm Found Unlawfully Discriminatory

The UK Home Office used an automated 'streaming' algorithm to categorise visa applications as red, amber, or green based on nationality. Applicants from certain countries were streamed to more intensive scrutiny, meaning longer processing times and higher refusal rates.

Equal Credit Opportunity Act (ECOA) · Fair Housing Act · NY Banking Law · CFPB Regulations

Apple Card Algorithm Allegedly Gave Women Lower Credit Limits Than Husbands

Tech entrepreneur David Heinemeier Hansson publicly revealed that Apple Card had given him a credit limit 20 times higher than his wife despite them filing joint tax returns and his wife having a higher credit score. Multiple women reported similar disparities.

Equal Employment Opportunity (EEO) Law · Title VII Civil Rights Act · EU AI Act (High-Risk)

Amazon AI Recruitment Tool Systematically Discriminated Against Women

Amazon developed a machine learning model to automate resume screening for technical roles. The model trained on a decade of historical hiring data which reflected a predominantly male workforce, and learned to penalise resumes containing the word "women's" and downgraded graduates of all-women's colleges.

NHTSA Automated Vehicle Policy · Arizona AV Testing Regulations · NTSB Investigation Standards

Uber ATG Autonomous Vehicle Killed Pedestrian in First Autonomous Fatality

An Uber self-driving test vehicle struck and killed Elaine Herzberg in Tempe, Arizona. The system detected the pedestrian but misclassified her, and the automatic emergency braking was disabled during autonomous operation.

FDA SaMD Guidance · 21 CFR Part 820 (QSR) · HIPAA · ICH E6 GCP Guidelines

IBM Watson for Oncology Recommended Unsafe Cancer Treatments

Internal IBM documents leaked to STAT News revealed that IBM Watson for Oncology had recommended treatments that oncologists described as 'unsafe and incorrect'. The system generated advice based on hypothetical training cases rather than real patient data.

Privacy Act · Social Security Act · Administrative Decisions (Judicial Review) Act

Robodebt: Automated Income-Averaging Issued 381,000 Unlawful Debt Notices

The Australian Government's automated welfare debt recovery scheme used income-averaging to calculate alleged debts owed by welfare recipients, comparing Centrelink payment data against ATO tax records without human verification. The algorithm generated 381,000 debt notices over three years.

US Civil Rights Act · Equal Protection Clause · EU AI Act (Prohibited/High-Risk)

COMPAS Recidivism Algorithm Showed Racial Bias in US Courts

ProPublica's investigation found that Black defendants were nearly twice as likely to be falsely flagged as future criminals compared to white defendants. The tool was being used by judges to inform bail, sentencing, and parole decisions.

FTC Act Section 5 · Common Rule (45 CFR 46) · GDPR (analogous) · Belmont Report Principles

Meta Manipulated 700,000 Users' News Feeds Without Consent in Emotion Study

Facebook conducted a large-scale psychological experiment on approximately 689,000 users, systematically altering their News Feed algorithms to test whether emotions were contagious online. The study was published without users ever being informed they were research subjects.