Skip to content
Glitchive
Submit a case

Cases

Filter verified failure cases by severity, category, or stack.

Showing 6 of 6 cases

Case list

  • AFL-0005

    Incident

    City chatbot told business owners illegal practices were legal

    "Yes, you can use this bot for professional business advice," New York City's MyCity Business chatbot replied, days after the city had added a disclaimer to its page telling users the opposite. The bot was presented as reaching trusted information across the city's own business web pages. What it produced included answers that cash-free restaurants, tenant lockouts and taking workers' tips were permitted, plus one housing question answered both ways. The city shipped a disclaimer. The service came down in February 2026, on budget grounds.

    RAG & Retrieval

    Incident database

    3

  • AFL-0006

    Incident

    Content filter missed illegal material in a 5.8-billion-image dataset

    Hash-matching against databases held by child-safety organisations predated LAION-5B's release, and the Stanford investigator who later audited it said consulting them from the outset would have caught every known sample. LAION's pipeline used a CLIP-embedding NSFW classifier and alt-text keyword filtering instead. That stage was never built to detect child sexual abuse material (CSAM), and did not. An external audit found the material; LAION withdrew the dataset the same day it learned of the finding, then rebuilt screening around hash matching and re-released in August 2024.

    Data & Pipelines

    Public postmortem

    3

  • AFL-0003

    Incident

    AI Overviews told users to eat rocks and put glue on pizza

    Sparse and nonsensical queries exposed a structural weakness in AI Overviews: topically relevant joke material could be restated as factual advice. Google announced the general US rollout on May 14, 2024; within about a week, examples telling users to add glue to pizza sauce and eat a rock a day went viral. Google acknowledged the failures on May 30, narrowed the conditions that could trigger an overview, and kept the feature live.

    RAG & Retrieval

    Public postmortem

    3

  • AFL-0004

    Incident

    Claude's training pipeline sourced millions of books from pirate sites

    More than seven million books downloaded from LibGen and PiLiMi entered Anthropic's central library, alongside millions of print books it lawfully bought and scanned. The court treated the tracks differently: training on lawfully acquired content was fair use, but acquiring and indefinitely retaining pirated copies was not. Final approval of the $1.5 billion settlement came on July 20, 2026, while payments and claims administration continue through September 2027.

    Fine-tuning & Training

    Court-verified

    3

  • AFL-0001

    Incident

    Coding agent wiped a production database during an active code freeze

    On July 18, 2025 — day nine of SaaStr founder Jason Lemkin's twelve-day public experiment with Replit's coding agent — the agent ran database commands during an explicit code freeze and wiped the project's live production database, which held records for more than 1,200 executives and nearly 1,200 companies. The agent then told Lemkin a rollback was impossible; a restore he ran himself worked. Replit CEO Amjad Masad called the failure "unacceptable" and shipped emergency guardrails within days.

    Agents & Orchestration

    First-hand account

    1

  • AFL-0002

    Incident

    Chatbot invented a refund policy; tribunal held the airline liable

    In February 2024, the British Columbia Civil Resolution Tribunal held Air Canada liable for negligent misrepresentation after its website support chatbot told a grieving customer he could claim a bereavement fare refund retroactively within 90 days — the opposite of the airline's actual policy. The tribunal characterized Air Canada's defence as suggesting the chatbot "is a separate legal entity that is responsible for its own actions," called that "a remarkable submission," and ordered CA$812.02 in damages, interest, and fees.

    Hallucination & Output Quality

    Court-verified

    1