The problem
Hash-matching against databases held by child-safety organisations predated LAION-5B's release, and the Stanford investigator who later audited it said consulting them from the outset would have caught every known sample. LAION's pipeline used a CLIP-embedding NSFW classifier and alt-text keyword filtering instead. That stage was never built to detect child sexual abuse material (CSAM), and did not. An external audit found the material; LAION withdrew the dataset the same day it learned of the finding, then rebuilt screening around hash matching and re-released in August 2024.
Blast radius
5.85B image-text pairs; dataset withdrawn from public distribution
Time to detect
About 21 months to external audit
Recovered
Partial — Dataset remediated; downstream model status unclear
01 — Context
LAION is a volunteer-run nonprofit. LAION-5B, released in March 2022, held 5.85 billion image-text pairs, was among the largest public image-text datasets, and was foundational to open-source diffusion models including Stable Diffusion. It was assembled by scraping Common Crawl at multi-billion scale, with the pipeline extracting image URLs and their associated alt text rather than hosting the images, which it referenced by link. The result was then published openly, and that is the structural fact that matters here: downstream trainers inherited whatever the filtering stage failed to remove, and inherited it as links.
02 — What failed
Two mechanisms made up the content-safety stage — a general NSFW classifier scoring CLIP embeddings, and keyword filtering over alt text. Neither is capable of the job. Keyword filtering inspects a label supplied by whoever uploaded the image, so material that uploaders do not label as such passes through; CLIP-based NSFW classifiers were never built or evaluated for this task. The Stanford Internet Observatory investigation, led by David Thiel and running from September to December 2023, applied an entirely different toolkit and found what the pipeline had not: many hundreds of known CSAM instances, plus additional candidates verified externally. The same corpus had been distributed for roughly twenty-one months in between.
03 — Symptoms
There is no observable runtime symptom, and that is the finding rather than a gap in the record. A contaminated corpus looks exactly like a clean one from the outside: no error, no degraded metric, no user complaint. The detection surface is an external audit run with purpose-built tooling, which is how this was found and, on the public record, the only way it was ever going to be. Practitioners reach the incident through phrasings such as "LAION-5B CSAM", "Stable Diffusion training data child abuse", "LAION dataset taken offline", "CLIP NSFW filter failed", and "Stanford Internet Observatory LAION report". The generalisable shape is uncomfortable: if your only content-safety control is a classifier trained on a different taxonomy, you have no symptom to look for and no basis for believing you are clean.
04 — Root cause
The detection method was mismatched to the category. A general-purpose content classifier scores an image against the concept it was trained on, whereas detecting known material is an exact-match lookup against a curated list. Those are different operations, and no amount of tuning the first produces the second — a threshold on an NSFW score cannot convert a similarity judgement into a database hit. Alt-text keyword matching fails for a related but cruder reason: it interrogates the uploader's label rather than the image. The technique that fits this specific category — hash matching against databases maintained by child-safety organisations — already existed and was available at build time. Thiel's counterfactual quantifies the gap precisely: consulting those organisations and hash databases "from the outset" would have caught "100% of all known samples of CSAM."
05 — Technical explanation
Three method sets are worth setting against each other, because the contrast is the technical content of this case. The original pipeline ran CLIP embeddings into an NSFW classifier score, then applied an alt-text keyword pass; both signals are approximate and both concern a broad category rather than a specific prohibited one. Stanford's audit used PhotoDNA perceptual hashing, cryptographic hashing, k-nearest-neighbour queries against the dataset's own embeddings, and ML classifiers — the kNN step turning the corpus's embedding index into a search instrument aimed at itself. LAION's remediation used MD5 and SHA-1 matching on image hashes and URL hashes against lists held by the Internet Watch Foundation and the Canadian Centre for Child Protection, described by LAION as done "without ever having to touch suspected links." That property is not incidental: hash comparison keeps the screening privacy-preserving and keeps the people operating it away from the material, which is what makes it viable as a routine, repeatable pipeline stage rather than a manual review effort. The counts belong with their sources. Stanford externally verified 1,008 links, alongside its finding of many hundreds of known instances, and separately estimated that the true number was likely "two to five times" larger than what it confirmed — an estimate, not a finding. LAION's remediation removed 2,236 links, a count LAION states subsumes Stanford's 1,008.
06 — Contributing factors
LAION is a volunteer-run nonprofit, and no partnership with a child-safety organisation existed when the dataset was built; the arrangements with the Internet Watch Foundation and the Canadian Centre for Child Protection were formed between February and May 2024, after the audit. No consultation with such an organisation preceded the release. The dataset was redistributed openly as image links, so downstream trainers took on whatever the filter had missed without a way to see it. The disclosure path was a separate gap: LAION learned of the finding through Bloomberg press coverage rather than from the researchers directly. Why the original pipeline used a general classifier rather than hash matching is not documented, and nothing here should be read as a claim about intent, negligence, or resourcing decisions.
07 — Attempted fixes
The CLIP-based NSFW classifier and the alt-text keyword filter were themselves the attempted control, and they did not detect the material. After disclosure, LAION took the dataset offline on December 19, 2023, the same day it learned of the finding; formed partnerships with the Internet Watch Foundation and the Canadian Centre for Child Protection between February and May 2024; rebuilt screening as image-hash and URL-hash matching against their lists; removed 2,236 links; and released Re-LAION-5B in two variants on August 30, 2024, with a technical writeup describing both the failure and the remediation. Downstream, Stability AI stated on the record that its models "were trained on a filtered subset of that dataset" and were subsequently "fine-tuned... to mitigate residual behaviors," with input and output prompt filters added; it declined to say whether it knew of the problem beforehand. Whether affected checkpoints were retrained from scratch or patched after the fact is not established — the company's own quoted language is ambiguous on the point and it did not clarify. That remains open.
08 — Lessons learned
A general-purpose content classifier and detection of one specific prohibited category are different problems, and treating the former as coverage for the latter yields a control that cannot work however carefully it is tuned. The alternative was neither exotic nor expensive: hash matching against maintained databases was available before this dataset shipped, cheap to run at scale, and executable without anyone needing to view flagged material. Because the corpus was published openly as links, whatever the filter missed propagated to every downstream trainer silently and without a marker. And a failure with no runtime symptom is only ever found by someone auditing with the right instrument — which makes the audit something a dataset project has to plan and commission, not something it waits to receive.
09 — Prevention checklist
- Hash-match every scraped image and URL against maintained child-safety databases before publication.
- Engage child-safety organisations during dataset construction, not after an external audit forces it.
- Make content-safety screening its own gated pipeline stage with its own pass criteria.
- Re-screen any dataset kept live for redistribution against updated hash lists, which grow after release.
- Publish the filtering methodology with the dataset so downstream trainers can assess residual risk.
References
- Identifying and Eliminating CSAM in Generative ML Training Data and Models (Stanford Internet Observatory) (opens in a new tab)
purl.stanford.edu
Other
-
Blog
- Investigation Finds AI Image Generation Models Trained on Child Abuse (Stanford FSI) (opens in a new tab)
cyber.fsi.stanford.edu
Other
- Stanford: AI image generator training data included child sex abuse material (The Register) (opens in a new tab)
theregister.com
News
- Exposing the Rotten Reality of AI Training Data (Tech Policy Press) (opens in a new tab)
techpolicy.press
News
10 — Solutions (3)
Hash-match against maintained databases as a gated pipeline stage
Root cause addressed
The pipeline relied on a general-purpose classifier for a category that requires exact-match detection against a curated list of known material.
Separate content-safety screening from quality filtering
Root cause addressed
Safety was folded into a general NSFW and quality filtering step, so no stage owned it and no criterion could fail the release on safety grounds alone.
Publish the filtering methodology and re-screen what stays live
Root cause addressed
Downstream trainers inherited contamination with no way to assess or bound it, and hash lists continue to grow after a dataset is released.
Fixed this yourself? Add the steps that helped with Content filter missed illegal material in a 5.8-billion-image dataset.
Solution received
Thank you for sharing what worked. Your solution will be reviewed by a human before it appears alongside the others. It does not publish automatically.