Solution 01
Verify grounding against source documents instead of asserting it
01 — Root cause addressed
This addresses a system represented as grounded in the city's official documentation while producing answers that contradicted rules actually on the books.
02 — Implementation
Before launch, build the audit set out of the authoritative corpus itself rather than out of imagined user questions. Enumerate the regulated topics the system will be asked about, pull the governing passage for each one from the source documents, and check the system's answer against that passage. A contradiction is a defect, not a limitation to note in a release document. Alongside the audit, require the system to surface the specific source passage behind every regulatory answer, so that auditors before launch and users afterwards are checking against the same text rather than against a summary of it.
03 — Prevention
Re-run the audit whenever the regulations or the source pages change, and keep passage-level citation attached to regulatory answers so any wrong answer can be traced back to the text it should have followed.
04 — Trade-offs
Building and maintaining the audit set is real ongoing work that scales with the corpus and has to be redone whenever regulations change. Passage-level citation constrains how answers can be phrased and makes them longer and less conversational. This is the wrong choice for a system answering low-stakes questions where a wrong answer costs the reader nothing.