Solution 02
Treat inconsistency on identical queries as a release blocker
01 — Root cause addressed
This addresses the same housing-voucher question returning "no" to ten staffers and "yes" to one reporter, a failure no individual user was positioned to observe.
02 — Implementation
Make repeat-and-compare testing a standing pre-release gate. Ask each high-stakes question many times, across separate sessions and separate users, and compare the answers against each other instead of reading them one at a time. A material contradiction — one run permitting what another forbids — fails the release rather than being documented as variance and shipped. Because this class of defect is invisible from any single session, the comparison has to be run deliberately by the team that owns the system, and it has to be run again after every change that could move the answers.
03 — Prevention
Keep the repeat-and-compare suite permanently attached to the high-stakes question set, so answer consistency is re-measured on every release rather than sampled once before launch.
04 — Trade-offs
This costs many times more test queries than single-pass checking, and it needs human judgement about what counts as materially contradictory rather than harmlessly reworded. Some inconsistency is inherent to generative systems, so the gate has to be scoped to the answers where contradiction actually matters; scoped too broadly it blocks every release and gets switched off.