On this page
AI quality assurance (AI QA) is the engineering discipline of defining measurable tests, acceptance thresholds, monitoring signals, and evidence artifacts that prove an AI system behaves acceptably in its real deployment context. The goal is not “never fail” — it is making failures measurable, bounded, detectable, and recoverable. A single aggregate benchmark score is not QA. A documented suite of test sets, SLOs, drift monitors, and remediation artifacts is.
Start here:
- Measure validity, reliability, and safety first — these three dimensions carry the highest consequence in most production deployments.
- Set at least one behavioral SLI (e.g., hallucination rate or human override rate) before your first release.
- Preserve every test set, threshold decision, and incident artifact as a citable, versioned document.
Standards anchoring this guide: NIST AI RMF, NIST ARIA/CoRIx, and ISO/IEC TS 5723. Glitchive’s verified failure case library provides concrete remediation evidence throughout.
Table of Contents
- What quality dimensions does your AI system actually need to measure?
- What test types belong in a risk-based AI QA suite?
- What should you monitor in production, and how do you detect drift early?
- The AI QA scorecard and release checklist
- How do you design evaluations that measure real-world impact?
- Who owns AI QA, and what documentation must you preserve?
- Key Takeaways
- The gap between “we ran evals” and “we have QA”
- Glitchive’s verified failure cases support your TEVV program
- Useful sources and further reading
- FAQ
What quality dimensions does your AI system actually need to measure?
Nine dimensions define trustworthy AI behavior. The NIST AI RMF Measure function organizes them; ISO/IEC TS 5723 defines robustness specifically as the ability to maintain performance under distributional shift and adversarial perturbation.
- Operational observability: — Latency, cost, and error rates are instrumented and alertable. Failure mode: silent degradation invisible to operators.
Prioritize by risk tier. A medical triage assistant demands validity, safety, and fairness at the top of the stack. A low-stakes internal search tool can defer fairness audits to a later sprint. The NIST AI RMF Measure playbook is explicit: measurement approaches must connect to deployment context and affected stakeholders, not just technical benchmarks.
“Graceful failures — confident but wrong outputs — are more dangerous than hard crashes. QA must prioritize detection and containment of these silent failures.” — AI Reliability Engineering: A Practical Framework
Pro Tip: For a first release, measure validity (hallucination rate), safety (policy compliance rate), and operational observability (latency p95, error rate). Add fairness and robustness in sprint two once baseline instrumentation is stable.
What test types belong in a risk-based AI QA suite?
The AI Assurance research argues that standard NLP metrics are insufficient for RAG systems and proposes a five-layer assurance pyramid. The practical translation: you need at least five test types, each with a distinct objective and ownership.
| Test Type | Objective | Example Metric | Frequency | Owner |
|---|---|---|---|---|
| Component / unit | Verify individual model behavior | Hallucination rate per prompt class | Every commit | ML engineer |
| Retrieval / RAG | Validate grounding and faithfulness | Context recall, faithfulness score | Every index update | ML engineer |
| Regression suite | Catch silent regressions on known failures | Pass rate on fixed-failure set | Every model version | QA / reliability |
| Red-team / adversarial | Probe for prompt injection, jailbreaks | Attack success rate | Pre-release, quarterly | Security / red team |
| Scenario / field | Test real-world task completion | Task success rate, human override rate | Monthly or on drift | Product / reliability |
| Human-in-the-loop (ARIA-style) | Measure contextual robustness with real users | CoRIx score, annotator agreement | Pre-major release | Independent assessor |
Test set hygiene matters as much as test design. Version every test set with a hash and a changelog. Label each example with its failure category, severity, and the date it was added. Annotators should record: the input, the model output, the expected output, the failure type (grounding, safety, fairness, etc.), severity (critical/major/minor), and a free-text note. Representativeness means covering demographic groups, edge-case phrasings, and the distribution of real user queries — not just the cases you already know fail.
Pro Tip: Lock a “golden set” of 50–200 labeled examples before your first production release. This set becomes your regression baseline. Never add to it without a review; never delete from it without a postmortem.
What should you monitor in production, and how do you detect drift early?
Google Cloud’s AI/ML reliability guidance recommends holistic observability: infrastructure, model performance, and behavioral signals together. SRE for AI agents adds judgment SLIs and OpenTelemetry GenAI semantic conventions for telemetry.
| Signal | Why It Matters | Alert Threshold | Responder |
|---|---|---|---|
| Hallucination rate (sampled) | Catches grounding failures before users report | Exceeds SLO or 2-week downward trend | ML engineer |
| Human override rate | Proxy for user trust and task failure | Rises above baseline by 10% | Product / reliability |
| Token cost per request | Detects prompt injection or runaway loops | Shows significant drift from rolling 7-day average | Reliability / SRE |
| Tool-call ratio (agents) | Flags reasoning loops and unexpected tool use | Exceeds expected ratio by 2× | ML engineer |
| Retrieval context recall | Catches index drift or embedding degradation | Drops below threshold on weekly eval | ML engineer |
| User feedback sentiment | Lagging but real signal of behavioral degradation | Sustained negative shift over 5 days | Product |
Automate TEVV pipelines to trigger on: model version change, prompt template update, detected statistical drift, or a fixed weekly cadence. A single bad output is noise. A sustained downward trend across two weeks is a signal worth a postmortem.
Pro Tip: Treat token-cost drift and tool-call ratio drift as leading indicators of prompt injection or reasoning-loop failures — they often surface before hallucination rate moves.
The AI QA scorecard and release checklist
Copy this scorecard into your CI/CD gate. Every dimension needs a current value, a threshold, and a pass/fail before a release ships.
| Dimension | SLI | Threshold | Current Value | Pass/Fail | Evidence Link |
|---|---|---|---|---|---|
| Validity | Hallucination rate | ≤ 5% (≤ 1% for critical deployments) | [measure] | — | Eval run ID |
| Safety | Policy compliance rate | ≥ 99% | [measure] | — | Red-team report |
| Robustness | Adversarial pass rate | ≥ 90% | [measure] | — | Red-team log |
| Fairness | Demographic parity gap | ≤ 5% | [measure] | — | Fairness eval report |
| Reliability | Regression suite pass rate | 100% | [measure] | — | CI run ID |
| Observability | Monitoring hooks active | All green | [check] | — | Dashboard link |
Release checklist:
- All test types in the matrix have run against the current model version.
- Golden regression set passes 100%.
- Red-team / adversarial tests completed; findings triaged.
- Human review completed for any critical-tier dimension.
- Monitoring hooks confirmed active; alert thresholds set.
- Rollback criteria documented and tested.
- Scorecard filed with evidence links in the TEVV report.
- Reliability contract updated if thresholds changed.
Store scorecard artifacts in version control alongside the model release tag. Auditors need to reconstruct exactly what was measured, at what threshold, and with what evidence — not a summary.
How do you design evaluations that measure real-world impact?
The NIST ARIA pilot introduced a three-layer evaluation stack: a testing layer (scenario runs with human testers), an assessment layer (expert annotation and questionnaires), and a measurement layer (CoRIx aggregation of contextual robustness indicators). This structure separates raw signal collection from interpretation — which matters when deployment context determines harm.
A simplified measurement tree for a customer-service LLM:
| Raw Signal | Aggregation | Contextual Robustness Indicator |
|---|---|---|
| Task completion rate | Weighted by task criticality | Operational robustness score |
| Policy compliance rate | Flagged failures reviewed by annotator | Safety robustness score |
| Demographic performance gap | Parity gap across user segments | Fairness robustness score |
| Adversarial pass rate | Proportion resisting injection attempts | Security robustness score |
| CoRIx composite | Weighted average of above | Overall contextual robustness |
Human subject protections apply when testers interact with a live system: informed consent, data minimization, and no use of tester data for model training without explicit permission. For a medium-risk deployment, a useful minimum is 30–50 diverse testers covering the primary demographic groups in your user base, with at least two independent annotators scoring each session for inter-rater reliability.
Pro Tip: When your deployment context determines harm (medical, legal, financial), independent assessors — not the model’s own team — should run the ARIA-style evaluation. Self-assessment at high risk tiers is a governance gap.
Who owns AI QA, and what documentation must you preserve?
Ownership gaps are where QA programs collapse. Assign every activity to a named role before the first release.
| Activity | Owner | Reviewer | Escalation |
|---|---|---|---|
| Test set design and versioning | ML engineer | QA lead | Engineering manager |
| Threshold and SLO setting | Reliability engineer | Product owner | Engineering manager |
| Red-team / adversarial testing | Security / red team | ML engineer | CISO |
| Human-in-the-loop evaluation | Product / UX research | Independent assessor | Engineering manager |
| Incident response and postmortem | Reliability engineer | ML engineer | Engineering manager |
| Periodic TEVV review | QA lead | Cross-team reviewer | Engineering manager |
The NIST AI RMF Measure playbook requires that measurement approaches be documented, connected to deployment context, and reviewed by domain experts and affected communities where risks are significant. Independent or cross-team review is recommended whenever a dimension is rated high-risk.
Documentation checklist:
- Versioned test sets with hashes and changelogs
- Annotation schema (fields, severity definitions, labeling guidelines)
- Measurement methods (how each SLI is computed, sampling rate, tooling)
- TEVV reports per release (scorecard + evidence links)
- Incident logs and postmortems with regression tests added
- Reliability contracts per major version
Pro Tip: Schedule a cross-team TEVV review at every major model version change and at least quarterly for production systems. The reviewer should not be the team that built the model.
Key Takeaways
Effective AI quality assurance requires documented test sets, behavioral SLOs, drift monitoring, and preserved evidence artifacts — not a single benchmark score.
| Point | Details |
|---|---|
| Measure behavior, not just accuracy | Set behavioral SLIs (hallucination rate, policy compliance, human override rate) before release. |
| Risk-tier your thresholds | For validity, use a hallucination rate threshold of ≤ 1% for critical deployments and ≤ 5% for others; policy compliance should be ≥ 99%. Low-consequence tools use trend monitoring only. |
| Preserve every artifact | Version test sets, scorecard results, and postmortems with evidence links for audits and regression. |
| Treat error budgets as autonomy gates | When a behavioral error budget is exhausted, graduated responses (throttle → pause → circuit break) reduce autonomy automatically. |
| Use Glitchive for verified failure evidence | Glitchive’s case library provides citable, real-world failure examples and remediation artifacts to seed regression test sets and TEVV playbooks. |
The gap between “we ran evals” and “we have QA”
Most teams ship with evals. Few ship with QA. The difference is not the number of tests — it is whether the tests are connected to thresholds, the thresholds are connected to release gates, and the gates are connected to documented evidence that survives the team that wrote them.
The organizational friction is real. Engineering managers face pressure to ship; reliability engineers inherit systems they did not design; product owners treat QA as a checkbox rather than a contract. The way through is small, concrete wins: one behavioral SLI on the dashboard before the next release, one golden regression set in version control, one postmortem that actually produces a new test. Each artifact builds the case that QA is not overhead — it is the evidence that the system does what it claims.
Glitchive’s case library exists precisely for this moment. When you need to justify a threshold, explain a failure taxonomy, or show a stakeholder what a real postmortem looks like, verified case evidence is more persuasive than a framework diagram. The coding agent database wipe case and the chatbot refund-policy liability case are exactly the kind of artifacts that make abstract risk tiers concrete to a product owner who has never seen a production AI failure up close.
Glitchive’s verified failure cases support your TEVV program
The hardest part of building a regression test set is finding labeled failure examples before your system produces them. Glitchive’s searchable library of verified AI failure cases gives your team a head start: each case documents the incident, contributing factors, technical analysis, and the specific fix applied — with permanent, citable URLs and fully sourced references.
Two cases worth pulling into your TEVV playbook now: the coding agent that wiped a production database during an active code freeze (directly relevant to action-control gates and canary gating for autonomous agents) and the support chatbot that invented a refund policy (a grounding and policy-compliance failure with a tribunal ruling attached). Both cases include remediation steps you can adapt as regression tests.
Browse the full Glitchive cases directory to find failure patterns matching your deployment context, then copy the scorecard above into your CI/CD gate.
Useful sources and further reading
The sources below are the primary references for designing and defending an AI QA program.
- AI Reliability Engineering: A Practical Framework
- NIST AI RMF — Measure playbook
- Artificial Intelligence Risk Management Framework (AI RMF) 1.0
- AI Assurance: A Comprehensive Testing Strategy for Enterprise AI Systems
- Applying site reliability engineering to autonomous AI agents
- AI and ML perspective: Reliability | Google Cloud Architecture
- Site Reliability Engineering for AI Agent Systems: Observability, Incident Response, and Operational Patterns
FAQ
What is AI quality assurance?
AI quality assurance is the practice of defining measurable tests, acceptance thresholds, monitoring signals, and evidence artifacts that prove an AI system behaves acceptably in its deployment context — grounded in standards like NIST AI RMF and ISO/IEC TS 5723.
What is the difference between a behavioral SLI and a technical SLI?
A technical SLI measures infrastructure behavior (latency, uptime, error rate); a behavioral SLI measures model output quality (hallucination rate, policy compliance rate, human override rate) — the latter requires sampling and human or automated evaluation, not just telemetry.
How often should you run TEVV evaluations in production?
Trigger TEVV pipelines on every model version change, every prompt template update, and on detected statistical drift; run a fixed cadence (weekly sampled eval, monthly full suite) regardless of changes.
When is an independent assessor required for AI QA?
Independent or cross-team assessment is recommended by the NIST AI RMF whenever a quality dimension is rated high-risk; for ARIA-style contextual robustness evaluations, the assessor should not be the team that built the model.
What artifacts must a postmortem include to support an audit?
A postmortem must include reproduction steps with the exact failing input/output, telemetry extracts from the failure window, root cause mapped to a failure taxonomy, regression tests added to the golden set, and an updated reliability contract.