On this page
Membership inference is the attack of confirming whether a specific record was used to train a model. The model’s raw training data never leaves the server — yet an adversary who can query the model may still learn that a particular individual’s record was in the training set. That membership fact alone can be a sensitive disclosure: knowing someone’s data appeared in a clinical depression cohort, a fraud-detection dataset, or a proprietary financial model reveals something real about them. The highest-stakes targets are health records, financial transaction histories, and small-population proprietary datasets where even a handful of confirmed memberships can cause regulatory and reputational harm.
Before you go deeper into testing, three steps belong on your immediate backlog:
- Restrict model outputs. Return hard labels or coarse confidence bins rather than raw probability vectors. Full softmax outputs are the most direct signal an attacker can exploit.
- Rate-limit and log inference queries. Shadow-model attacks require many queries. Anomaly detection on query volume and pattern is a lightweight first layer of defense, though not sufficient on its own.
- Plan a privacy-aware training strategy now. Differential privacy (DP-SGD) or aggressive L2 regularization and early stopping are the interventions with the strongest empirical backing. Retrofitting them after training is costly; baking them in from the start is far cheaper.
The risk is not theoretical. Shokri et al. reported median attack accuracy of 94% against Google-trained classifiers and 74% against Amazon-trained classifiers on 10,000-record retail datasets using black-box shadow-model attacks — experiments run against real public cloud APIs.
Key Takeaways
Membership inference risk is highest for models trained on small, sensitive, or partially synthetic datasets — and the defenses that actually work are training-time interventions, not output restrictions.
| Point | Details |
|---|---|
| Evaluate at low FPR | Report precision at 0.1% FPR, not just AUC; aggregate metrics overstate operational risk. |
| Training-time defenses win | DP-SGD and strong regularization substantially reduce leakage; confidence masking alone does not. |
| Partial synthesis is not safe | Attack precision reached up to 0.9 for sizable subpopulations in VUMC and All of Us experiments. |
| Audit before release | Run a shadow-model or metric-based audit on every major model release; schedule annual re-tests for sensitive-data models. |
| Glitchive case library | Free, verified case studies and audit-ready runbook templates at Glitchive support pre-release MI testing. |
Table of Contents
- What exactly is membership inference, and how does it differ from related attacks?
- What attacker access models apply to membership inference?
- How do canonical membership inference attacks actually work?
- What factors make a model more vulnerable to membership inference?
- How do you run a membership inference audit?
- What defenses actually work, and what do they cost?
- What do real experiments tell us about membership inference risk?
- What are the real privacy risks when membership inference succeeds?
- How do you interpret MI test results for operational decisions?
- What advanced and emerging attack variants should you know about?
- What tools and frameworks are available for MI testing?
- What legal and regulatory obligations apply to membership inference risk?
- Glitchive’s perspective on membership inference governance
- Glitchive’s case library covers membership inference and beyond
- Sources
- FAQ
What exactly is membership inference, and how does it differ from related attacks?
The formal problem: given a trained model f and a data point x, decide whether x was a member of the training set D. The attacker’s output is a binary decision, not a reconstruction of x’s content. That distinction matters operationally, because membership inference is often conflated with two related but different threats.
| Attack type | Attacker’s goal | Required model output | Typical consequence |
|---|---|---|---|
| Membership inference | Confirm whether x ∈ training set | Predictions (labels or probabilities) | Sensitive membership disclosure (e.g., patient in a disease cohort) |
| Model inversion | Reconstruct a representative input for a class or individual | Class probabilities or gradients | Approximate reconstruction of training-data features |
| Attribute inference | Infer a missing sensitive attribute of a known individual | Predictions conditioned on partial input | Exposure of a sensitive attribute (e.g., income, diagnosis) |
Model inversion tries to recover what the training data looked like; attribute inference tries to fill in a missing field for a known person. Membership inference asks only whether a specific record was present. An attacker does not need to extract the record’s content to cause harm — knowing that a named individual’s record appears in a dataset of HIV-positive patients is damaging on its own.
To make the distinction concrete: suppose a hospital trains a readmission-risk classifier. A model-inversion attacker tries to reconstruct a patient’s lab values from the model’s outputs. An attribute-inference attacker knows the patient’s age and diagnosis and tries to infer their insurance status. A membership-inference attacker simply asks: “Was this specific patient record in the training set?” All three are privacy attacks; they differ in what the adversary already knows and what they are trying to learn.
What attacker access models apply to membership inference?
Threat modeling for membership inference starts with access. The table below maps access type to practical capability and detectability.
| Access type | Attacker capability | Query footprint / detectability | Notes |
|---|---|---|---|
| Black-box | Queries model API; observes predictions only | High query volume for shadow-model training; potentially detectable | Most common real-world scenario |
| Label-only | Receives hard labels only, no probabilities | Moderate; uses perturbation-based robustness signals | Confidence masking does not stop this |
| White-box | Full access to model weights and gradients | Low query footprint; harder to detect | Insider threat or model-weight leak scenario |
| Query-limited | Restricted API with rate limits or query caps | Very low footprint; may evade volume-based monitoring | Attacker amortizes queries over time |
When threat-modeling your own system, evaluate these attacker capabilities explicitly:
- Auxiliary data: Does the attacker have access to data drawn from the same distribution as your training set? Shadow-model attacks require this.
- Shadow-model compute budget: Training shadow models is feasible on consumer hardware for small-to-medium models. Do not assume compute is a meaningful barrier.
- Side-channel access: Timing differences in inference, cached responses, or API error codes can leak membership signals independently of the model’s predictions.
- Query persistence: An attacker with a long time horizon can spread queries across days or weeks, defeating rate-limit thresholds calibrated for burst detection.
Pro Tip: Sparse, low-volume probing is the hardest MI pattern to catch. An adversary running a label-only attack against a query-limited API can stay well below anomaly-detection thresholds by spacing queries over time. Monitoring designed for noisy, high-volume attacks will miss this entirely. Pair query logging with statistical tests on the distribution of queried inputs, not just raw query counts.
Label-only membership inference attacks demonstrate that hiding confidence scores is not a sufficient defense: the robustness of hard labels under input perturbations carries its own membership signal.
How do canonical membership inference attacks actually work?
The shadow-model pipeline introduced by Shokri et al. remains the conceptual foundation for most black-box attacks. Here is the pipeline at a reproducible conceptual level:
- Collect auxiliary data drawn from the same distribution as the target model’s training set (or a close approximation).
- Train multiple shadow models on disjoint subsets of the auxiliary data, mimicking the target model’s training procedure as closely as possible.
- Generate labeled attack training data by querying each shadow model with records known to be in its training set (members) and records known to be out (non-members). Record the model’s output vector for each query.
- Train an attack classifier on this labeled dataset. The classifier learns to distinguish the output signatures of members from non-members.
- Apply the attack classifier to the target model. Query the target with the record of interest, feed the output to the attack classifier, and read the membership prediction.
Label-only variants skip step 3’s probability vectors entirely. Instead, they measure how the hard label changes as the input is perturbed (e.g., by adding noise or applying small transformations). Training members tend to produce more stable predictions under perturbation than non-members — a signal that persists even when the API returns only a class label.
Single-query statistical methods take a different approach: they compare the model’s loss or confidence on a candidate record against a threshold derived from the model’s overall behavior, without training a shadow model at all. These are lower-cost attacks that trade precision for simplicity.
Generative models introduce a newer attack surface. For seq2seq and summarization models, research on privacy risks in summarization tasks shows that document-only attacks using text-similarity and robustness signals can infer membership even without access to reference summaries, exceeding random-baseline performance and producing high-confidence inferences for certain training examples.
What factors make a model more vulnerable to membership inference?
Overfitting is the single strongest driver. A model that has memorized its training data will assign systematically higher confidence to training members than to non-members — and that gap is exactly what an attack classifier exploits. But overfitting is not the only factor.
Memorization and per-example influence vary across training examples. Rare, atypical, or outlier records are memorized more strongly than common ones. A model trained on a dataset with long-tail class distributions will tend to memorize the minority-class examples most aggressively, making those individuals the easiest targets.
Model capacity amplifies the problem. Larger models with more parameters have more room to memorize, though scale alone does not determine vulnerability. Research on MI attacks against large language models shows that MI methods adapted from classification tasks still find signals in large models — practitioners should not assume that scale eliminates membership leakage.
Dataset size and composition matter significantly. Small datasets produce more vulnerable models because each training example has a larger influence on the model’s parameters. Experiments on synthetic health data found that partially synthetic datasets can be highly vulnerable, with attack precision reaching up to 0.9 for sizable subpopulations, while fully synthetic approaches showed substantially lower risk.
Confidence and probability outputs are the most direct attack signal. Models that return full probability vectors are easier to attack than those returning only hard labels — though, as noted above, label-only attacks close much of that gap.
One important caveat on metrics: average attack accuracy and AUC can badly overstate operational risk. A balanced test set with equal members and non-members will produce an AUC that looks alarming even when the attack’s precision at realistic false-positive rates is low. Evaluate at the operating point that matters, not the aggregate.
How do you run a membership inference audit?
Evaluation metrics and when to use them
| Metric | When to use it | Operational meaning |
|---|---|---|
| AUC (ROC) | Screening; comparing attack variants | Aggregate discriminability; can overstate risk at realistic operating points |
| Precision at 0.1% FPR | Primary operational metric | Fraction of positive predictions that are correct when the attacker is very selective |
| TPR at fixed FPR | Regulatory or governance thresholds | How many true members the attacker finds at an acceptable false-alarm rate |
| Precision–recall curve | When member prevalence is low | More informative than ROC when positives are rare |
IJCAI 2024 evaluation guidance stresses reporting performance at very low false-positive rates (0.1% or 0.01%) for security-minded audits, because adversaries will only act when precision is high enough to be actionable.
Audit runbook
- Prepare test data. Assemble a balanced set of confirmed members (records used in training) and confirmed non-members (held-out records from the same distribution). Aim for at least 1,000 of each; more is better for low-FPR evaluation.
- Train shadow models. Use the same architecture and training procedure as the target model. Train at least 4–8 shadow models on disjoint subsets of your auxiliary data to generate a robust attack-training dataset.
- Generate attack training data. Query each shadow model with its member and non-member sets. Record the full output vector (or perturbation-robustness signal for label-only variants).
- Train the attack classifier. A simple binary classifier (logistic regression or a small MLP) on the shadow-model outputs is sufficient for most audits. Evaluate it on a held-out split of the shadow data first.
- Apply to the target model. Query the target with your test set. Feed outputs to the attack classifier. Compute AUC, precision at 0.1% FPR, and TPR at your chosen FPR threshold.
- Log everything. Record query counts, timestamps, and attack-classifier outputs. These logs are your audit trail for governance review.
- Set detection hooks. Instrument your production API to flag query patterns that match the shadow-training footprint (high-volume, systematically varied inputs from a single source).
- Schedule re-tests. Run the audit before every major model release and after any significant training-data change.
# Pseudocode: black-box MI audit (shadow-model loop)
for i in range(NUM_SHADOW_MODELS):
shadow_train, shadow_out = split(auxiliary_data)
shadow_model = train(shadow_train, architecture=TARGET_ARCH)
member_outputs = query(shadow_model, shadow_train)
nonmember_outputs = query(shadow_model, shadow_out)
attack_data.extend(label(member_outputs, 1))
attack_data.extend(label(nonmember_outputs, 0))
attack_clf = train_classifier(attack_data)
# Evaluate on target
target_outputs = query(target_model, test_set)
predictions = attack_clf.predict(target_outputs)
report_metrics(predictions, test_labels, fpr_thresholds=[0.001, 0.0001])What defenses actually work, and what do they cost?
| Defense | Typical effectiveness | Implementation cost | Utility impact |
|---|---|---|---|
| DP-SGD (differential privacy at training) | High; reduces leakage substantially in experiments | High; requires tuning ε, δ, clipping norm, noise multiplier | Measurable accuracy drop, especially at tight ε |
| L2 regularization + early stopping | Moderate; reduces overfitting-driven leakage | Low; standard training practice | Minimal if tuned carefully |
| Dropout | Moderate; reduces memorization | Low | Minimal |
| Confidence masking (output restriction) | Low; fragile against label-only attacks | Very low | Minimal |
| Label-only API (hard labels only) | Low-moderate; raises attacker cost but does not eliminate risk | Low | Minimal for most use cases |
| Rate-limiting and query monitoring | Low standalone; useful as detection layer | Low | None |
The most important implementation notes:
- DP-SGD configuration: Start with a privacy budget of ε = 8 as a conservative baseline for non-critical applications; tighten toward ε = 1–3 for sensitive health or financial data. Set the gradient clipping norm before tuning the noise multiplier. Use the TensorFlow Privacy or Opacus libraries rather than implementing DP-SGD from scratch.
- Regularization ordering: Apply L2 regularization and early stopping first — they are cheap and reduce overfitting-driven leakage with minimal engineering overhead. Add dropout as a secondary measure.
- Output restriction: Returning hard labels or coarse confidence bins raises the attacker’s cost but does not eliminate risk. Label-only attacks show that robustness-to-perturbation signals survive confidence masking. Treat output restriction as a cost-raiser, not a defense.
- Rate-limiting: Useful for detecting high-volume shadow-model training queries. Ineffective against patient, low-volume adversaries.
OWASP ML04:2023 lists randomized training, model obfuscation, regularization, and monitoring as its recommended mitigation strategies, consistent with the experimental evidence above.
For teams constrained by compute or product requirements, the practical ordering is: (1) regularization and early stopping, (2) output restriction, (3) rate-limiting and logging, (4) DP-SGD when the data sensitivity justifies the utility cost. Define your privacy budget before training begins — retrofitting DP after the fact requires retraining from scratch.
What do real experiments tell us about membership inference risk?
Shokri et al. on cloud classifiers
The canonical Shokri et al. shadow-model attack ran against real public cloud ML APIs. These are not worst-case theoretical numbers — they are results against production APIs in default configurations.
Synthetic health data: VUMC and All of Us
Research on membership inference against synthetic health data tested attacks against datasets derived from Vanderbilt University Medical Center (VUMC) and the All of Us Research Program. Partially synthetic data was highly vulnerable: attack precision reached very high levels for sizable subpopulations. Fully synthetic data showed substantially lower risk in those experiments. The practical implication for data-release decisions is significant: partial synthesis is not a privacy guarantee.
Summarization models
The ACL 2023 findings on privacy risks in summarization tasks show that seq2seq models leak membership signals even when the attacker has no access to reference summaries. Document-only attacks exceed random-baseline performance and can identify certain training examples with high confidence.
Translating literature results to your own model: Treat published attack accuracies as upper bounds for well-resourced adversaries, not as predictions for your specific deployment. Your model’s vulnerability depends on its degree of overfitting, the size and composition of your training set, and what outputs you expose. Run your own audit rather than extrapolating from published numbers.
What are the real privacy risks when membership inference succeeds?
The harm from a successful membership inference attack depends almost entirely on what the training dataset represents. For a model trained on public web text, confirming that a given document was in the training set is rarely sensitive. For a model trained on a disease cohort, a financial fraud dataset, or a proprietary customer database, the same confirmation can cause direct harm.
Health data is the clearest high-stakes case. Confirming that a patient’s record appeared in a training set for a psychiatric diagnosis model, an HIV-treatment classifier, or a substance-abuse-risk predictor discloses a sensitive health condition — even if no clinical details are extracted. This is a HIPAA-relevant disclosure in the United States, regardless of whether the model’s raw data is ever exposed.
Financial data carries similar risks. A model trained on fraud-labeled transaction records could reveal, through membership confirmation, that a specific individual was flagged as a fraud suspect. That disclosure could affect creditworthiness, employment, or legal proceedings.
Small-population and proprietary datasets amplify every risk. When a training set contains records from a small, identifiable group (employees of a specific company, residents of a small town, participants in a niche clinical trial), membership confirmation can effectively re-identify individuals even without extracting any features.
Recommender systems present a subtler risk. Confirming that a user’s interaction history was in the training set of a recommendation model can reveal sensitive preferences — political views, health interests, or relationship status — inferred from what the user engaged with.
The key point for practitioners: membership disclosure is a privacy harm in its own right, independent of content extraction. Regulatory frameworks in the United States, including HIPAA and the California Consumer Privacy Act (CCPA), treat certain membership disclosures as privacy violations even when no raw data is exposed. For teams working with sensitive data, the question is not only “can an attacker extract training records?” but “can an attacker confirm that a specific person’s data was used?”
How do you interpret MI test results for operational decisions?
Raw attack metrics rarely translate directly into governance decisions. Here is a practical framework for converting test results into thresholds and actions.
That is a very different risk profile than AUC alone suggests. Always compute precision and TPR at the FPR thresholds that reflect realistic adversary behavior.
For health or financial data, set the threshold lower. For public-text models with no sensitive training data, a higher threshold may be acceptable. Document the threshold and the rationale before running the audit, so the result cannot be rationalized post-hoc.
Periodic re-testing. A model that passes an audit at release may become more vulnerable as new attack variants emerge. Survey-level analysis notes that the membership inference literature has expanded rapidly, with new attack surfaces for foundation models, recommenders, and generative models requiring updated risk assessments. Schedule re-audits after significant model updates and at least annually for models handling sensitive data.
For teams that need structured governance support around threat modeling and audit workflows, engineering advisory services like Solano Advisory Group can help operationalize these thresholds within existing development processes.
What advanced and emerging attack variants should you know about?
Beyond the canonical shadow-model pipeline, several attack families have emerged that change the threat model in important ways.
Metric-based attacks skip the shadow-model training step entirely. They compute a scalar membership score directly from the target model’s output — typically the loss, the confidence gap between the top-two predicted classes, or the modified entropy of the output distribution. These attacks are fast, require no auxiliary data, and can be run with a single query per candidate record. Their precision is generally lower than shadow-model attacks, but they are practical for adversaries with limited compute or auxiliary data.
Likelihood ratio attacks compare the model’s loss on a candidate record against the loss of a reference model trained without that record. When a reference model is available (or can be approximated), this approach produces well-calibrated membership scores and tends to outperform simpler threshold methods.
Generative model attacks target diffusion models, variational autoencoders, and large language models. For generative models, the attacker typically measures whether the model assigns higher likelihood to a candidate record than to similar non-members. Research on MI against large language models shows these signals persist even at scale, though the effect sizes vary by model architecture and training procedure.
Contrastive and representation-learning-based attacks are particularly relevant for synthetic-data scenarios. The PMC research on synthetic health data shows that contrastive representation learning can strengthen attacks against partially synthetic datasets, achieving high precision even when the attacker has limited auxiliary data.
Federated learning is not a safe harbor. Gradient-based membership inference attacks can infer membership from gradient updates shared during federated training, without ever accessing the central model. This is an active research area and a practical concern for healthcare and financial institutions using federated learning for privacy.
What tools and frameworks are available for MI testing?
Several open-source frameworks make it practical to run membership inference audits without implementing attacks from scratch.
ML Privacy Meter is a Python library designed specifically for membership inference and attribute inference auditing. It implements multiple attack variants (shadow-model, metric-based, likelihood ratio) and produces evaluation reports at configurable FPR thresholds. It supports PyTorch and TensorFlow models.
Adversarial Robustness Toolbox (ART) from IBM Research includes membership inference attack implementations alongside other adversarial ML tools. It supports a broad range of model types and integrates with standard ML frameworks.
TensorFlow Privacy provides DP-SGD training utilities and includes tools for computing privacy guarantees (ε, δ) for trained models. It does not implement MI attacks directly but is the standard library for adding differential privacy to TensorFlow training pipelines.
Opacus is the PyTorch equivalent of TensorFlow Privacy. It implements DP-SGD with per-sample gradient clipping and supports most standard PyTorch model architectures.
Foolbox is primarily an adversarial-example library but includes perturbation utilities useful for implementing label-only MI attacks that rely on robustness-to-perturbation signals.
For practitioners building a testing pipeline from scratch, the practical starting point is ML Privacy Meter for attack evaluation and Opacus or TensorFlow Privacy for defense implementation. Run attacks before and after applying defenses to measure the actual reduction in leakage, not just the theoretical privacy guarantee.
What legal and regulatory obligations apply to membership inference risk?
In the United States, membership inference risk intersects with several regulatory frameworks, and the obligations depend on the type of data used in training.
HIPAA (Health Insurance Portability and Accountability Act) applies to covered entities and their business associates handling protected health information (PHI). A membership inference attack that confirms an individual’s presence in a clinical training dataset may constitute an unauthorized disclosure of PHI, even if no clinical details are extracted. The HIPAA Privacy Rule’s de-identification standards (Safe Harbor and Expert Determination methods) do not explicitly address membership inference risk from trained models, creating a gap that practitioners should address proactively.
CCPA (California Consumer Privacy Act) and its amendment, the CPRA, give California residents the right to know how their personal information is used, including in AI training. A model that leaks membership information about California residents may trigger disclosure obligations or the right to deletion — which, for a trained model, may require retraining.
FTC Act Section 5 gives the Federal Trade Commission authority over unfair or deceptive practices. The FTC has taken enforcement action against companies that failed to adequately protect consumer data used in AI systems. Membership inference vulnerabilities in consumer-facing models are a plausible enforcement target.
NIST AI Risk Management Framework (AI RMF) and the NIST Privacy Framework provide voluntary guidance for managing AI privacy risks, including membership inference. The AI RMF’s “Measure” function explicitly calls for evaluating privacy risks from model outputs.
Executive Order 14110 (AI safety and security, October 2023) directed federal agencies to develop standards for AI safety and privacy, with downstream implications for federal contractors and regulated industries.
The practical obligation for most US teams: treat membership inference risk as a data-privacy risk, not just a model-security risk. Document your audit methodology, your thresholds, and your mitigation decisions. For models handling health or financial data, consult legal counsel before release. This article provides general technical guidance and is not a substitute for legal advice specific to your deployment.
Glitchive’s perspective on membership inference governance
The pattern Glitchive sees repeatedly in documented AI failures is that privacy risks like membership inference get treated as research problems rather than engineering problems. Teams read the papers, acknowledge the risk, and then ship without running an audit — because no one owns the audit step in the release process.
The fix is structural, not technical. Membership inference testing belongs in the pre-release checklist alongside accuracy evaluation and bias auditing. The runbook in this article is a starting point, but the governance question is who runs it, when, and what happens when the results exceed the threshold. Verified case studies, like those in the Glitchive case library, show consistently that the failures with the largest downstream impact are the ones where the risk was known but the escalation path was undefined.
Failure documentation is also a better teacher than theoretical risk assessment. When you can read a specific incident, its contributing factors, and the remediation that was actually applied, you build intuition that no paper can fully convey. That is the argument for maintaining a living record of what went wrong and why, not just a static threat model.
Glitchive’s case library covers membership inference and beyond
The Glitchive case library is a free, searchable repository of verified AI failure case studies, each documenting the incident, contributing factors, technical analysis, and the specific remediation applied. For membership inference specifically, the library includes audit-ready artifacts: runbook templates, threshold-setting guidance, and pre-release checklists you can adapt directly to your stack.
Every case study carries a permanent, citable URL and fully sourced references — so you can use them as evidence in governance reviews, not just as background reading. The coding agent production incident and the chatbot liability case illustrate the depth of root-cause analysis Glitchive applies to every documented failure. Browse the full case index to find incidents relevant to your model type and data domain, then download the audit checklist to run your first membership inference test before your next release.
Pro Tip: Run the Glitchive MI audit checklist during staging, not production. Staging gives you a controlled environment to measure attack precision at low FPRs without exposing your production API to the query patterns the audit generates.
Sources
- Membership Inference Attacks Against Machine Learning Models (Shokri et al.)
- Membership inference against synthetic health data (PMC article)
- Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks
- ML04:2023 Membership Inference Attack (OWASP)
- IJCAI 2024 proceedings (membership inference evaluation guidance)
FAQ
What exactly is membership inference?
Membership inference is the task of determining whether a specific data record was used to train a machine learning model. The attacker’s output is a binary decision (member or non-member), not a reconstruction of the record’s content.
What is the difference between membership inference and attribute inference?
Membership inference asks whether a specific record was in the training set. Attribute inference assumes the record exists and tries to recover a missing sensitive field (such as a diagnosis or income) from the model’s predictions. They require different attacker knowledge and produce different types of privacy harm.
What is the difference between model inversion and membership inference?
Model inversion tries to reconstruct a representative input for a class or individual from the model’s outputs. Membership inference only confirms presence in the training set without reconstructing any content. Model inversion is a harder attack that requires more from the model’s outputs.
Can membership inference be refuted or prevented entirely?
Not entirely, but it can be reduced to operationally acceptable levels. Differential privacy at training time and strong regularization substantially reduce leakage in experiments. Confidence masking and output restriction raise the attacker’s cost but do not eliminate the risk, as label-only attacks demonstrate.
How do you know if your model’s MI test results are acceptable?
Document the threshold before testing and treat a breach as a release blocker requiring retraining or additional defenses.