Experimentation System Audit: Can You Trust Your A/B Test Results?

Quick answer: Treat an A/B result as decision-ready only when an end-to-end audit verifies that the intended units were eligible, assigned to a stable variant, meaningfully exposed, measured with the right business outcome and denominator, and analyzed with a method that matches the design; treat a precise lift as untrusted when uncertainty or upstream bias remains.

An experimentation system audit determines whether the machinery producing an A/B test result preserves a valid comparison from assignment through decision.

A result is trustworthy only when the correct units were eligible, randomly assigned, kept in a stable variant, exposed to the intended experience, measured with valid outcomes and denominators, and analyzed with a statistical method that matches the design and stopping rule.

The lift estimate is the last output of that chain. If assignment, identity, exposure, event collection, or analysis is biased, a precise-looking result can still be wrong.

A conversion tracking audit asks:

Can I trust the events and funnel data telling me what users did?

An experimentation system audit asks:

Can I trust the system telling me that one variation caused a different outcome?

The second depends on the first, but adds randomization, allocation, persistence, exposure, contamination, sample-ratio, and inference checks.

The experimentation trust chain

A trustworthy experiment preserves the intended comparison across seven linked stages.

Stage | Required condition | Typical failure
Eligibility | The intended population can enter the experiment | Existing customers enter a new-user test
Assignment | The correct unit is randomized according to the configured allocation | Sessions are randomized while analysis treats users as independent
Persistence | The assigned unit remains in the same variant | Anonymous users change variants after login
Delivery and exposure | The unit receives and reaches the intended treatment | A flag call counts users who never saw the tested experience
Outcome measurement | The metric represents the intended business result for eligible units | All exposed users are used as the denominator for a trial-start metric
Analysis | The estimator, variance, stopping, and multiplicity rules match the design | A clustered experiment is analyzed as independent users or stopped on a lucky day
Decision | Effect size, uncertainty, guardrails, and deployment reality are considered | A proxy metric wins while downstream revenue deteriorates

The audit should record evidence at every stage. A platform screenshot showing a 95% probability or a small p-value is evidence only about the final analysis step, and even that is conditional on everything upstream being correct.

1. Define the population, estimand, and randomization unit

Before inspecting a feature flag, define the decision the experiment is intended to support.

The audit should specify:

• the eligible population;
• the first moment a unit becomes eligible;
• the randomization unit;
• the analysis unit;
• the treatment and control experiences;
• the primary outcome;
• the conversion window;
• the effect being estimated; and
• the decision rule.

Eligibility is part of the causal question

Suppose an onboarding experiment is intended for new users who have not started a trial. If existing subscribers, users who already started on the web, or people on unsupported app versions enter the denominator, the metric no longer answers the intended question.

Eligibility should be determined using information available before treatment whenever possible. A rule that depends on behavior caused by the variation can select different populations into the two arms and break the comparison.

The randomization unit must match the product reality

Common units include:

• user;
• account;
• browser or device;
• session;
• household;
• organization or tenant;
• store, market, or region; and
• time period.

The analysis must respect that unit and any clustering it creates. Randomizing organizations and then treating every employee as an independent observation can understate uncertainty because behavior within an organization is correlated. Microsoft Research has documented the variance and power problems that arise in tenant-randomized experiments. (Tenant-randomized A/B testing)

2. Audit assignment, bucketing, rollout, and sample-ratio mismatch

Assignment should be reproducible, stable, and independent of the outcome.

The audit should verify:

• which identifier is hashed or bucketed;
• whether the same identifier always receives the same assignment;
• whether identifiers are unique enough for the intended unit;
• whether allocation weights remained stable during the analysis period;
• whether rollout percentage was separated from variant allocation;
• whether forced overrides or targeting rules introduced selection;
• whether assignment logs exist independently of outcome logs; and
• whether observed counts match the configured ratio.

Equal allocation is a default, not a law

For two variants with similar cost and variance, a stable 50/50 allocation usually provides the most statistical information for a fixed total sample.

A prespecified and stable 70/30 allocation is not automatically invalid. It may be used for risk, cost, or operational reasons. The danger is changing weights during the test without accounting for time, or using unequal variant weights as a substitute for product rollout.

If Monday’s traffic is assigned 80/20 and Friday’s traffic 50/50, differences in weekday behavior or channel mix can become unevenly represented across variants. Use rollout to control how much total traffic enters the experiment; keep the comparison allocation stable unless the design explicitly supports adaptation.

Check sample-ratio mismatch before interpreting lift

Sample-ratio mismatch, or SRM, occurs when observed variant counts differ from the configured allocation by more than ordinary random variation would explain.

Do not judge SRM by visually comparing 49% and 51%. The same percentage difference may be unsurprising in a small test and nearly impossible in a very large one. Use an appropriate statistical goodness-of-fit test, commonly chi-square or an exact alternative for small counts.

Microsoft’s experimentation practice treats SRM as a data-quality symptom that must be diagnosed before experiment effects are trusted. Root causes can occur in assignment, execution, logging, joins, or analysis, and an SRM can reverse a ship decision. (Diagnosing sample-ratio mismatch)

Check SRM at more than one stage:

• eligible units;
• assigned units;
• treatment delivered;
• meaningful exposures;
• units with the primary outcome available; and
• important pre-treatment segments.

A clean assignment ratio with a broken exposure ratio can reveal a delivery or logging problem rather than a bucketing problem.

3. Audit identity persistence and variant crossover

The assigned unit must remain identifiable through the outcome window.

This is especially difficult when an experiment begins before signup and the primary outcome occurs after authentication, payment, app installation, or a return visit.

Audit:

• anonymous ID creation;
• identify and alias behavior;
• login, logout, and reset behavior;
• device or cookie replacement;
• app reinstall;
• cross-domain movement;
• cross-device use;
• server events arriving without the original identifier; and
• the treatment used when one person has multiple assignment records.

Track at least three identity diagnostics:

1. Crossover rate: the proportion of experiment units observed in more than one variant.
2. Unlinked outcome rate: eligible outcomes that cannot be joined to an assignment.
3. Identity transition retention: the proportion of pre-auth assignments that preserve the same variant after authentication.

A low aggregate crossover rate can still conceal a severe issue in a high-value path, such as users who convert after logging in.

4. Separate assignment, treatment delivery, and meaningful exposure

These are different events.

• Assignment means the system selected a variant for an eligible unit.
• Delivery means the product rendered or executed the assigned branch.
• Exposure means the user reached the first point where the treatment could affect behavior.

A feature flag evaluation is not automatically a valid exposure event.

If an app fetches every flag on startup, polls flags repeatedly, or evaluates a flag for users who cannot enter the tested flow, a $feature_flag_called event can overstate exposure. The experiment denominator then contains people who never had a chance to experience the change.

The strongest exposure event is semantic and close to the treatment boundary: the first screen, component, offer, or decision point where control and test diverge.

Preserve intent-to-treat as the primary safety analysis

An intent-to-treat analysis compares units according to their original assignment. It preserves the benefit of randomization even when not every assigned unit receives or uses the treatment.

A triggered or exposed-only analysis can improve sensitivity, but the trigger must not be caused by the treatment. If treatment changes whether an event is logged or whether a user reaches the trigger, conditioning on that event can select different populations into the arms.

A good audit keeps assignment and exposure analyses separate and explains why any triggered population is valid.

5. Audit contamination, interference, and concurrent experiments

An experiment is contaminated when units receive elements of multiple variants or when one unit’s treatment affects another unit’s outcome.

Audit for:

• users switching variants;
• cached treatment content;
• shared accounts or devices;
• server and client disagreeing about assignment;
• control users receiving treatment assets;
• staff or QA overrides entering production analysis;
• variants changing during the experiment;
• overlapping experiments on the same component or funnel; and
• spillovers between users, accounts, or markets.

Mutual exclusivity should be deliberate, not automatic

Concurrent experiments do not always invalidate one another. Large experimentation programs often assign users independently into multiple tests, and many combinations do not produce meaningful interactions. Microsoft has argued against assuming that every overlap is dangerous. (A/B interactions: a call to relax)

Use mutual exclusion when:

• experiments change the same element or mechanism;
• one treatment changes eligibility or exposure for another;
• the business cannot interpret an interaction;
• traffic is too limited to estimate combinations; or
• product risk requires a clean isolated rollout.

Document concurrent tests and inspect interaction risk. Do not solve every overlap by splitting traffic until all experiments lose power.

A bundle test validates the bundle

If one variation changes signup placement, onboarding questions, pricing copy, and navigation, the result can support a decision about that combined experience. It cannot identify which sub-change caused the movement.

The audit should align the causal claim with what was actually randomized.

6. Audit conversion events, metric definitions, denominators, and windows

A correct experiment assignment can still produce the wrong conclusion if the outcome is poorly defined.

For each primary and guardrail metric, verify:

• event semantics and deduplication;
• numerator and denominator;
• eligibility for the outcome;
• attribution and conversion window;
• delayed and late-arriving events;
• missing data;
• unit of analysis;
• aggregation method;
• value, currency, and refund treatment;
• sensitivity to repeated heavy users;
• downstream business meaning; and
• consistency across variants.

Denominators must match the business question

Examples:

• Trial start: users actually eligible to start a trial.
• Trial conversion: trial starters with sufficient maturity to convert.
• Renewal: subscriptions that reached a renewal opportunity.
• Onboarding completion: users who entered onboarding.
• Step conversion: users who reached the previous step.
• Purchase: users or sessions eligible to purchase, depending on the estimand.

Using all flag exposures as the denominator because the platform makes that easy can answer a different question from the one the business intended.

Choose metrics that can survive downstream scrutiny

The nearest event is often more sensitive but less valuable.

99ways has published an experiment in which easier application questions increased form submissions by 56.7% while downstream lead quality and purchase performance deteriorated. The experiment was functioning; the proxy metric was not sufficient for the business decision. (Form Design That Sells)

A metric audit therefore asks whether a declared win remains a win after qualification, attendance, purchase, retention, refunds, or contribution are considered.

7. Audit the statistical analysis and decision process

Statistical software can calculate exactly what it was configured to calculate and still answer the wrong question.

The audit should verify:

• effect measure: absolute difference, relative lift, ratio, mean, quantile, or another estimator;
• variance estimator and independence assumptions;
• randomization and analysis units;
• cluster or repeated-user handling;
• sample-size or sequential monitoring plan;
• minimum detectable or practically meaningful effect;
• confidence or credible interval construction;
• multiple metrics and multiple variants;
• segment analyses;
• missing and delayed outcomes;
• SRM gating;
• stop-loss or safety rules;
• final stopping decision; and
• whether the observed treatment was deployed as analyzed.

Do not use fixed-horizon inference with opportunistic stopping

Ordinary fixed-horizon p-values and confidence intervals assume that the sample size or stopping rule was set independently of the emerging result. Repeatedly checking and stopping when the result looks favorable inflates false positives.

Use a prespecified fixed horizon, or a valid sequential method designed for continuous monitoring. Research on always-valid inference formalizes p-values and confidence intervals that remain valid under supported sequential decisions. (Always Valid Inference)

Match variance to the randomization structure

Naive independent-and-identically-distributed assumptions can understate uncertainty when the randomization is clustered, users contribute repeated observations, or treatment effects vary substantially. Microsoft Research has documented this failure mode in online experiments. (Trustworthy analysis of online A/B tests)

Interpret the metric, not only the threshold

A statistically reliable movement can still be irrelevant, transient, or harmful to another outcome. Microsoft’s “Dirty Dozen” paper documents recurring ways teams misinterpret experiment metrics even when the calculations themselves are correct. (Metric interpretation pitfalls)

The Zero Experiment: a production A/A calibration test

A Zero Experiment is an A/B test in which the control and test experiences are intentionally identical.

There is no intended difference in product behavior, design, copy, offer, or user experience. The experiment still runs through the production system:

• eligibility;
• assignment;
• bucketing;
• persistence;
• treatment delivery;
• exposure logging;
• conversion measurement;
• data processing; and
• statistical analysis.

The expected treatment effect is zero.

The observed estimate will not remain exactly at zero. Sampling variation moves it above and below zero. The diagnostic question is whether the system behaves like a valid null experiment, or manufactures a persistent or implausible difference between identical experiences.

The established name for this design is an A/A test. Microsoft describes A/A testing as a recommended end-to-end check of an A/B testing system and notes that allocation imbalance or unexpected metric movement can expose telemetry and data-pipeline defects. (Data quality for trustworthy A/B testing)

99ways uses Zero Experiment as the operational name for a production A/A test with a prespecified calibration plan.

What a Zero Experiment should test

A useful Zero Experiment has three diagnostic objectives.

1. Detect systematic arm-specific bias

If identical experiences consistently produce materially different outcomes, investigate the system before trusting real tests.

Potential causes include:

• incorrect bucketing;
• unstable or duplicated randomization IDs;
• unequal or changing allocation;
• assignment loss after login or reset;
• one variant receiving different caching or delivery behavior;
• asymmetric exposure logging;
• event loss or duplication in one arm;
• sample contamination;
• source, device, market, or time composition imbalance;
• different conversion windows or joins;
• analysis filters that depend on post-assignment behavior; and
• branch-specific implementation that was assumed to be equivalent.

A Zero Experiment should record counts at each stage:

Stage | Control | Zero variation | Diagnostic question
Eligible |  |  | Did both arms draw from the same intended population?
Assigned |  |  | Does allocation match the configured ratio?
Delivered |  |  | Did the system execute the intended branch?
Meaningfully exposed |  |  | Did both arms reach the treatment boundary at similar rates?
Outcome observable |  |  | Is measurement availability symmetric?
Converted |  |  | Is the remaining difference compatible with the null and practical bounds?

Do not inspect only the final conversion rate. The stage at which the arms diverge narrows the root cause.

2. Characterize noise and convergence in the environment

Track the treatment-effect estimate and uncertainty over a prespecified period.

For a binary conversion metric:

Absolute difference

Delta = conversion rate in zero variation - conversion rate in control

Relative lift

Relative lift = (zero-variation rate / control rate) - 1

For calibration, absolute difference is often easier to interpret. Relative lift can look extreme when the baseline conversion rate is very small.

A cumulative estimate does not need to move monotonically toward zero. It may cross zero repeatedly. A defensible pattern is:

• no persistent directional drift;
• no unresolved SRM;
• uncertainty narrows as information accumulates;
• daily or cohort results do not reveal a consistent arm-specific defect;
• the final interval is compatible with zero; and
• when sufficiently powered, the interval falls within a prespecified range of effects considered operationally negligible.

The last condition is stronger than merely failing to find significance.

3. Diagnose low information density and unstable populations

A Zero Experiment can reveal that raw traffic volume overstates the information available for experimentation.

Warning signs include:

• very few eligible conversions despite high session volume;
• repeated or non-independent activity from the same users, accounts, bots, or devices;
• long and highly variable conversion delays;
• unstable source, market, device, or platform composition;
• a large share of assigned users never reaching meaningful exposure;
• identity fragmentation that turns one person into several units;
• rare outcomes that leave intervals wide for a long time; and
• large segment differences that make one aggregate estimate difficult to generalize.

Call this low information density, not automatically low-quality traffic.

Low-intent acquisition can reduce the conversion information contributed by each visitor. But cheap traffic is not intrinsically invalid, and heterogeneity does not automatically bias a properly randomized comparison. It can increase variance, reduce power, and make the aggregate result less representative of high-value segments.

Possible responses include:

• defining eligibility more closely around the decision population;
• diagnosing invalid, bot, internal, or duplicated traffic;
• stratifying or blocking assignment on important pre-treatment variables;
• reporting prespecified source, device, market, or platform segments;
• separating materially different funnels;
• improving identity and exposure measurement;
• using a higher-frequency valid metric for system health while retaining the business outcome for decisions; and
• increasing duration or sample only when the population and measurement remain stable.

Do not exclude inconvenient segments after seeing the result merely to make the experiment look cleaner. Eligibility and exclusions should be justified before treatment or treated as exploratory.

What does convergence around zero actually mean?

“The line eventually looked close to zero” is not a sufficient criterion.

Use four layers of interpretation.

1. Point estimate

The estimate should not show persistent, repeatable directional displacement across the full run and major prespecified periods.

A small point estimate alone is weak evidence when uncertainty is wide.

2. Confidence or credible interval

The interval should contract as independent information accumulates and should remain compatible with zero.

An interval that contains zero does not prove the effect is absent. It may simply mean the test is underpowered.

3. Practical equivalence

Before launch, define an equivalence range [-delta, +delta]: the largest difference that would be operationally negligible for the calibration purpose.

In a frequentist design, evidence for practical equivalence can be established when the relevant confidence interval falls entirely inside those bounds, or through an equivalent two-one-sided-tests procedure. Equivalence-testing research emphasizes that a nonsignificant result is not the same as evidence that meaningful effects are absent. (Equivalence Tests: A Practical Primer)

A Bayesian implementation can use a prespecified region of practical equivalence and report how much posterior probability falls inside it. The principle is the same: define what “close enough to zero” means before seeing the data.

The equivalence margin is not universal. It depends on metric scale, baseline rate, traffic, decision risk, and the size of bias that would damage future tests.

4. Repeated null calibration

One Zero Experiment is one realization of sampling noise. It cannot estimate the platform’s false-positive rate reliably.

To assess calibration, use many independent or approximately independent null comparisons, such as:

• repeated A/A experiments over time;
• A/A/A or multi-arm null splits;
• historical re-randomization or pseudo-experiments;
• shadow assignments on representative traffic; or
• a library of commissioning tests across distinct surfaces and metric classes.

Across repeated valid null tests:

• effect estimates should be centered near zero;
• standardized effects should have the expected spread under the analysis model;
• confidence intervals should achieve approximately their intended coverage;
• p-values should behave consistently with the null after accounting for discreteness and analysis rules; and
• the observed false-positive rate should be compatible with the declared threshold and multiplicity policy.

With many metrics, some nominally significant null movements are expected by chance. A single significant A/A metric is a reason to investigate, not automatic proof of a broken platform. Persistence, concentration in one layer or segment, SRM, and repetition make the evidence stronger.

A Zero Experiment protocol

Step 1: Define the system path being calibrated

Specify:

• product surface;
• eligible population;
• randomization unit;
• assignment service;
• identity transitions;
• exposure event;
• primary and diagnostic metrics;
• conversion windows;
• analysis method; and
• data destinations.

A site-wide label does not validate every architecture. A web checkout, mobile onboarding flow, server-side subscription event, and account-randomized B2B feature may need different calibration runs.

Step 2: Prespecify the null and practical bounds

State that the intended treatment effect is zero.

Define:

• the primary calibration metric;
• absolute and relative effect reporting;
• the equivalence margin where feasible;
• SRM threshold or platform rule;
• fixed-horizon or sequential monitoring method;
• minimum calendar coverage;
• diagnostic segments; and
• pass, investigate, rerun, or fail criteria.

Do not choose the equivalence margin after seeing the apparent difference.

Step 3: Make the experiences genuinely identical

Control and zero variation should produce the same intended experience.

There are two useful implementations:

1. Pure calibration: both labels route through the same rendering and business logic. This isolates assignment, logging, metric, and analysis behavior.
2. Path-parity calibration: distinct experiment branches are intended to render identical output. This exercises more delivery plumbing, but any latency, flicker, caching, or execution difference becomes a real treatment difference and should be treated as a defect rather than dismissed as noise.

Document which form is being used.

Step 4: Log assignment and exposure separately

Retain an immutable assignment record if possible. Then record delivery and meaningful exposure independently.

For each unit, the audit should be able to determine:

• eligibility time;
• assigned variant;
• assignment identifier and version;
• delivery status;
• first meaningful exposure;
• identity transitions;
• outcome availability; and
• conversion or metric value.

Step 5: QA both variants before launch

Force each label and confirm:

• identical visible experience;
• identical product behavior;
• equivalent performance within the intended tolerance;
• correct assignment and exposure events;
• no double rendering or flicker;
• stable identity through login or signup;
• correct conversion events and properties;
• internal/test traffic rules; and
• raw logs agree with the platform’s displayed variant.

Step 6: Monitor data quality without opportunistic ship decisions

Monitor:

• SRM at assignment and exposure;
• crossover;
• missing outcomes;
• duplicate events;
• exposure-to-assignment rate;
• metric trajectories;
• source, device, market, and version composition;
• latency and conversion-delay distribution; and
• guardrail differences.

Use the prespecified stopping rule. A calibration test can still be invalidated by peeking and stopping when it happens to look reassuring.

Step 7: Diagnose before declaring pass or fail

If the test shows a difference:

1. Check assignment and SRM.
2. Compare assignment, delivery, exposure, and outcome availability by arm.
3. Inspect identity changes and crossover.
4. Verify event payloads, missingness, and duplicates.
5. Check time, source, device, market, and version composition.
6. Inspect branch-specific performance and errors.
7. Reconcile raw logs with the experiment platform.
8. Confirm the analysis and stopping rule.
9. Rerun after remediation when the root cause is material.

What a Zero Experiment validates

A well-designed Zero Experiment can provide evidence that, for the tested population, surface, period, metrics, and analysis:

• allocation behaves as configured;
• gross sample-ratio mismatch is absent;
• assignment is sufficiently stable;
• identity transitions do not create obvious arm-specific crossover;
• delivery and exposure logging are symmetric;
• outcome availability is not materially different by arm;
• the telemetry and data pipeline do not systematically favor one label;
• the analysis returns a result compatible with the null;
• uncertainty behaves plausibly as information accumulates; and
• the environment’s noise and conversion-delay characteristics are understood better than before.

This is valuable. It is also bounded evidence.

What a Zero Experiment cannot validate

A successful Zero Experiment does not prove that:

• both arms measure the true business outcome correctly if the same error affects them equally;
• actual treatment code will render, perform, or log correctly;
• the selected metric is strategically meaningful;
• the experiment has enough power for the effect a future treatment may produce;
• the variance model is calibrated across all metric types and randomization units;
• every segment, market, browser, device, or app version is valid;
• cross-device identity is complete if the Zero Experiment did not exercise it;
• concurrent real experiments will not interact;
• novelty, learning, or long-term effects are absent;
• the future population will behave like the calibration population;
• the platform will remain correct after the next SDK, consent, identity, or pipeline change; or
• a real variant’s business effect will persist after deployment.

The most important limitation is symmetry: if both identical arms undercount purchases by the same 20%, the Zero Experiment can converge perfectly around zero while the conversion tracking remains wrong.

That is why the Conversion Tracking Audit and the Experimentation System Audit are complementary rather than interchangeable.

How often should Zero Experiments be run?

Use three triggers.

1. Commissioning

Run a Zero Experiment before relying on a new experimentation platform, assignment path, identity model, exposure event, metric pipeline, or statistical service.

2. Material system change

Rerun after changes to:

• SDK or feature-flag implementation;
• randomization or hashing;
• identity and aliasing;
• authentication flow;
• consent behavior;
• cross-domain architecture;
• exposure logging;
• data warehouse joins;
• event deduplication;
• statistical engine;
• bot/internal-traffic filtering; or
• the primary conversion definition.

3. Periodic calibration

For an active high-stakes program, maintain periodic null checks or shadow calibration rather than assuming an old commissioning test still applies.

The cadence should follow change rate and decision risk, not a universal monthly or quarterly rule. Continuous automated data-quality checks can supplement full production A/A runs.

Should a Zero Experiment run across the entire site?

Not necessarily.

Choose representative surfaces that exercise the system paths on which decisions depend.

Useful candidates include:

• a high-volume web funnel;
• a pre-auth to post-auth journey;
• a cross-domain checkout;
• a mobile onboarding path;
• a server-side subscription or purchase outcome;
• an account- or tenant-randomized feature; and
• a delayed conversion metric.

A single high-volume landing-page A/A test does not validate a mobile app experiment whose identity changes at signup and whose outcome arrives from a billing webhook.

A 99ways audit pattern: why configuration review and a Zero Experiment are both needed

In an anonymized subscription-app audit, the visible experiment was directionally usable, but the trust chain had several weaknesses:

• assignment began before signup but persistence across authentication was not enabled;
• feature-flag evaluations may have counted users who never reached onboarding;
• the experiment denominator included users who were not eligible for trial-start outcomes;
• variant weights had been used as a rollout mechanism;
• temporary QA targeting remained in production configuration; and
• the variation bundled several product changes while other experiments overlapped the same limited population.

A Zero Experiment could have helped detect:

• allocation mismatch;
• assignment instability;
• exposure asymmetry;
• arm-specific missing outcomes; and
• unexpected null metric movement.

It would not, by itself, have proved that the trial-start denominator was conceptually correct, that the bundled treatment could support component-level claims, or that overlapping real experiments would not interact.

The static audit defines the intended causal design. The Zero Experiment tests whether several important parts of that design behave correctly in production. Both are required.

The experimentation system audit scorecard

Audit layer | Evidence required | Decision risk if wrong
Population and estimand | Eligibility rule, treatment definition, outcome, unit, window | The result answers a different question from the business decision
Assignment | Hashing or randomization logic, immutable logs, expected ratio | Selection bias or untraceable allocation
Rollout and weighting | Change history, stable allocation, overrides | Temporal composition confounds the arms
Identity persistence | Anonymous/auth trace, crossover and unlink rates | The same person changes variants or loses outcomes
Delivery | Forced-variant QA, branch verification, performance | Users receive the wrong or incomplete treatment
Exposure | Semantic boundary, assignment/exposure counts | Denominator includes users who never experienced the test
Contamination and interference | Overlap map, cache/override checks, mutual-exclusion policy | Effects cannot be attributed to the intended treatment
Outcome integrity | Event QA, denominator, dedupe, value, window | The metric does not represent the business result
SRM and missingness | Statistical ratio checks by stage and segment | Compared populations are disproportionate
Statistical analysis | Estimator, variance, stopping, multiplicity, clustering | Uncertainty and significance are wrong
Decision process | Prespecified rule, guardrails, analysis record | Teams cherry-pick or overstate the result
Calibration | Zero Experiment protocol and repeated-null evidence | The platform has not demonstrated null behavior

The working scorecard should also include status, severity, evidence, owner, remediation, validation method, and retest date.

What an experimentation system audit should deliver

A serious audit should produce:

1. An experimentation architecture map from eligibility through analysis.
2. A population and metric specification defining the estimand, unit, denominators, and windows.
3. Assignment and identity evidence including ratio, persistence, crossover, and unlinked outcomes.
4. Exposure validation showing where meaningful treatment begins.
5. A concurrency and contamination map for overlapping tests and shared surfaces.
6. An SRM and data-quality report at assignment, exposure, and outcome stages.
7. A statistical-method review covering variance, stopping, multiplicity, and practical effect thresholds.
8. A Zero Experiment protocol and result for representative production paths.
9. A prioritized remediation plan with owners and acceptance criteria.
10. A launch and analysis checklist for future experiments.
11. A decision-readiness statement explaining which experiment types can currently be trusted and which require repair or recalibration.

Frequently asked questions

What is an experimentation system audit?

An experimentation system audit examines whether eligibility, assignment, bucketing, identity, treatment delivery, exposure, outcome measurement, statistical analysis, and decision rules preserve a valid causal comparison.

It audits the system producing the result, not only the variation being tested.

What is a Zero Experiment?

A Zero Experiment is 99ways’ operational term for a production A/A calibration test. Control and test are intentionally identical, so the expected treatment effect is zero.

The full production system still assigns, exposes, measures, and analyzes users. Unexpected allocation or metric differences can reveal system defects.

Is a Zero Experiment just an A/A test?

The underlying design is an A/A test. The term Zero Experiment names a stricter operating protocol: production execution, prespecified zero effect, explicit equivalence or investigation criteria, stage-level diagnostics, and a written statement of what the result does and does not validate.

Use both terms together so experienced practitioners immediately recognize the design.

What should a valid A/A test show?

It should show no unresolved sample-ratio mismatch, stable assignment, symmetric delivery and exposure, no material arm-specific missingness, and outcome differences compatible with the null.

When sufficiently powered, the uncertainty interval should fall inside a prespecified range of differences considered operationally negligible.

Does a confidence interval containing zero mean the system passed?

No. A wide interval can contain zero and still permit a large harmful bias.

A stronger criterion is practical equivalence: define an acceptable range around zero before launch and require enough information for the interval to fall within it. Also require the assignment, exposure, identity, and data-quality checks to pass.

How do you distinguish normal sampling noise from systematic bias?

Normal noise produces fluctuations whose magnitude is compatible with the estimated uncertainty and does not create persistent arm-specific patterns, unresolved SRM, crossover, or segment-specific missingness.

Systematic bias is more likely when the difference persists in one direction, repeats across null runs, begins at a specific pipeline stage, concentrates in one segment or version, or coincides with allocation, identity, exposure, or logging defects.

Can a Zero Experiment detect sample-ratio mismatch?

Yes. It should test whether observed assignment and exposure counts match the configured allocation after accounting for sample size.

SRM is a symptom, not a diagnosis. The root cause may be bucketing, unstable IDs, differential delivery, missing logs, bad joins, or biased analysis conditions.

Can one Zero Experiment estimate the false-positive rate?

No. One null test provides one realization.

False-positive calibration requires many null comparisons or repeated pseudo-experiments. A single Zero Experiment can reveal obvious anomalies and provide a commissioning check, but it cannot establish long-run coverage or false-positive behavior by itself.

What happens when conversions are rare?

The business outcome may remain too uncertain to establish practical equivalence within a reasonable period. Relative lift will also be unstable when the baseline rate is near zero.

Use higher-frequency system-health metrics for assignment, exposure, and data-pipeline diagnostics, while retaining the rare business outcome as the final decision metric. Do not treat a wide, nonsignificant result as a pass.

Does slow convergence mean the traffic is low quality?

Not necessarily.

Slow convergence can result from rare outcomes, long conversion delays, repeated or clustered users, unstable segment mix, identity fragmentation, exposure dilution, bots, invalid traffic, or genuine behavioral heterogeneity. Low-intent traffic can reduce information per visitor, but price or intent alone does not make traffic invalid.

Diagnose information density and population stability before changing eligibility.

How often should Zero Experiments be run?

Run one when commissioning a new experimentation path, after material changes to assignment, identity, consent, exposure, metrics, or analysis, and periodically for high-stakes programs whose infrastructure changes frequently.

Use change rate and decision risk to set cadence.

Should a Zero Experiment run across the entire site?

No single run validates every surface.

Choose representative high-volume paths that exercise the identities, domains, apps, server events, delays, and randomization units used by important real experiments. Several targeted calibration tests are often more informative than one generic site-wide split.

What can a successful Zero Experiment not validate?

It cannot detect errors shared equally by both arms, prove the business relevance of a metric, validate actual treatment code, guarantee power, eliminate interaction or external-validity risk, or certify every future experiment.

It is a bounded calibration test, not a universal quality certificate.

How does an experimentation system audit differ from a conversion tracking audit?

A conversion tracking audit validates whether events, identities, sources, and funnel outcomes record what actually happened.

An experimentation system audit validates whether random assignment and the subsequent measurement and analysis support a causal comparison. It adds bucketing, persistence, exposure, SRM, contamination, stopping, and inference checks.

The next useful decision

If your team is running A/B tests but cannot independently verify who was assigned, who was actually exposed, whether the split held, and whether the metric and analysis match the decision, the next test should not be another treatment.

It should be an audit of the experimentation system, including a properly specified Zero Experiment where the architecture and traffic support one.

See how 99ways approaches managed CRO and experimentation.