Working premise
Changing prompt, model, retrieval, tools, and evaluator together makes results difficult to interpret. A useful experiment isolates the decision being made and preserves enough context to reproduce it.
Operational context
AI experiments are easily confounded because prompts, models, retrieval, tools, evaluators, random seeds, and provider behavior may all vary. A useful experiment defines the decision, hypothesis, baseline, controlled variable, cases, metrics, and analysis plan before looking at results.
Paired comparisons are often more informative than independent averages because the same cases expose where behavior changed. Repeated runs may be needed for nondeterministic outputs. Row-level inspection remains essential when a small number of high-severity regressions can outweigh a broad average improvement.
Statistical significance does not establish product importance. Teams should define a minimum practically important effect and examine segment-level outcomes, latency, and cost. Results that do not change an engineering decision should be treated as exploration rather than release evidence.
Questions to resolve first
What decision will the experiment inform and what is the falsifiable hypothesis?
Which variable changes while prompts, data, tools, and evaluators remain controlled?
How much run-to-run variance is expected for the selected metrics?
What effect size is practically meaningful for the product and its critical segments?
Recommended method
- P1
Write the hypothesis and release decision before running the experiment.
- P2
Freeze the baseline and change one material factor when possible.
- P3
Use representative data and preserve important segments.
- P4
Repeat stochastic runs when variance could change the conclusion.
Common failure modes
Model, prompt, retrieval, and evaluator move together, so the cause of improvement cannot be identified.
The candidate wins globally while regressing materially on a small, important group of cases.
Metrics and acceptance rules are chosen after the team has seen which candidate looks best.
Control points
- Do not tune against the final acceptance set.
- Record provider, model, parameters, prompt version, tool schema, and evaluator version.
- Inspect row-level deltas and not only averages.
- Require a new run when material inputs change.
Implementation sequence
- S1
Write an experiment brief with hypothesis, decision, baseline, controlled variable, cases, metrics, and stop conditions.
- S2
Freeze versions and run paired comparisons with repeats where evaluator or model variance is material.
- S3
Inspect effect size, uncertainty, severity, and segment deltas before declaring a winner.
- S4
Store the full configuration and promote accepted results into the release record rather than a separate notebook.
Measures worth reviewing
Report effect size and uncertainty with segment and severity analysis. Add reproduction rate, run-to-run variance, critical regression count, latency, and cost so significance is not mistaken for operational value.
Primary references
This note is an editorial synthesis of public standards, primary institutional guidance, and common engineering control patterns. It is intended to support engineering design and review. It is not a substitute for legal, security, safety, audit, or other professional advice for a specific system.