Working premise
Online evaluation trades coverage for cost and latency. The sampling policy must preserve critical segments, rare failures, and release comparisons instead of selecting traffic only by convenience.
Operational context
Online evaluation is a monitoring design problem before it is an evaluator problem. The sampling policy determines which behavior becomes visible, how quickly change is detected, and which user populations may be systematically missed. Convenient random sampling can underrepresent rare failures and high-consequence workflows.
A robust policy combines stable cohorts with targeted oversampling. Stable cohorts support comparisons across releases; targeted samples capture errors, low-confidence outputs, novel inputs, important customers, or policy-sensitive actions. The inclusion reason and probability should travel with each evaluated trace.
Sampling also creates its own cost and latency budget. Expensive evaluators may run asynchronously or only on selected strata. Severe application events should follow an independent escalation path so they are not discarded merely because a trace was not selected for routine scoring.
Questions to resolve first
Which production change or failure must the sample be capable of detecting?
Which segments are too rare or important for simple random sampling?
How will sampling differ between baseline, canary, and full rollout?
What evaluator budget and detection delay are acceptable?
Recommended method
- P1
Start with the monitoring question and the minimum detectable change.
- P2
Stratify samples by feature, environment, release, model, customer segment, and risk.
- P3
Oversample errors, low-confidence cases, novel inputs, and important customer journeys.
- P4
Maintain a stable comparison cohort during phased releases.
Common failure modes
Only easy, short, or already successful traces are evaluated, creating optimistic quality estimates.
Release comparisons use different sampling policies without adjustment or disclosure.
A severe failure is ignored because it did not enter the routine evaluator cohort.
Control points
- Record inclusion probability and sampling reason with every evaluated trace.
- Do not compare rates across cohorts with materially different sampling policies.
- Budget evaluator cost and latency separately from application inference.
- Escalate severe events even when they fall outside the routine sample.
Implementation sequence
- S1
Define the monitoring question, important strata, minimum coverage, and expected detection interval.
- S2
Attach policy version, inclusion reason, and probability to every selected trace.
- S3
Keep a stable comparison cohort during rollout and analyze targeted samples separately.
- S4
Audit sampled versus unsampled traffic for segment, latency, error, and outcome differences.
Measures worth reviewing
Coverage should be reported by segment and release alongside total sample rate. Monitor detection delay, evaluator spend, sampling-policy drift, and differences between sampled and unsampled traffic to identify selection bias.
Primary references
This note is an editorial synthesis of public standards, primary institutional guidance, and common engineering control patterns. It is intended to support engineering design and review. It is not a substitute for legal, security, safety, audit, or other professional advice for a specific system.