Working premise
An evaluator can become less reliable even when its own aggregate score appears stable. Drift monitoring should compare agreement, score distribution, difficult segments, and reference cases over time.
Operational context
Evaluators drift when their model, prompt, rubric, examples, input distribution, or reference labels change. Even a fixed evaluator can become less useful as the product introduces new workflows and edge cases. Stable aggregate scores can therefore hide declining agreement at important boundaries.
Monitoring requires anchor cases with adjudicated labels and a stream of recent production cases. Anchors detect changes in the measurement system; recent cases reveal distribution shift and new ambiguity. Disagreement should be analyzed by task, severity, language, and product segment.
Not every shift requires an evaluator update. A change may indicate that the product, rubric, or expert policy has moved. Teams need an investigation path that can distinguish evaluator defect, label inconsistency, product change, and genuinely novel behavior before recalibration.
Questions to resolve first
Which anchor cases and labels are stable enough to detect evaluator change?
Which production segments are most likely to expose new disagreement?
What change threshold triggers investigation rather than automatic replacement?
Who adjudicates whether the rubric, reference label, or evaluator is wrong?
Recommended method
- P1
Maintain a frozen calibration set with expert labels and known difficult boundaries.
- P2
Rerun calibration when the judge model, rubric, prompt, or application distribution changes.
- P3
Monitor score distribution and agreement by task and segment.
- P4
Review cases where evaluator confidence and human judgment diverge.
Common failure modes
Average scores remain stable while agreement with experts deteriorates on material cases.
A provider model alias changes and historical comparisons mix different evaluator behavior.
Reference labels are changed without preserving the prior policy or decision context.
Control points
- Version every evaluator dependency.
- Keep prior evaluator results available for comparison.
- Block silent judge-model upgrades in release gates.
- Route severe evaluator uncertainty to human review.
Implementation sequence
- S1
Maintain versioned anchor cases, difficult boundaries, and recent sampled cases with label provenance.
- S2
Monitor agreement, false-pass, false-fail, abstention, and score distribution by segment.
- S3
Investigate drift with independent review before changing the evaluator or reference set.
- S4
Version recalibration decisions and rerun affected release evidence when the measurement system changes.
Measures worth reviewing
Calibration agreement and false-pass rate usually matter more than raw score stability. Track segment disagreement, anchor-case changes, reviewer consistency, and time since the last formal calibration review.
Primary references
This note is an editorial synthesis of public standards, primary institutional guidance, and common engineering control patterns. It is intended to support engineering design and review. It is not a substitute for legal, security, safety, audit, or other professional advice for a specific system.