Working premise
Answer scores alone cannot distinguish a retrieval failure from a generation failure. The trace should preserve each retrieval decision and the evidence available to the model.
Operational context
Retrieval quality cannot be inferred from the final answer alone. The trace needs to show how the query was formed, which stores and filters were used, what candidates were returned, which items entered model context, and whether the answer's claims are supported by those items.
Coverage and precision answer different questions. Coverage asks whether the necessary evidence was available in the retrieved set; context precision asks whether irrelevant or conflicting material crowded the model. Citation validity then tests whether the cited source actually supports the associated claim.
Permission and freshness are part of retrieval quality. A semantically relevant document may still be unauthorized, superseded, or outside the required time window. Evaluation cases should therefore include access boundaries, conflicting versions, missing evidence, and explicit abstention behavior.
Questions to resolve first
Can the trace distinguish candidates retrieved from items actually sent to the model?
Which source authority, permission, version, and freshness attributes are required?
How will unsupported claims and justified abstentions be scored?
Can the system reproduce the index and retrieval configuration used for a decision?
Recommended method
- P1
Record transformed queries, filters, retrieval method, candidate ranks, and selected context.
- P2
Evaluate source authority, coverage, relevance, freshness, and permission handling.
- P3
Separate retrieval recall from answer faithfulness and task usefulness.
- P4
Inspect missing-evidence cases and contradicted sources explicitly.
Common failure modes
The output appears plausible while required evidence was absent or the citation does not support the claim.
The retriever selects a strong match that the user or workflow was not entitled to access.
Index, chunking, embedding, reranking, or filters change without a version linked to the trace.
Control points
- Do not log restricted source content to a broader telemetry audience.
- Version index, embedding, ranking, and chunking changes with the release.
- Test access-controlled cases as part of every material retrieval change.
- Flag answers that proceed when required evidence is absent.
Implementation sequence
- S1
Instrument query construction, retrieval configuration, candidates, ranks, filters, context selection, and citations.
- S2
Build cases with known relevant, irrelevant, conflicting, restricted, and superseded sources.
- S3
Score coverage, context precision, citation entailment, permission adherence, and abstention separately.
- S4
Gate material retrieval changes and retain the index or configuration version needed for reproduction.
Measures worth reviewing
Use evidence recall and context precision together. Add citation validity, restricted-source leakage, unsupported-claim rate, and appropriate-abstention rate so better retrieval cannot be claimed from answer fluency alone.
Primary references
This note is an editorial synthesis of public standards, primary institutional guidance, and common engineering control patterns. It is intended to support engineering design and review. It is not a substitute for legal, security, safety, audit, or other professional advice for a specific system.