Working premise
Token totals are an input, not an outcome. Engineering teams need to understand which context is useful, which work is repeated, and how cache behavior affects cost and latency.
Operational context
Token cost is shaped by context assembly, repeated instructions, conversation history, retrieved evidence, tool results, provider cache rules, and output length. A task may use the same model but have very different economics depending on how much context is reused or repeatedly transmitted.
Cache reporting needs a clear denominator. A high hit rate can be meaningless if only a small portion of expensive input is eligible, while a modest hit rate on large stable prefixes may create substantial savings. Provider-reported cached tokens should be reconciled with trace-level context structure.
Optimization should preserve evidence and behavior. Removing context may reduce input cost while increasing unsupported answers, tool retries, or human escalation. The accepted unit is therefore quality-adjusted cost per successful task, with token and cache metrics explaining the drivers.
Questions to resolve first
Which context components are stable, reusable, or repeated across tasks?
How does each provider define cache eligibility, retention, and billing?
Which context growth is required evidence and which is accidental duplication?
How will a context or cache change be evaluated for quality and privacy impact?
Recommended method
- P1
Attribute input, output, cached, and reasoning tokens to individual spans and complete tasks.
- P2
Measure context growth across conversation turns and agent steps.
- P3
Identify stable prefixes and repeated retrieval that may benefit from caching.
- P4
Compare quality before and after context reduction or routing changes.
Common failure modes
The reported rate rises while the expensive, non-cacheable portion of the request continues growing.
One provider's cache semantics are treated as a universal application behavior.
Cost falls because necessary instructions or evidence are removed, increasing failure or escalation.
Control points
- Version provider pricing and caching rules.
- Do not count theoretical cache savings as realized savings.
- Monitor cache invalidation when prompts, tools, policies, or source content change.
- Protect required evidence from aggressive context trimming.
Implementation sequence
- S1
Decompose input tokens into stable instructions, examples, history, retrieval, tools, and user content.
- S2
Record eligible, written, read, and billed cache usage with provider and rate version.
- S3
Compare context and cache candidates against task success, latency, and failure segments.
- S4
Alert on repeated-context growth, falling effective savings, and routes with missing cache attribution.
Measures worth reviewing
Track tokens and cost per successful task, eligible-cache share, effective cache savings, repeated-context share, and context growth. Provider cache-hit rate should remain a diagnostic input rather than the business outcome.
Primary references
This note is an editorial synthesis of public standards, primary institutional guidance, and common engineering control patterns. It is intended to support engineering design and review. It is not a substitute for legal, security, safety, audit, or other professional advice for a specific system.