Working premise
A tool call can be syntactically valid and still be wrong for the task. Reliability requires evaluating selection, arguments, authorization, execution, retries, and the effect on the final outcome.
Operational context
Tool reliability spans the entire action path: selecting the right tool, constructing valid arguments, obtaining an authorized result, interpreting the response, recovering from failure, and confirming the intended outcome. A successful HTTP response proves only one step.
The trace should distinguish model decisions from deterministic system controls. Authorization, schema validation, idempotency, timeout, retry, and confirmation belong in the application boundary even when a model proposes the call. This separation makes failures diagnosable and prevents evaluation from rewarding unsafe autonomy.
Test cases should include tool unavailability, ambiguous intent, stale state, partial completion, conflicting results, and repeated requests. The expected behavior may be to ask for clarification, refuse, retry safely, choose an alternative, or escalate rather than simply complete the action.
Questions to resolve first
Which tools can create irreversible, external, or high-impact changes?
Where are authorization and argument validation enforced outside the model?
How does the system detect partial success and prevent duplicate action?
What confirmation proves the user's intended outcome occurred?
Recommended method
- P1
Define expected tool choices and prohibited actions for representative tasks.
- P2
Validate arguments against schema and business constraints.
- P3
Record tool status, duration, output, retry, and recovery path in the trace.
- P4
Score the full trajectory, not only each isolated call.
Common failure modes
A 200 response is counted as success even though the result is incomplete or does not match the requested outcome.
The model is trusted to decide whether the user can perform an action.
A timeout triggers a repeated external action without idempotency or state reconciliation.
Control points
- Separate model intent from actual side effects.
- Require confirmation or approval for sensitive actions.
- Detect repeated, contradictory, and cyclic tool use.
- Preserve authorization context without exposing credentials.
Implementation sequence
- S1
Map each tool's impact, authorization, schema, idempotency, timeout, and confirmation contract.
- S2
Instrument selection, arguments, validation, execution, retries, result interpretation, and final outcome as distinct evidence.
- S3
Build trajectory tests for wrong tools, malformed arguments, denied access, partial success, and ambiguous state.
- S4
Review unnecessary-call rate and recovery outcomes alongside raw tool success.
Measures worth reviewing
Report correct selection, valid arguments, authorized execution, tool response, recovery, and confirmed task completion separately. A high tool-success rate can coexist with poor task outcomes or excessive unnecessary calls.
Primary references
This note is an editorial synthesis of public standards, primary institutional guidance, and common engineering control patterns. It is intended to support engineering design and review. It is not a substitute for legal, security, safety, audit, or other professional advice for a specific system.