What goes wrong in agents
- A tool is called with a wrong or invented argument.
- An intermediate answer drifts from the data a tool returned.
- The agent ignores a rule from the system prompt halfway through.
- The final answer sounds right but is built on an earlier mistake.
How we evaluate agents today
Agent trace evaluation is delivered as a Custom Pilot. We take your agent's traces (inputs, tool calls, tool results and answers), define requirements with you, and evaluate each step:
- Tool call arguments checked against the user request and your rules.
- Each intermediate answer grounded in the tool results, with the same claim-level engine as the free checker.
- Instruction adherence checked at every step, not only at the end.
- A failure taxonomy per step, so you can see where in the workflow things break.
Honest status
Self-serve agent trace evaluation is on our roadmap. Today, single answers can be checked in the free checker, and full agent evaluation is scoped inside a pilot, with human review where it is needed.
Learn about AI quality audits and RAG evaluation, or tell us about your agent below.