Evaluation is the product

Agent quality is not static. Models change, source data drifts, tools evolve, and the people using the system discover new edge cases.

That makes evaluation a continuous product—not a gate passed before launch.

Build an evidence loop

Start with real tasks sampled from the workflow. Record the expected outcome, acceptable variance, and conditions that demand escalation. Then evaluate more than the final answer:

• Was the correct context retrieved?
• Were tools used in the right order?
• Did the agent cite the evidence behind its decision?
• Did it stop when authority was unclear?

The score is useful. The trace explains what to improve.

Measure the operating outcome

Model accuracy matters, but the business case lives elsewhere: cycle time, escalation quality, rework, and human attention returned to higher-value decisions.