Two replies can be worded completely differently and both be right. Two refunds can look identical and one of them be wrong. That’s why we evaluate agents on decisions and actions, not text.

What an eval case contains

• A recorded trigger and the tool outputs from a real run

• The expected decision, such as eligible or not eligible

• Constraints on actions: amount, recipient, field values

• Optional rubric for tone, scored separately

Replay makes evals cheap

Because tool outputs are recorded, eval runs don’t touch production systems and finish in seconds. Teams run their full suite on every change to an agent’s instructions, tools or model.

Grow the suite from production

Every disagreement in shadow mode and every human correction becomes a new case. The best suites we see are built almost entirely from real runs.