Agent Evaluation

This note aggregates information about how to evaluate different agents.

• used in τ -bench
  • pass@k
    • measures the likelihood that an agent gets at least one correct solution in k attempts
  • pass^k
    • measures the probability that all k trials succeed. As k increases, pass^k falls since demanding consistency across more trials is a harder bar to clear

• Demystifying evals for AI agents
  • 2026-01-09