Agent Evaluation

This note aggregates information about how to evaluate different agents.

• used in τ -bench
  • pass@k
    • measures the likelihood that an agent gets at least one correct solution in k attempts
  • pass^k
    • measures the probability that all k trials succeed. As k increases, pass^k falls since demanding consistency across more trials is a harder bar to clear

• Agent Behavior Standard
  • Braintrust/Basis open standard for writing behavior specs — Markdown files under .agents/behaviors/ describing intent, evidence, decision, execution, recovery for recurring agent conduct
• Behavior Specs for Supervising Long-Horizon Agents
  • 2026-07-29 — Braintrust announcement blog on process-supervision evals for hour-long agent trajectories, contrasting with outcome-only evals

• Demystifying evals for AI agents ( Link)
  • 2026-01-09