taubench-static-eval

taubench-static-eval

All 165 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

Original τ-bench

All 50 airline and 115 retail test scenarios retain native tools, policies, initial databases, customer prompts, 30 agent steps, state-hash checks and required-output scoring. Upstream labels these tasks outdated; corrected τ³-bench is a separate benchmark.

Each episode is one native trial with a learner in seat zero and a frozen customer policy in seat one. Initialization and reset customer calls are retained. The source's first-tool-call-only behavior, tool error handling, terminal transfers and ###STOP### marker remain unchanged. Replay retains both policies' exact requests/responses, native reward details, customer history and pre-scoring state hash. Native cost bookkeeping is zero in the transport facade, not a measurement of inference spend.

The versioned explicit-caps-v1 adaptation sets learner temperature 0, 4096 output tokens, and customer temperature 1, 1024 output tokens; both use 60-second request deadlines. Author code leaves token caps and customer temperature unspecified. This is not a claim of identical paper sampling. Provider seeds remain unset. The native Python RNG uses each case seed, default 10; repeated suite seeds identify separate trials, not deterministic customer common randomness.

Success is the official binary native reward. Repeated-trial reliability is native Pass^k, the mean of comb(successful trials,k)/comb(total trials,k), requiring all planned trials. It is not coding pass@k. Shared customer accounts define components. No training or development tasks replace test scenarios. Certification uses two explicitly offline negative controls; real evaluations must pin both policies. A no-op can pass native tasks requiring neither state changes nor specified response substrings; negative-control success is not forced to zero.

Source: https://github.com/sierra-research/tau-bench/tree/59a200c6d575d595120f1cb70fea53cef0632f6b