tau3-bench-static-eval

tau3-bench-static-eval

All 164 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

Corrected τ³-bench text

Pins sierra-research/tau2-bench revision dd8757d4c8791dad7b3cdaf02a105a3d5f9d86bd, distribution 1.0.1, MIT. Includes all 50 airline and 114 retail corrected base scenarios, with original IDs, policies, databases, tools, and initial states. Scope is text half-duplex.

Seat 0 is the learner. Customer and judge identities in seats 1 and 2 remain frozen across arms. Native prompts, full histories, tool execution, 200 steps, 10 tool errors, termination, and state replay remain unchanged. EvaluationType.ALL preserves each task's reward basis. Forty retail scenarios require a batched natural-language assertion judgment. Optional conversation review remains disabled. The native parser accepts absent or empty judge results as all-true; raw replies and an explicit diagnostic preserve that source defect.

The versioned transport specifies 32,768 context tokens, 4,096 learner/judge output tokens, 1,024 customer output tokens, temperature 0, top-p 1, no stop strings, and 17-second requests. Author code leaves token and transport caps unspecified. At most 201 requests, ten seconds of grace per request, 180 seconds for connection, and 393 seconds for simulation fit the 6,000-second wall. This does not claim identical provider retry behavior.

Each episode is one native trial. Random(case_seed).randint(0, 1000000) produces the native first-trial seed, also passed to learner/customer generation. The default case seed is 300. The judge has no provider seed. Native reliability is comb(successes, k)/comb(trials, k). Shared domain databases define conservative sampling components. Replay retains the native simulation, final database hash, policy states, all seat requests and replies, and inference settings. Redundant tool names are checked before transport normalization; original native requests remain available.

Certification substitutes labeled offline controls for all three identities. The customer stops after one exchange, and the judge emits empty results; this qualifies transport, not judgment. Evaluation variants retain the configured frozen customer and judge.

Source: https://github.com/sierra-research/tau2-bench/tree/dd8757d4c8791dad7b3cdaf02a105a3d5f9d86bd