planbench-static-eval

planbench-static-eval

All 2 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

PlanBench

Runs every task1 released one-shot prompt: mystery_blocksworld500 IDs2-501 and logistics285 IDs2-286. Uses verbatim queries and the author's chat system message. These are the original plan-bench panels, not the later600-row static leaderboard. No problem regeneration or prompt sampling. Generator/config/PDDL source is pinned to https://github.com/harshakokel/PlanBench/tree/1f4d600f4806bedbefebbbe44b168372aaf5060d.

The unchanged native text_to_plan module translates responses. The pinned bundled Linux x86_64 VAL executable checks full action applicability and final goals; success is its Plan valid outcome, matching native validate_plan. Native extraction exceptions are incorrect and retained with stderr; only hash-pinned input-dependent IndexError/ValueError sites are classified as invalid. Other infrastructure or programming errors fail the episode. Parser/VAL calls have30-second deadlines and execute outside the async event loop. VAL process exits0/1 are accepted; exit0 alone does not imply plan validity, which requires Plan valid in stdout. Signals and other exit statuses fail the episode. Empty/unparsable and incorrect plans remain in denominator. Invalid count covers parser exceptions or an empty extracted plan; nonempty invalid plans get score0. All provider bytes, extracted plans and validator diagnostics are retained. The adapter uses greedy text generation with an explicit2048-token cap.

Each one-shot example comes from the preceding instance. Because example/target links connect the panel, its rows share one conservative sampling component; individual rows are not independent samples. Other PlanBench tasks and PlanBench-XL's interactive retail environment are separate scopes.