spin-planning-static-eval

spin-planning-static-eval

All 21 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

SPIN-Bench planning

All 1,280 released Planning Domain Definition Language tasks across 21 domains. Each variant runs its complete domain, with the author's zero-shot prompt generator, process_instance, JSON plan extraction, and VAL Plan valid scorer. VAL rejects recorded fatal parser errors before using an incomplete syntax tree, fixing the bundled validator crash on malformed released Grid tasks. Prompt and task hashes, source revision, and the actual validator binary are pinned. Exact provider bytes, extracted plans, native errors, and validator output remain in the replay. Domain tasks share one conservative sampling component.

The original OpenAI string-prompt path sends one user message and omits its wrapper's nominal sampling defaults. This reconstruction preserves that user-only prompt and declares temperature zero, top_p one, and 10,000 output tokens. It runs serially and bounds each validator invocation to 30 seconds. Timeouts, provider disconnects, and abnormal validator exits are unscored. Native exit 255 denotes input rejection, including type errors in released tasks; these retain zero scores and their diagnostics. Native parser failures retain their original error plan files and native validation outcomes.

This planning league excludes SPIN's Tic-Tac-Toe, Connect 4, Chess, Hanabi, and Diplomacy. They require separate qualification; this release does not claim their completion or paper-score parity.