spin-hanabi-static-eval

spin-hanabi-static-eval

All 4 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

SPIN-Bench native games

Tic-Tac-Toe runs ten original solver games, reversing learner roles halfway through. Hanabi runs five complete games per 2/3/4/5-player cohort. One evaluated policy controls every Hanabi seat through separate native histories, matching the author's same-model experiment. The complete panel is one sampling task. Original prompts, retries, parsers, game loops, and scorers are retained. Tic-Tac-Toe reports learner win rate, draw rate, and native best-move fraction. Hanabi reports the author's last-pre-action reward and terminal reward separately; these differ when the last action loses the game.

The native OpenAI history path forwards exact messages but omits nominal sampling defaults. This downstream profile explicitly uses temperature zero, top_p one, and 10,000 output tokens. Python and NumPy use the suite seed; the native competitive environment retains seed 42. Incomplete panels, parser/scorer crashes, and provider disconnects are unscored. The bundled legal-action policy is an offline wiring control, not a model-quality result. Connect 4, Chess, and Diplomacy require separate adapters and qualification.