gamebench-static-eval

gamebench-static-eval

All 2 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

GameBench

This release covers Air, Land, Sea and Santorini only. Codenames is excluded because it overlaps training. Each variant has twenty complete games: seeds 0–9, each with the learner in both seats against the unchanged native RandomAgent. Paired roles share a seed sampling component. The author scripts use ad hoc match counts and random role assignment; this frozen paired panel is an explicit sampling adaptation, not an author-released test set or paper-score reproduction.

The native mode-0 OpenAITextAgent builds all prompts, rules headings, explanations, observations and JSON actions. It retains three response attempts per decision. Air, Land, Sea additionally retries three decisions before its native random fallback. Santorini retains native random gameplay fallback. No action ranking, shortened rules, vision substitution, or hidden opponent cards enter learner prompts. Temperature 1/top-p 1 follow provider defaults; the 512-output-token cap is an explicit decoding adaptation because the source has no output cap. Session.generate replaces only model transport.

Air, Land, Sea plays complete battles until a player reaches twelve victory points and returns native normalized victory-point shares. Santorini retains its alternating-pawn rule and plays to native terminal victory. Primary win_rate is binary final victory; native_score preserves the author's distinct score unchanged. Raw generations, rejected responses, native parsed actions, visible states, fallback counts and terminal state remain auditable. Native errors fail the episode without scores.

Known native limitation: after three invalid initial placement responses Santorini assigns a raw string as its fallback, then accesses action_id and crashes. This remains an unscored source failure, with raw response evidence retained; no invented zero or replacement action is introduced. The optional offline gamebench_control player chooses the first visible legal placement, then emits invalid gameplay to qualify complete Santorini runs; it is a test control only, never the model policy or scorer.

Reproducibility uses SHA256-derived case/game/panel seeds with one isolated shared native random stream per game. The loader gives Rules an identity hash to preserve the original Python 3.10 shared dataclass default semantics under newer Python; it does not change rule contents. Imports are isolated and credential-loading/provider initialization is omitted. SantorinAI 1.3.3's exact board/pawn code is pinned by wheel and file hashes; unused GUI dependencies are omitted.

GameBench: https://github.com/Joshuaclymer/GameBench/tree/2968355fcf3d9a0870e4ff9f53f201ca404c3415. Copyright 2024 Anthony Costarelli, MIT. SantorinAI: Tom Mansion, Apache-2.0. Both licenses and upstream README attribution are retained in the image. No match-result dataset is used.