clembench-referencegame-static-eval

clembench-referencegame-static-eval

All 180 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

clembench English text games

Complete current English instances from the pinned release: Taboo 60, Referencegame 90, PrivateShared 50, Textmapworld main 50. Two-model games pair both learner roles against one explicit baseline partner; PrivateShared's questioner and Textmapworld's describer remain native programs. Four releases cover 400 role variants over 250 source instances. Historical, translated, graphreasoning and specificroom files are outside this explicit subset. No historical model-score reproduction claim.

Primary completed_success is explicitly derived: 1 only for non-aborted native Success 1, otherwise 0. It is not native clemscore. Played is 1-Aborted. Native Success is nullable because Referencegame sets it to NaN on abort. Undefined native Main Score remains null in aggregate_input and original NaN in native_scores_json. Native clemscore uses per-game quality and played means, rounded then macro-averaged and multiplied. Report ordered roles separately and require every expected task; reversed roles lack the two single-model games.

Native clemcore 3.7.2 OpenAIModel normalization, stripping and exception retries execute unchanged, with exact raw provider responses retained. Versioned native-openai-pseudo-label-omission-v1 removes only the unsupported pseudo bookkeeping label from provider wire messages; original labeled messages remain in native_provider_messages. Role/content bytes are otherwise unchanged. Taboo's verified import-time downloader is replaced by pinned offline stopwords provisioning. Native rules/parsers/scorers and all game retries remain unchanged.

Temperature 0, top-p 1, output 300, context 32,768, 10-second provider timeout and 6,000-second episode cap are frozen. PrivateShared's largest game has 255 ordinary calls and 1,215 with all five probe attempts, before provider exception retries; worst-case completion within the wall cap is not guaranteed. Transport failure or timeout is visible and unscored, preserving evidence and making complete comparisons unavailable. Native format aborts retain their exact native Success, including undefined values, and quality.

Certification uses explicit scripted-valid offline fixtures, never checkpoint inference or a semantic quality claim. Source https://github.com/clp-research/clembench/tree/adff278de1cf1cbc30a93f3000347b102cdfc0c5