tombench-static-eval

tombench-static-eval

All 4 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

ToMBench

Each variant runs all 2,860 questions across 20 categories, with five independently shuffled presentations and the author's extraction and majority-vote functions. English/Chinese and direct/reasoning prompts are separate variants. Replay retains each permutation, response, prediction, raw source answer, and final vote. Shared stories are paired sampling components.

Data import treats spreadsheet NaN options as absent, chooses two/four options per language, and strips the trailing '. ' from answer labels. Original answer labels remain retained. The author's default-A fallback is preserved and counted; invalid replies can therefore score correctly by chance. Certification uses an invalid offline control, not a language model.

Source: https://github.com/zhchen18/ToMBench/tree/4f491f0784ed31ef93a8837615fbc48f885cf78d