simpletom-static-eval

simpletom-static-eval

All 6 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

SimpleToM

Each variant runs all 1,147 questions at one reasoning level: mental-state, behavior prediction, or behavior judgment. Direct and chain-of-thought prompts are separate variants. Native prompts and answer extraction are retained. Replays include original records, prompts, responses, and accuracy. Shared stories are paired sampling components.

The optional native token-probability diagnostic is not reported; raw provider evidence is retained. Certification uses an invalid offline control, not a language model.

Source: https://github.com/yulinggu-cs/SimpleToM/tree/2031aea8eec3920e3fda46e7a69bfadf238d0620