stepgame-static-eval

stepgame-static-eval

All 10 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

StepGame

Runs all 100,000 held-out test examples in the author's recommended Hugging Face release, divided into ten complete 10,000-question hop panels. No training or validation rows are used. The author publishes a classifier, not a chat prompt or free-text scorer. This adapter renders the original story sentences and question verbatim, then requests one of the native nine answer labels. Only outer whitespace is removed; other responses are invalid. Scoring preserves native exact class accuracy. Invalid responses remain in the denominator. This is a documented text interface to the classification task, not a reproduction of TP-MANN model inference. Retains original records, raw provider responses, tokens, invalid/truncated counts and per-hop accuracy. Identical story sentence sets share sampling components. Component IDs use the first matching row index across the complete pinned test file; original story SHA256 values remain in replay.

Source: https://github.com/ShiZhengyan/StepGame/tree/5e6aff1563a6d7f46ee1f1aeff98b94e68c29005

Data: https://huggingface.co/datasets/michaelszx/StepGame/tree/6d859381dfd518cae3f073b268aaa323bf4dcf04