fantom-static-eval

fantom-static-eval

All 2 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

FANToM

Complete author release, native short and full contexts, no chain of thought. All question families and native response parsers are preserved. Reports include set ALL*/ALL, per-family metrics, accessible controls, and character consistency. The score is inaccessible set ALL* before percentage rounding; original reports retain author percentage rounding. Each eligible question contributes its set correctness times total questions divided by eligible sets and questions in that set; other questions contribute zero. Their mean is exact ALL*. Conversations are sampling components. Learned belief grading uses all-roberta-large-v1 at revision cf74d8acd4f198de950bf004b262e6accfed5d2c. Undefined report subsets are null. Native name-substring list grading rejects six gold list answers in sets 163-1-0, 163-1-1, and 175-1-1 (Erik/Erika and Cindy/and Cindy); these author data/scorer quirks remain unchanged. The reference control therefore need not score 100%. Short-context reports exclude binary no:long rows. Full-context character reporting uses matched-character-observations-v1: the author reporter crashes for set 255-2-0 because binary and list accessibility labels differ. Only character reports use inner joins and comparable set/character pairs with matching list and binary observations for every observed family. Coverage reports retain observed, comparable and excluded denominators and unmatched set/character/family IDs. Short reporting is unchanged. All task rows, native grades, labels, ALL*/ALL and per-family scores remain unchanged; full character reports are explicitly corrected reporting, not unchanged author reporting. All released conversations are evaluation-only. This is the pinned zero-shot author protocol, not a claim to reproduce any paper model's score.

Source: https://github.com/skywalker023/fantom/tree/1cae6fa30f5ba04ca0fff5f5716b5ba7055e2e85