crab-static-eval

crab-static-eval

All 4 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

CRAB

Four complete panels: binary, four-class and scalar pairwise (2,730 rows each), and graded causality MCQ (3,600 rows). Source: https://github.com/agromanou/CRAB/tree/21ed62fd4501f3faf51dab2eb9e93ba4db0f27e9. Data: author angelika/CRAB revision 46cb52506de0fb606dc89286f4ea675e197c7146, byte-identical to the README's Drive archive. No train/test partition is released; all rows are evaluation-only here.

The author releases prompts/inference but no scorer. This adapter uses paper-defined macro-F1 over fixed gold classes, plus exact per-row accuracy. Row macro_f1 contributions average to panel macro-F1. Invalid responses remain false negatives in the denominator, never an extra gold class. Binary responses must be yes/no after whitespace stripping and lowercasing. Other panels require exactly one <Answer> tag pair containing an uppercase choice or decimal score in [0,100]. Score boundaries follow released labels: D<20, C<50, B<80, A>=80; binary yes>=50. No fuzzy extraction or response repair.

Prompts are verbatim author templates. All pinned shot fields are empty and the official scripts ignore them: this is zero-shot. Released nested graded records map to the template's effect and A-D choices; contexts use effect article then A-D articles. The original flat script's 0-3 article order cannot be recovered from released nested records. Iteration begins at row0 and includes every row, avoiding the source script's hardcoded batch122 resume. Greedy decoding and the source256-token budget are explicit adapter settings. Related rows share narrative-topic components. Frame/chain subgroups are deferred because their extraction protocol is not released. The optional three-class prompt is outside the four paper panels.