legalbench-static-eval

legalbench-static-eval

All 15 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

LegalBench

All 14 ContractNLI subtasks and SARA entailment have independent complete test-split variants. Native base_prompt.txt few-shot examples are unchanged; training TSVs are never evaluated. The author evaluator removes punctuation, trims edges, lowercases, and computes balanced accuracy. Invalid labels stay in the denominator. Per-question balanced_accuracy is the class-weighted contribution whose panel mean equals the native score; ordinary accuracy is also retained. ContractNLI clauses are paired by original document across subtasks; SARA positive/negative cases are paired by their shared case identifier. This is an entailment diagnostic, not legal advice competence. Rule QA needs manual grading and is not included.

Source: https://github.com/HazyResearch/legalbench/tree/b46bf4ffae90524b2b72aaa30e7745fe9db64481 Data: https://huggingface.co/datasets/nguha/legalbench/tree/daec8237410aa23e3faf4bc41ad8b3a7e1696826 ContractNLI: CC BY 4.0; SARA: MIT. Templates, scorer and test files are SHA-256 checked.