Wiki · main
writing-preference-static-eval
writing-preference-static-eval
All 2 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.
WritingPreferenceBench
All 1,200 English and 600 Chinese human preference pairs are retained in separate variants. Each pair is presented in both A/B orders; the seed shuffles which order comes first. Judge prompts include only the writing instruction and response texts, excluding source model names, scores, and chosen/rejected labels. Trimmed, case-normalized A/B answers receive exact credit; invalid answers receive zero. Pair accuracy averages its two orders; the episode score averages every pair. Chinese source rows 2, 179, and 448 have tied numeric scores (0/0); their author-assigned chosen/rejected labels are retained, and replay lists these three anomalies. No rows are dropped. Categories and complete raw responses remain in replay. Pairs sharing prompt_id form one sampling component.
The upstream release describes LLM judge criteria but publishes no exact prompt or parser. This two-order text-judge-v1 interface is an explicit adaptation, not a reproduction of published judge scores or scalar reward-model scoring. It measures agreement with these human preferences, not writing-generation quality.
Attribution: Shuangshuang Ying et al., Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures (2025), https://arxiv.org/abs/2510.14616. Data and documentation: Open Data Commons Attribution License (ODC-BY), https://opendatacommons.org/licenses/by/. The image retains the upstream README and its license notice. Dataset bytes are SHA-256 checked.