paintbench-static-eval

paintbench-static-eval

All 228 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

PaintBench

All 2,616 frozen native-generated cases are partitioned into 228 panels: 1,920 PaintBench edits, 96 separate preservation diagnostics, and 600 TinyGrafixBench edits. This dataset is pinned to the native generator; it is not asserted identical to the paper's hosted dataset. Only the original input image and instruction reach the player. Native eval.py scores the original returned PNG and retains every threshold and per-case record. Panel scores are descriptive means, not the paper's hierarchical bootstrap estimates. Preservation diagnostics must remain separate from the PaintBench task mean. The image transport uses the explicit OpenRouter Images API profile. The unchanged-input control proves execution, not model quality.