barterbench-static-eval

barterbench-static-eval

All 2 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

BarterBench

The pinned executable suite uses five learner agents and five native RandomAgent anchors in ten-agent, twelve-round Spice Wars markets. Each of ten matches assigns one model per paired role. All five learner agents call the same pinned Coworld policy with separate native private observations and complete native own-action history. Temperature 1, maximum 512 output tokens, original tools, OpenRouter prompt construction, action parser, engine transitions, and scoring remain native. The engine's normal all-goals-complete early stop is retained. Auctions are disabled by Spice Wars.

The spice-wars variant repeats the unchanged author scenario ten times. Repetitions share one sampling component. The procedural-holdout variant is an adaptation, not an author-released test split: ten scenarios from the unchanged official generator, seeds 100000–100009, disjoint from development 0–9, ten agents/five items/two scarce items/twelve rounds, native scarcity_ratio 0.5 and mixed theme. Whole scenario states define independent sampling components. No generated holdout is used for training.

Reproducibility adaptations replace salted Python hash seeds with SHA256-derived task/case seeds and resolve unordered model sets in sorted order before the same role-pair shuffle. Native engine and RandomAgent RNGs are independently seeded; source and anchor seeds are retained. Coworld replaces source-tree checkpoints/dashboard writes with full replay, and the loader omits credential-file loading and binds the original solvability import in isolation. Transport failures abort rather than scoring upstream provider fallback passes. The native parser rejects malformed text via a pass action, but valid JSON tool arguments that are not objects (for example []) crash the source parser; this native limitation remains visible as an episode failure rather than receiving an invented score.

Primary goal_completion is the native rounded learner-model aggregate. Native pairwise anchor scoring uses its two-percentage-point draw threshold; win rate excludes draws, while anchor_match_score awards half for draws. Raw provider responses, parsed actions, invalid counts, source metrics and full market history remain in replay. The offline invalid control is qualification only, not a model score.

The README's haiku-anchor and eight-agent prose conflicts with the executable random-anchor/ten-agent release; this adapter follows the code and scenario JSON. README declares MIT via a badge, but the repository has no standalone LICENSE file. Upstream README attribution is retained in the image.

Source: https://github.com/JamesEBall/BarterBench/tree/54b864401569c1bc13a323faad8469f24c0ba544