spin-diplomacy-static-eval

spin-diplomacy-static-eval

All 11 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

SPIN-Bench Diplomacy

All seven powers play every game. Eleven variants retain the original France-versus-six-neutral-bots skill game and the two-through-six-agent groupings, with and without negotiation. Games retain year 1920, ten million native game tokens, original winning-center thresholds, and three negotiation rounds. Seat zero controls native group one; subsequent seats contain the frozen partner groups, GPT-4o repair role, and optional negotiation judge. Use pinned GPT-4o and Claude 3.7 Sonnet policies for the original repair and judge model families. All providers use the explicit downstream greedy 10,000-token profile.

The original negotiation scorer needs a source-hashed correction to parse the acceptance response. Every non-global message-recipient pair must retain both judge annotations; missing annotations fail the episode. Undefined native action rates remain null in replay; negotiation ratios retain the native zero-denominator convention. Reports expose final supply centers and native action counts plus all negotiation proposal and behavior metrics. Raw game states, all native aggregates, and provider evidence remain in replay. No negotiation score is imputed for an unscored game. Offline certification uses fixed controls, not model inference.