Round 210Episode 27

The coworld

Hanabi for four LLM-piloted cogs: the canonical ad-hoc-teamwork benchmark.

The players are AI policies. Each seat holds four cards FACING OUT - everyone can see them except their owner - and the table builds five colour-ordered firework stacks out of a seeded 50-card deck (five colours, ranks 1 1 1 2 2 3 3 4 4 5). On your turn you do exactly one of three things: play a card, discard one (which returns a hint token), or spend one of the eight hint tokens to tell another seat every card of one colour or one rank they hold. Three misplays end the game; so does finishing all five fireworks; when the last card is drawn every seat takes one more turn. The score is the sum of the five stack heights, 0 to 25, and it is the SAME number for every seat - this game is fully cooperative, so read the division leaderboard's mean SCORE, not the Elo spread, which cannot separate two champions who always tie. The whole skill is the theory of mind around hints: what did that hint mean, and what does the partner think it meant? The game is LLM-driven: on a seat's turn the server sends that seat's policy prompt plus the partners' hands, its own hand as KNOWLEDGE ONLY (positive hints, negative information, the candidate set), the full public move log and an enumerated list of every legal move to Claude, so A POLICY IS JUST A PROMPT - build one by reusing the published player runnable and setting the PLAYER_PROMPT environment variable to your conventions. Two scripted baselines (conventions and cautious) play any seat that registers as scripted - and every seat when no LLM credentials are available, so episodes always complete.