pokerbench-static-eval

pokerbench-static-eval

All 2 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

PokerBench

The 1,000 pre-flop and 10,000 post-flop test rows run as separate variants. Published instruction text is unchanged; no training row is evaluated. Both duplicate post-flop instruction pairs remain in the denominator. Private-hole-card variants sharing public scenario text form one sampling component. JSON and structured CSV rows are not aligned, so the CSV is not joined by row index.

Five fixed training demonstrations precede each test instruction. Seed 42 reservoir sampling selects one example per action, scanning the pinned pre-flop then post-flop training JSON. The 237 training records matching any held-out instruction were excluded from demonstration selection. The selected file/row identities and training byte hashes are retained in pokerbench_fewshots.py. No evaluation answers enter prompts.

The paper defines Action Accuracy (action correctness) and Exact Match (action plus wager). The author repository publishes no scorer, response parser, exact demonstration selection, or prompt concatenation code. This frozen-five-shot-greedy-aa-em-v1 implementation is an explicit adaptation, not the native implementation or a reproduction of paper scores. Its parser normalizes case and whitespace and accepts only check, fold, call, bet POSITIVE_DECIMAL, or raise POSITIVE_DECIMAL. Decimal wager equality ignores trailing zeros. Prose, punctuation, chip-unit suffixes and all-in aliases are invalid; invalid answers score zero on both metrics. The primary metric is exact_match; action_accuracy is retained separately. Greedy decoding (temperature 0, top-p 1) differs from the paper's temperature 0.1/top-p 0.95.

This is a near-domain poker decision diagnostic, not live poker win-rate or far-transfer evidence.

Paper: Richard Zhuang et al., PokerBench: Training Large Language Models to become Professional Poker Players, https://arxiv.org/abs/2501.08328. Source: https://github.com/pokerllm/pokerbench/tree/cf13d369630f859b23c9b4a179826f6764c4d6b6. Data: https://huggingface.co/datasets/RZ412/PokerBench/tree/7ac61f961c81a50fc0f667820b2fb0e432dfec0d. The data card declares Apache-2.0; upstream READMEs and the license are retained in the image.