deal-or-no-deal-static-eval

deal-or-no-deal-static-eval

All 212 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

Deal-or-No-Deal

Source and held-out data are licensed CC-BY-NC-4.0 by Facebook, Inc. The upstream LICENSE and README are retained in the image. Source: https://github.com/facebookresearch/end-to-end-negotiator/tree/bbb93bbf00f69fced75d5c0d22e855bda07c9b78.

All 1,052 official test perspective rows appear in both learner roles: 2,104 complete games, 212 infrastructure shards of at most ten games. Repeated scenarios retain their original row hashes and share canonical unordered private-context components across roles. No training or selfplay rows are evaluation tasks. Human transcripts and reference allocations never enter prompts.

The pinned native Dialog driver, ObjectDivisionDomain scorer, and metrics execute unchanged. Each game retains the native 5,000-word dialogue budget and exact single-token selection termination. The source CLI's max_turns=20 is unused by its driver; no 20-turn cap is added. Budget exhaustion still permits private selection and agreement, as in the original driver. First speaker uses a deterministic row/seed derivation shared across paired roles. Native elapsed-time metrics are diagnostic. Primary realized_utility is native role reward, zero on disagreement; potential rewards and agreement remain separate. No hidden utility targets enter either seat's messages.

This release adapts the interface to two pinned LLMs; the suite supplies an explicit baseline partner. It does not reproduce the author's RNN/latent agents: trained checkpoints are absent from the repository and no download is documented. The new text prompt/parser is versioned explicitly. Both seats use temperature 1, top-p 1, and a 512-output-token cap per request. Dialogue text is split on whitespace, clipped only to remaining native word budget, and terminated with an EOS token; exact stop markers remain single tokens. Raw responses and consumed native words are both retained. Histories remain complete. Private selection requests are never sent to the other seat. They accept native legal six-token allocations or a no-agreement marker. Invalid selections map to native no-agreement and are counted; this mapping is an explicit interface adaptation, not an author parser.

The native scorer interprets special selection tokens as zero allocated items. Consequently an invalid selection mapped to no-agreement can still yield native agreement when the partner takes every item. This source behavior is preserved, not replaced with forced disagreement or zero scores.

Certification uses two offline invalid-selection controls, never model identities. Runtime proof and model evaluation are separate; no historical model scores or native-checkpoint coverage are claimed.