sotopia-hard-static-eval

sotopia-hard-static-eval

All 100 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

Sotopia-hard v0.1.4

Native MIT author source: https://github.com/sotopia-lab/sotopia/tree/0fe46e0ce9cf36b259c5da69e3bb0876cb5f5b1a (published v0.1.4). Newer a0aaafb440e570e5e61b7c44a44e5e417c545383 was rejected after a complete native episode demonstrated mismatched judge score fields and discarded terminal ratings. No scorer repair or surrogate is used.

Exactly 100 cases follow EnvironmentList 01HAK34YPB1H1RWXQDASDKHSNS: 20 scenario/learner-role pairs across 14 scenarios, each with five original profile combinations. The native agent_index is preserved, including asymmetric roles. Records comprise 70 distinct combinations and 37 agent profiles. Case order preserves the author hard-list order and sorts combinations by primary key. The source of these original records is the author installer-published CC-BY-SA-4.0 cmu-lti/sotopia-pi dataset revision e583406958ff132f6749ca87a2f9aa31ae3c0fa1, dump SHA256 324c037f4ac3ca6605c800bfe74eb1c11b05ed8c96c26df48c7560a7dac90da2. The original Box download was unavailable; byte identity with that alternate dump is not asserted.

Seat 0 is the learner, seat 1 its frozen conversation partner, and seat 2 the frozen native judge. The partner and judge identities are part of the protocol fingerprint; distinct choices form separate comparison panels. Coworld seats are independent of the original learner role. All prompts, backgrounds, hidden goals, native LLMAgent histories, round-robin actions, parsing, repair and fallback logic run unchanged. Native limits remain 20 turns and max_stale_turn=2. No extra history truncation or shortened episode is introduced.

The native seven score dimensions and native overall mean remain separate metrics. Coworld learner score is the unscaled native overall, which may be negative; infrastructure partner/judge seats receive zero. The complete native EpisodeLog, structured messages, every completion request/response and every native parser acceptance/rejection are retained. invalid_responses counts rejected native learner parse attempts.

Only completion transport and dataset loading are adapted. Author-supported custom text endpoint aliases route to the three Coworld policies. Native actor temperature 0.7 and judge temperature 0.0 are preserved; same-model JSON-object repair omits temperature. Native max_tokens and top_p remain omitted, and no Qwen reasoning/template settings are injected. Original drop_params=True kwargs are retained as provenance; Coworld forwards native supported parameters directly and never silently drops them. Generic suite metadata reserves 8192 output tokens; it is not a native completion cap. Hosted transport adds 32768 context metadata, 40-second calls and a 6000-second wall. Transport, context and wall failures remain unscored.

An owned loopback Redis process runs native JSON/Search persistence with Redis Stack 7.2.0-v10 binaries pinned to image digest e44b2b49d059ee0e404305624d78f920b0f932434f42a0a9b9e7d07053799324. Redis Stack components carry their own RSALv2/SSPLv1/Apache notices; Sotopia MIT is not asserted for them. Exact source records are loaded into fresh per-episode storage. Certification uses explicit offline leave actions and zero-valued synthetic judge responses. It proves engine/transport wiring, not social ability or model quality; provider-backed evaluation requires separately allocated partner and judge policies.