discoveryworld-normal-static-eval

discoveryworld-normal-static-eval

All 40 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

DiscoveryWorld text ReAct

Official Apache-2.0 code sources: https://github.com/allenai/discoveryworld/tree/fd591323920be0d3786ef350955de1945aa571e5 and author-pinned Recoma https://github.com/allenai/recoma/tree/b262ab5476841551c5d6874e8ddecef78c71a072. Bundled CuteRPG World art is by PixyMoon: https://pixymoon.itch.io/cute-rpg-world. Its separate terms permit personal/commercial project use and modification, require credit, and prohibit resale of the asset pack. Apache-2.0 does not cover these assets. The author requests re-purchasing the art pack when releasing derivative projects. The native worker pins Python 3.11.14. The author README suggests 3.9; 3.11 retains the native integral-float randint bounds used by Rocket Science, which Python 3.12 rejects. Engine source is unchanged. Raw rendered PNGs and every world-history segment are retained. Native video export can fail when a scenario starts beyond frame 4 (Plant Nutrients); this does not remove raw frames or alter native scoring. Four releases cover 40 Easy, 40 Normal, 40 Challenge discovery cases and 50 SmallSkills cases. Tutorial is excluded. Inventory follows agents/recoma/configs/react.jsonnet and ScenarioMaker: seeds 1–5, not the conflicting generic agents/README.md example stating 0–4. No alternate seed suite is silently pooled. Theme components preserve pairing across difficulties.

The world seed is immutable in the native task ID and discoveryworld.seed. The evaluation case/runtime seed identifies suite repeats independently and never changes that world. Learner generation retains the author ReAct sampling seed 123.

The author's ReAct config, Jinja prompt, native search/controller, parser, task engine, tick order, termination and scorer execute unchanged. The native 10,000-character action-observation history window remains part of prompt construction; no additional prompt or recorded-trajectory truncation is applied. Text ReAct drops vision from its model observation while retaining native UI text. All cases are offered by that author text reader. This is not a vision-agent reproduction or a claim that text exposes every visual cue. Full raw native tracking, observations, rendered frames, videos, world-history exports and every submitted prompt/response are retained in replay. Native files are stored as a SHA-256-verified base64 gzip tar archive with per-file hashes.

Seat 0 is the learner. Discovery tasks require a frozen seat-1 knowledge judge; SmallSkills has one seat and no knowledge metric. Native scoreNormalized is procedure, completedSuccessfully is completion. Knowledge uses scripts/knowledgeScorer.py with the exact thought concatenation from analysisReactKnowledge.py. The metrics remain separate; Coworld learner score is procedure and the judge seat receives zero. Knowledge follows the native permissive JSON parser, including its zero contribution for malformed judge answers; raw responses and evaluations remain inspectable. The adapter does not infer an additional invalid-response classifier.

Native environment-call limits remain 100 for Easy/SmallSkills and 1,000 for Normal/Challenge. Native ReAct also limits learner LLM calls to 1,000 and reference estimated cost to $50. The native LiteLLM generator retains its response postprocessing, counters and frozen bundled gpt-4o-2024-05-13 pricing using accepted token usage. This reference-cost guard is not actual provider billing. Provider calls/cache/retries are replaced by Coworld policy transport. The unavailable PyPI litellm==1.37.19 is hash-locked from its official v1.37.19 Git commit f20184cd49c77ee66da21001a960f50b8875c321, with its bundled model-price map.

Learner settings retain 400 output tokens, temperature 0, seed 123 and the native closing-fence stop string; the judge retains 800 output tokens, temperature 0 and JSON mode. Transport adds 32,768 context tokens, 20-second requests and a 6,000-second episode wall. These hosted limits are explicit adaptations; they do not shorten native action budgets. Context overflow, transport and wall failures remain unscored. Native terminal states alone produce task scores. Certification uses an explicit offline SUBMIT and zero-judgment fixture, with a separate judge identity from evaluated variants. It proves wiring, not scientific ability or provider quality.