llfbench-metaworld-a-static-eval

llfbench-metaworld-a-static-eval

All 260 declared native panels are available as separately frozen evaluation leagues. Each submitted policy runs one complete selected panel per episode. Panel scores retain their native aggregation; this release defines no score across different panels. The fleet.json index records every panel, population, metric, and fixed reference roster. Qualification and coverage must be established separately for every selected panel.

LLF-Bench

Microsoft LLF-Bench, MIT license, pinned source https://github.com/microsoft/LLF-Bench/tree/fce2e11d8ceac33ae81265d09cb3e12a1d9e9030. Complete environment registry: 22 core, 50 MetaWorld, ALFWorld, and movie recommendation. This manifest is partition metaworld-a. Partitions only bound upload size. Movie recommendation remains an explicit OMDB provider/credential gate; no invented cached ratings.

Each environment has 50 frozen evaluation seeds 100000..100049, disjoint from development seeds 0..49, under both all-feedback and partial r/hp/hn feedback. Five complete episodes per variant. This is a deterministic generator adaptation, not author-released held-out scenario assets. MetaWorld retains the native MT1 train-task generator. All seeds/feedback panels of an environment share one conservative component because native task pools and dynamics can repeat. Four author-not-recommended optimization environments remain explicitly diagnostic.

Native BasicAIAgent prompts, Response parser, buffer20 history, greedy decoding and 1000 output tokens are retained. An explicit twenty-second model timeout plus ten-second transport grace bounds 150 calls to 4500 seconds, leaving 1500 seconds for connection and native execution. This timeout is an infrastructure adaptation. Native rollout horizon30, environment termination, reward, and success rules are unchanged. Raw environment rewards/info are never passed to the model: only native observation/instruction/feedback. Full actions and feedback remain in evidence even when the native prompt buffer drops older entries. Primary success matches author reporting for Gridworld/MetaWorld/ALFWorld; other environments use native_return. Both metrics are retained. Native parser feedback and rewards remain unchanged, including zero rewards from the outer parser and -0.01 rewards from the optimization parser. Zero parser rewards can exceed negative returns for legal actions.

Versioned handle-pull-goal-vector-v1 corrects a native MetaWorld expert-feedback scalar target to its intended three-dimensional handle position. Original source and failure evidence are preserved. This corrected feedback branch is not claimed identical to the crashing native implementation. Continuous actions use versioned finite numeric-literal arrays, replacing native Python exec. Parser failures still flow through the native TextWrapper's feedback path. No generated Python executes. Native imports and dependencies run in isolated Python3.10, never the shared server environment. ALFWorld selects valid_unseen through its released evaluation data path while retaining native expert feedback; this split adaptation is explicit. Only text assets are required, not detector/model weights. Certification is an offline negative control and does not report model performance.