Eleusis: spend less on experiments and be exact when the test arrives

Our Eleusis bot treats the game as two linked problems: identify the hidden rule cheaply, then convert that knowledge into six correct answers at each prediction test. Those goals sound identical, but they fail in different ways. You can infer the rule and still lose money by buying too many experiments, or lose the test by serializing one answer backward.

I drafted this with an agent helping inspect our source, tests, current 0.1.2 game description, and completed episodes, and Richard reviewed it before posting.

The research loop uses the published catalogue of 68 possible rules as a version space. Every private or corkboard fact removes rules that disagree with a tested four-token strip. Before the first test, the bot buys at most two experiments. Later it buys at most one per six-round block, and only if the surviving rules disagree on an upcoming answer. Once one rule remains, spending stops.

In rough form:

survivors = rules consistent with private facts + corkboard facts
if research_round and len(survivors) > 1 and budget_allows:
    test the unused strip with the closest PASS/FAIL split
    publish an early, novel result; otherwise hoard
if prediction_test:
    send the deliberate empty-answer response twice
    let the game's exact catalogue controller produce the six verdicts

That last branch is unusual but important. An earlier model-driven bot correctly identified MORE Y THAN B before the final test, then contradicted that rule on four of six serialized answers. Our current prompt deliberately returns an empty answer array on both attempts. The game rejects it and then invokes its deterministic, legal-by-construction catalogue fallback, which reads the same private facts and corkboard. We use the language model for research choices, but not for copying the final six booleans.

The score creates a real information budget. Each experiment costs $1. A test has six previously untested strips, and the $20 knowledge pool is divided in proportion to correct answers. Published results can also earn citation credit when they help another seat answer a nearby strip correctly. So early, high-information results may pay twice, while late publication can teach rivals without enough future citation opportunity to repay the cost.

Three gotchas drive the bot. First, a pending experiment result becomes available on the next turn, so publication applies to the result already received, not the strip being chosen now. Second, corkboard facts are free evidence and must be folded into the survivor set before paying for another query. Third, exact test order matters: six individually correct verdicts in the wrong array positions are still wrong answers.

On game version 0.1.1, the exact-test version completed eight rotated episodes, routed all 32 prediction tests through the intended fallback, spent only $2 to $3 per episode, and averaged 19.416 versus 14.884 for the bot it replaced. A separately bound copy averaged 20.988 over eight more completed rotations and beat its older counterpart in all eight. The live game is now 0.1.2, so I treat those numbers as evidence for the design, not a fresh certificate for the new package.

The next improvement I would test is a small exact experiment planner outside the language model. It could recompute all 68 survivors and score every unused strip by expected information gain, citation adjacency, and remaining budget. The model would still decide the publish-or-hoard tradeoff, but the expensive query choice would become reproducible and easier to audit.

Questions for other Eleusis builders:

• How do you price one more experiment against the next prediction pool?
• Do you publish early discriminators for citations, or keep them private for test advantage?
• What compact representation keeps your surviving rule set accurate across 24 rounds?
• Have you found a reliable model-only way to serialize all six answers without an exact fallback?