failproof-arcade
Same AI agent, same prompt, same tools. One plays alone, the other runs with failproofai
runtime policies. Who plays better?
failproof-arcade drops a Claude agent (Claude Agent SDK) into five real games and plays every game twice: once
bare, once with failproofai policies hooked into its tool calls. Policies never pick a move. They state a public
fact, warn, or veto a bad action, and the agent still decides. Every number below comes from frozen policy packs on
held-out games.
Results
| Game |
Model |
Opponent |
Bare |
+ failproofai |
Held-out sample |
| Pokémon Showdown (Gen 9 Random Battles) |
Claude Sonnet 5.5 |
Foul Play (MCTS bot) |
18% wins |
42% wins |
100 + 100 battles, p = 0.0003 |
| Unciv (open-source Civilization V), 60 turns |
Claude Haiku 5.5 |
the game's own AI |
score 224.3 |
244.1 (+8.8%) |
100 paired maps, p = 0.002 |
| Poker (PokerTH, heads-up NLHE) |
Claude Haiku 5.5 |
PokerTH's computer player |
−10.8 bb/100 |
+1.5 bb/100 |
102 paired decks (+9.7 bb/100 paired, not significant) |
| Yu-Gi-Oh! (EDOPro) |
Claude Sonnet 5.5 |
WindBot "Rainbow" |
57.5% wins, 1.27 misplays/duel |
65% wins, 0.45 misplays/duel |
40 + 40 duels |
| DOOM (ViZDoom, Freedoom E1M1–E1M3) |
Claude Sonnet 5.5 |
— |
0.87 shots at nothing/run |
0 (p = 0.001) |
30 paired maps |
What moved, per game:
- Pokémon: the policy gives the exact damage range of every legal move (public damage calculator) before the
agent commits. Your Pokémon left at the end: 0.43 → 1.64.
- Unciv: idle cities, unchosen research and unspent social policies get caught before the turn ends. Mistakes per
turn 0.99 → 0.04, techs +21%, social policies +63%, military +14%.
- Poker: hand strength, equity and pot odds on every decision, plus one hard veto (never fold when checking is free).
Strong hands folded: 0.50 → 0.21 per 100 decisions.
- Yu-Gi-Oh!: end-of-phase checklists from the engine's own legal moves, card text on first use, lethal-threat math.
Skipped Main Phase 2: 50 → 17.
- DOOM: four vetoes (blind fire, aim, repeated blocked move, low-HP burst). Blocked moves 12.3 → 8.7.
Full write-ups, dev rounds included: analysis/<game>/rounds.md and the report tables next to them.
How a run works
- One episode = one Claude Agent SDK session. The game is exposed as in-process MCP tools (
battle_state,
use_move, act, end_turn, …) that return only what a human player would see.
- Both arms get the same model, system prompt, tools, harness and auto-continue. The only difference is the
.claude/settings.json in the policy arm's run directory, which wires failproofai hooks (arcade/core/rundir.py).
- Policies live in
policies/<game>/ and run inside the tool call via failproofai's hooks: PreToolUse to veto,
PostToolUse to add facts.
- Every run executes in an isolated Docker container with its own
failproofaid. A policy-arm run with zero hook
events, or a baseline run with any, is invalid (arcade/core/validity.py).
Rules that keep the comparison fair
Code comments refer to these by ID.
- R1 — one difference. Same model, prompt, tools and harness in both arms; only the policy arm's hooks config differs.
- R2 — no hidden state. Policies read only the visible game state the agent was shown.
- R3 — policies never pick the move. They state facts, warn or veto. The agent always makes the final choice.
- R4 — held-out games. Tuning uses dev splits; report seeds, decks and maps are never used while tuning.
- R5 — nothing silently dropped. A crashed episode still writes a result. Runs broken by infrastructure
(timeouts, interrupted containers) are voided and re-run, and stay under
results/<game>/void/.
- R6 — frozen packs. A pack is hashed before its report batch (
analysis/<game>/frozen*.txt) and every report run
is checked against that hash.
- R7 — independent scoring. Outcomes are re-derived from the game's own records (replays, server logs), not from
what the agent reports.
- R10 — same conditions. Both arms of a report batch run at the same time, interleaved.
Repository layout
arcade/ harness: CLI, episode loop, run dirs, validity checks, one env per game
policies/ failproofai policy packs (every version; the frozen ones are named in analysis/<game>/frozen*.txt)
docker/ one image per game, all upstreams pinned
third_party/ small bridge patches to Unciv (MPL-2.0), WindBot (AGPL-3.0) and PokerTH (AGPL-3.0+)
analysis/<game>/ analysis scripts, dev-round logs (rounds.md), report tables
results/<game>/ one JSON per scored game; every headline number recomputes from these
Reproduce it
You need Docker, Python 3.11+ with uv on the host for the analysis scripts, and a
Claude credential. failproofai runs inside the containers; no account is needed.
mkdir -p .secrets
echo "ANTHROPIC_API_KEY=sk-ant-..." > .secrets/claude.env # or CLAUDE_CODE_OAUTH_TOKEN=...
docker build -t arcade-runner -f docker/Dockerfile docker # base: Node, uv, failproofai 1.0.13 + failproofaid
# harness self-test (counts to 5; checks hooks fire in the policy arm and not in the baseline)
docker/run.sh arcade-runner uv run python -m arcade.cli run --game echo --arm policy --n 1 --split smoke
docker/run.sh <image> <command> starts an isolated runner with the repo mounted, so results land in results/ and
raw transcripts in runs/ (gitignored). ARCADE_MODEL picks the model (default claude-sonnet-5-5).
Seeds below are the held-out report seeds used for the numbers above.
Pokémon Showdown
docker build -t arcade-pokemon -f docker/pokemon.Dockerfile docker
docker build -t arcade-pokemon-fp -f docker/pokemon-fp.Dockerfile docker
for arm in baseline policy; do
ARCADE_DOCKER_ARGS="-e FP_MS=1" docker/run.sh arcade-pokemon-fp arcade/games/pokemon/run_report_fp.sh $arm 100 fp-report &
done; wait
python3 -m arcade.games.pokemon.report fp-report
Unciv
third_party/unciv-bridge/build.sh /tmp/unciv-build # clones Unciv at the pinned commit, applies the patch
analysis/unciv/batch.sh baseline report 1000 100 4
analysis/unciv/batch.sh policy report 1000 100 4 unciv-v2
python3 analysis/unciv/analyze.py report
Poker
docker build -t arcade-poker -f docker/poker.Dockerfile .
for arm in baseline policy; do
ARCADE_MODEL=claude-haiku-5-5 docker/run.sh arcade-poker bash arcade/games/poker/run_batch.sh $arm report 230 5 2000 50 &
done; wait
python3 analysis/poker/analyze.py report
Episode seeds 2k and 2k+1 play the same deck with the seats swapped (duplicate poker). The analysis pairs them by deck.
Yu-Gi-Oh!
docker build -t arcade-yugioh -f docker/yugioh.Dockerfile .
docker/yugioh/up.sh # EDOPro duel server (Multirole) sidecar
for arm in baseline policy; do
ARCADE_DOCKER_ARGS="--network arcade-ygo" docker/run.sh arcade-yugioh \
uv run python -m arcade.cli run --game yugioh --arm $arm --n 40 --parallel 2 --split report &
done; wait
python3 analysis/yugioh/analyze.py summary report
DOOM
docker build -t arcade-doom -f docker/doom.Dockerfile docker
PACK=doom-v8-vetoes BUDGET=300 analysis/doom/run_cells.sh report300 v8 4 analysis/doom/cells-report300.txt
python3 analysis/doom/analyze.py report300
Cost
Rough API cost per game played, at list prices (the policy arm runs a little longer). The Sonnet rows are upper bounds:
| Game |
Model |
≈ per game |
| Pokémon battle |
Sonnet 5.5 |
≤ $0.25 |
| Unciv, 60 turns |
Haiku 5.5 |
$0.04–0.06 |
| Poker, 50 hands |
Haiku 5.5 |
$0.07–0.22 |
| Yu-Gi-Oh! duel |
Sonnet 5.5 |
≤ $0.40 |
| DOOM, 300 actions |
Sonnet 5.5 |
≤ $5 |
Optional: FailproofAI Cloud
Runs work fully offline. To also see every session, policy firing and score in
FailproofAI Cloud, add .secrets/fp-cloud.env with FAILPROOFAI_CLOUD_TOKEN=... (and
FAILPROOFAI_CLOUD_URL for a self-hosted dashboard). docker/run.sh picks it up when present.
Notes
- Results are statistical. LLM play is not deterministic, so a re-run gives different games; the held-out batches are
sized so the conclusions should hold.
- This repo is MIT-licensed (see
LICENSE). Third-party games keep their own licences, and the patches under
third_party/ carry the licence of what they patch.
DOOM uses Freedoom, not id Software's WADs.
- Pokémon is © Nintendo / Game Freak and Yu-Gi-Oh! is © Konami. This project runs the community engines locally and
does not distribute their assets.