ComposeBench
ComposeBench is a small, deterministic benchmark for measuring one thing in isolation:
how much an agent's action substrate — the interface it uses to call tools —
affects its ability to compose dependent tool calls.
Every substrate is given the same tools, the same tasks, the same turn budget, and
the same grader. The only variable is the action surface. This makes it a clean
control for questions like "does letting the model write code instead of emitting one
JSON tool call per turn actually change behavior, or is it just style?"
What it measures
The domain is a deterministic sensor network with seven pure integer functions:
| Tool |
Signature |
read_sensor |
read_sensor(sensor_id) -> int |
calibrate |
calibrate(value, offset) -> int |
scale |
scale(value, factor) -> int |
combine |
combine(a, b) -> int |
difference |
difference(a, b) -> int |
aggregate |
aggregate(values) -> int |
threshold_count |
threshold_count(values, threshold) -> int |
Ground truth for every task is computed by these same functions, so grading is exact
(a single integer) and independent of the substrate.
108 tasks in two families:
- Chain (60 tasks): a linear dependency of depth 1–5, 12 tasks per depth. Each step
consumes the previous step's result.
- Fan-in (48 tasks): read and calibrate N sensors (width 3, 5, 7, 9), then
aggregate and scale them; 12 tasks per width.
Depth 1 is a deliberate falsifier: it has no compositional structure, so a code
substrate should not show an advantage there. Advantages that appear only as depth or
width grows are attributable to composition, not to the interface being generally better.
The four substrates
All four receive the identical tool catalog and task text; they differ only in how an
action is expressed and executed:
repl — one persistent Python code block per turn. The model may call multiple
tools, bind variables, loop, and compose values within a single action. The namespace
persists across turns. Runs in a restricted interpreter (no imports, no dunder access).
batchfc — one dependent JSON call-DAG per turn (steps with intra-DAG $id
references plus a return). Composition without free-form code.
fc — one JSON tool call per turn; prior results are addressable as $r0, $r1, …
cli — one shell-style command per turn; prior results addressable as $r0, $r1, …
An optional --noise condition wraps observations in distractor telemetry fields to test
robustness to noisy tool output.
Quickstart
No third-party dependencies; Python 3.10+.
# 1. Generate the task set (also verifies every gold answer against the oracle).
python3 gen_tasks.py
# 2. Sanity-check counts, gold answers, and DAG execution.
python3 self_check.py
# 3. Run a model across all four substrates (needs an API endpoint; see below).
python3 harness.py --model claude-sonnet-4-6 \
--substrates repl,batchfc,fc,cli --reps 2 --n_turns_limit 20 \
--out results.jsonl
# 4. Analyze (success rate + mean turns by depth/width, with Wilson intervals).
python3 analyze.py results.jsonl
Add --noise in step 3 for the noisy-observation condition.
Model configuration
harness.py talks to an OpenAI-compatible (/v1/chat/completions) or
Anthropic-compatible (/v1/messages) endpoint. Configure it either way:
- Environment variables: set
OPENAI_API_BASE + OPENAI_API_KEY, or
ANTHROPIC_BASE_URL + ANTHROPIC_API_KEY. Optionally LLM_API_STYLE=openai|anthropic.
- Config file: copy
llm.example.yaml, add your endpoints/keys, and point
LLM_YAML at it (export LLM_YAML=./llm.yaml).
Never commit real keys — llm.yaml is git-ignored.
Results
harness.py writes one JSONL row per (task, substrate, rep):
{"task_id": "chain_d3_00", "kind": "chain", "depth": 3, "width": null,
"substrate": "repl", "rep": 0, "model": "claude-sonnet-4-6",
"is_correct": true, "n_turns": 2, "turn_limit": 20, "noise": false,
"input_tokens": 8903, "output_tokens": 55, "api_calls": 2, "error": ""}
Runs are resumable: re-running with the same --out file skips
(task, substrate, rep) triples already present.
results/ contains reference runs on claude-sonnet-4-6 (clean and noise conditions,
chain and fan-in). python3 analyze.py results/*.jsonl reproduces the headline pattern:
repl and batchfc stay at ~2 turns per task at every depth and width, while fc and
cli turn counts grow with the dependency structure, and their success collapses on the
widest fan-in tasks.
Repository layout
gen_tasks.py task generator + oracle (writes tasks.json)
harness.py the four substrate runners
analyze.py success / turns / token analysis
self_check.py deterministic integrity check
llm_client.py minimal stdlib chat client (OpenAI/Anthropic-compatible)
llm.example.yaml model-config template
tasks.json the generated 108-task set
results/ reference runs (claude-sonnet-4-6, clean + noise)
License
Released under the MIT License. See LICENSE.