🧪 Run experiments — compare harnesses and RSI methods under recorded conditions
Track how a harness performs as an improvement method edits it over successive rounds. The charts below show runs on three benchmarks.
| ALFWorld |
AppWorld |
ScienceWorld |

 |

 |

 |
🚀 Quick start
How does one round of DGM change Loop’s performance on math word problems? The task agent uses Loop 0.2.0 (study source) with Qwen3.5-9B to solve five GSM8K training questions. DGM uses DeepSeek Harness (headless) with DeepSeek-V4.1-Flash to edit Loop, then HarnessGrad scores the candidate on the same questions. Set up the two agents and GSM8K.
harnessgrad run --preset gsm8k-dgm-study
Open runs/gsm8k-dgm-study/end-summary.md to compare the two rounds. For a repeat run, add --run-id with a new name.
🔬 From a question to an experiment
Choose a harness → Select an RSI method → Run tasks → Inspect the evidence. Use the same recorded conditions when comparing results, then explore the learning curve, versions and task trajectories in the Console.

🧩 Experiment integrations
A research run is one point in a harness × method × benchmark space: a harness solves tasks, a method proposes edits to that harness between rounds, and a benchmark scores each candidate. HarnessGrad provides tasks and scoring, isolated execution, and version control for candidate harnesses, with a recorded curve for each experiment.
[!NOTE]
The inventory lists declared integrations; each harness and benchmark has its own preparation and runtime requirements.
Full harness, method and benchmark inventory
The table records the inventory at this README revision; the benchmark row is checked against the declared adapters.
| axis |
what ships |
where |
| harnesses |
8: loop, codex, opencode, deepseek-harness, openclaw, miniagent, miniswe, oracle |
base_harness/<name>/ |
| RSI methods |
19: dgm, rrsi, adas, ahe, ace, skillopt, evoprompt, evo_harness, opro, tthe, dream_rsi, sica_ci, gepa, darwinx, arbor, hyperagents, meta_harness, harnessx, modular_rsi |
experiments/method-<name>.json |
| benchmarks |
56 declared adapters; preparation and runtime prerequisites vary — every one listed: agentdojo, aime, alfworld, appworld, authored_example_fix_bug, bbeh, bfcl_v3, bird_minidev, browsecomp, chessbench_puzzles, cooperbench, costbench, costbench_dynamic, deepsearchqa, deepswe, ds1000, fever, gaia, gsm8k, harvey_lab, hle, hotpotqa, jericho, jobbench, kumo, livecodebench, livemathbench, locomo, math, math500, memoryagentbench, musique, nq, officeqa_pro, personamem_v2, physicsquestions, reasoning_gym, scienceworld, searchqa, sportsunderstanding, spreadsheetbench, spreadsheetbench2, strategyqa, strongreject, supergpqa, swe_bench, swe_bench_pro, tau2_bench, tau3_bench, tau_bench, terminal_bench, travelplanner, triviaqa, twowikimultihopqa, widesearch, workbench |
data/datasets/, detail in DATASETS.md |
Where to start
- Bring a harness: supply a directory with
harness.json and an entrypoint accepting --task and --workdir. See bring your own harness.
- Bring a method: supply a program that reads a trajectory and writes a candidate tree. See bring your own method.
- Bring a benchmark: write an adapter that materialises tasks and runs its official scorer. See dataset onboarding.
⚙️ Why one command is enough
A harness supplies harness.json and an entrypoint accepting --task and --workdir (interface). The platform runs dataset tasks and scoring, isolates execution, versions candidate harnesses, and records each run's outcomes and receipts. Dataset and harness prerequisites still need to be installed separately.
⚖️ Why the comparison is fair
Each round records the candidate harness hash and its changes, alongside the run's model roles and measurement metadata.
[!IMPORTANT]
Compare results only when the task selection, scorer, models and other relevant conditions support it. Failed or incomplete measurements are not scores; do not substitute proxy scores under a benchmark name. See the measurement policy.
📦 Installation
Requires Linux, Python 3.12+, Git, uv, and a working bubblewrap. See the first-use guide for setup details.
git clone https://github.com/zjunlp/HarnessGrad.git
cd HarnessGrad
uv venv --seed --python 3.12 ~/.venvs/harnessgrad
source ~/.venvs/harnessgrad/bin/activate
uv pip install -e .
./install.sh --check
🗂️ Dataset storage
Set the shared external dataset directory in your local .env:
HG_DATA_ROOT=/path/to/datasets
See downloads for preparation and locations, and datasets for adapter details.
[!TIP]
New to HarnessGrad? The first-use guide includes an installation control that does not need a model call or dataset download.
🖥️ Experiment workbench
Use the shared Console to inspect experiments, harness versions and task trajectories.
export HG_CONSOLE_RUNS="$PWD/runs" # existing run-record directory
python console/server.py --port 4460
Open http://127.0.0.1:4460/ from the repository root, or point HG_CONSOLE_RUNS to another existing run-record directory. The Console binds to localhost by default.
 Experiments |
 Versions |
 Trajectories |
🌱 Contributors
Thank you to everyone who contributes harnesses, methods, benchmarks, and improvements to HarnessGrad! The three charts show the most recent 30 days; each line includes contributions made before that window.
Want to join? Bring a harness, add a method, or onboard a benchmark, then open a pull request. We will add your name to the relevant contributor chart.



🛠️ Customize a harness — build for a model, environment or task

Ocean Harness is one example built with HarnessGrad. It combines domain sources, research tools, and an answer workflow to answer ocean questions with citations. Build a harness for your own domain using the workflow below.
📦 Installation
Use Python 3.12+ and install from the HarnessGrad checkout. For model-generated edits, configure an editor endpoint in the customization guide.
Current capabilities
Choose a starting harness → Edit a candidate → Inspect the diff → Try it on tasks.
Start from a base or saved candidate, edit its files with an assistant and evaluate the result. See bring your own harness. The customization proposal command can also generate an unevaluated candidate from explicitly supplied feedback and text, with options to refine instructions from failure feedback, propose memory-backed skills and export a source-checked patch for review.
[!TIP]
Already have a harness? Run harnessgrad customize inspect --root ./my-harness to see the components declared in its manifest, or see bring your own harness for the entrypoint and manifest format.
🚀 Quick start
Install HarnessGrad and configure an OpenAI-compatible editor endpoint as described in the customization guide. Choose the files and feedback you want to share with that endpoint; for example, put the following in feedback.json:
[{"id": "review-1", "kind": "failure", "text": "The task ended without running the project tests."}]
harnessgrad customize collect --root ./my-harness --skill skills/main.md --feedback ./feedback.json > request.json
harnessgrad customize propose --request request.json --max-calls 8 > proposal.json
harnessgrad customize review --request request.json --proposal proposal.json --root ./my-harness
Replace ./my-harness and skills/main.md with your own selected paths. The proposal includes candidate text and a diff for review; use the guide to export a patch and test it on a copy.
Customization workbench
The same Console used to inspect experiments also provides a workspace for editing harness candidates. See the workbench guide for setup and sessions.

Harness editor
If the platform, the measurement protocol or a curve produced here is useful in your work:
MIT. Third-party benchmark code keeps its own licence and provenance next to the code that uses it (PROVENANCE.json, LICENSE.upstream).