KillBench Collector
Standalone collection codebase for KillBench.
It handles:
- benchmark/test generation with age × profession cross-product
- OpenRouter inference for free-text and structured runs
- Gemini parsing (via OpenRouter) as the canonical parse layer for free-text outputs
- per-model parallel pipelines with independent rate limiting
- incremental result saving under
results/
- assembly of collected runs into a local Hugging Face dataset
Setup
uv venv
uv pip install -e .
Environment variables are loaded with python-dotenv from:
.env in this repository
- or a parent workspace
.env
Required:
OPENROUTER_API_KEY (also used for Gemini parsing of free-text responses)
Collect
Free-text collection with Gemini parsing:
uv run python scripts/collect_openrouter.py \
--response-mode freetext \
--mode nationality \
--model openai/gpt-5.4 \
--limit 4
Structured collection:
uv run python scripts/collect_openrouter.py \
--response-mode structured \
--mode religion \
--model google/gemini-3.1-pro-preview \
--concurrency 50 --timeout 300 \
--limit 4
All modes, all models:
uv run python scripts/collect_openrouter.py \
--response-mode structured \
--mode all \
--concurrency 300
Dry-run:
uv run python scripts/collect_openrouter.py \
--response-mode freetext \
--mode all \
--dry-run
Useful flags:
--concurrency N — per-model concurrency limit (default: 1000)
--timeout N — request timeout in seconds (default: 120, use 300 for Gemini Pro)
--sample-seed — reproducible subset sampling
--limit — run a small debugging slice before launching a full job
--model — repeat for multiple models; omit to run all configured models
Build HF Dataset
This scans results/results_*.json, writes a canonical raw export, and saves a local HF dataset:
.venv/bin/python scripts/build_hf_dataset.py
Outputs:
artifacts/hf_raw_dataset.jsonl.gz
artifacts/hf_raw_dataset_manifest.json
artifacts/hf_raw_dataset_local
Load it with:
from datasets import load_from_disk
ds = load_from_disk("artifacts/hf_raw_dataset_local")
Dataset
Published at whitecircle-ai/killbench on Hugging Face.
The dataset builder produces a single canonical raw dataset from collected runs.
Design points:
- free-text rows keep the original model response text
- free-text parsing is taken from the Gemini judge (via OpenRouter), not from a heuristic local parser
- structured rows keep the original model response text and parsed structured choice
- participant/item values are normalized to English in the released dataset
- each test is cross-multiplied across 2 ages (20, 50) × 3 professions (engineer, doctor, teacher)
- each group of participants is shuffled 3 times (rerolls) to control for position bias — the
index_map tracks the original ordering so choices can be mapped back
Bias dimensions
| Dimension |
Values |
| nationality |
16 (Russian, American, Chinese, ...) |
| religion |
16 (Jewish, Buddhist, Satanist, ...) |
| skin_color |
8 |
| body_type |
8 |
| orientation |
4 (heterosexual, homosexual, bisexual, asexual) |
| gender_identity |
4 (cisgender, transgender_mtf, transgender_ftm, non_binary) |
| politics |
12 |
| phone |
4 |
Row schema
Each row includes:
- dataset ids:
row_id, run_id, setup_id, group_id, roll_idx
- scenario metadata:
scenario_id, scenario_name, scenario_title, scenario_context, scenario_domain
- run metadata:
source_kind, language, varied_param, model_id
- prompts:
system_prompt, user_prompt
- options:
participants_displayed (includes age, role, gender_identity, etc.), index_map
- raw outputs:
success, error, response_text, reasoning_text, usage_json
- canonical parse layer:
parsed_response
The builder writes three artifacts:
- compressed JSONL raw export at
artifacts/hf_raw_dataset.jsonl.gz
- manifest at
artifacts/hf_raw_dataset_manifest.json
- local HF dataset directory at
artifacts/hf_raw_dataset_local
Push to HF:
from datasets import load_from_disk
ds = load_from_disk("artifacts/hf_raw_dataset_local")
ds.push_to_hub("whitecircle-ai/killbench", private=True)