dify
Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.
- 162k
- Python
- 1 h ago

๐ฆ ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery
Repo InternScience/ResearchClawBench, read from GitHub 1 min ago. InternScience has not joined Take My Repo.
InternScience/ResearchClawBench is a public repo on GitHub by InternScience. ๐ฆ ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery. It declares the MIT license on GitHub. It is written mainly in Jupyter Notebook, has 282 stars and last changed on 2026-10-28. Nobody has reviewed it on Take My Repo yet, which says nothing about its quality. Read from GitHub on 2026-11-22.
In the maintainer's words. Shown from GitHub; Take My Repo does not keep a copy.
Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery
Quick Start | Submit Tasks | How It Works | Domains | Leaderboard | Add Your Agent
ResearchClawBench is a benchmark that measures whether AI coding agents can independently conduct scientific research โ from reading raw data to producing publication-quality reports โ and then rigorously evaluates the results against real human-authored papers.
Unlike benchmarks that test coding ability or factual recall, ResearchClawBench asks: given a curated scientific workspace and the same research goal, can an AI agent arrive at the same (or better) scientific conclusions?
| ๐ Two-Stage Pipeline Autonomous research + rigorous peer-review-style evaluation |
๐งช 40 Real-Science Tasks 10 disciplines, curated datasets from published papers |
๐๏ธ Expert-Annotated Data Tasks, rubrics (checklists) & datasets curated by domain experts |
๐ค Multi-Agent Support Claude Code, Codex CLI, OpenClaw, ResearchClaw, ... & custom agents |
| ๐ Re-Discovery to New-Discovery 50 = match the paper, 70+ = surpass it |
๐ Fine-Grained Rubric (Checklist) Per-item keywords, weights & reasoning |
๐ก Live Streaming UI Watch agents code, plot & write in real-time |
๐ Lightweight Dependencies Pure Flask + vanilla JS, no heavy frameworks |
Most AI benchmarks evaluate what models know. We evaluate what agents can do.
rcb-eval, a YAML-configured command-line evaluation workflow powered by ResearchHarness, supporting concurrent runs, repeated trials, automatic scoring, and Markdown evaluation reports with per-run and per-task statistics.agents.json for easy customization.Every task in ResearchClawBench is built through a rigorous, expert-driven pipeline to ensure scientific validity and reproducibility:
flowchart TD
A["๐ High-Quality Paper Collection\n(Target Paper)"] --> B["๐งโ๐ฌ Human Expert Extraction\n(Core Task Instructions)"]
B --> C["๐ Evaluation Rubric (Checklist)\n(Criteria + Keywords + Weights)"]
B --> D["๐ Data & Related Work Collection\n(Datasets + Reference Papers)"]
C --> E["โ
Human Reproduction & Validation\n(Verify rubric (checklist) is reproducible)"]
D --> E
style A fill:#e0f2fe,stroke:#0284c7,stroke-width:2px
style B fill:#fef3c7,stroke:#f59e0b,stroke-width:2px
style C fill:#fce7f3,stroke:#ec4899,stroke-width:2px
style D fill:#f0fdf4,stroke:#22c55e,stroke-width:2px
style E fill:#f5f3ff,stroke:#8b5cf6,stroke-width:2px
High-Quality Paper Collection โ Domain experts select recent, high-impact publications with clear methodology and reproducible results across 10 scientific disciplines.
Expert Task Extraction โ Human experts read each paper and distill the core research task into structured instructions, identifying the key scientific question, input data, and expected outputs.
Rubric (Checklist) Design โ Experts create a fine-grained evaluation rubric (checklist) with weighted criteria (text and image items), each with specific technical keywords that a judge must verify.
Data & Related Work Collection โ The datasets and related reference materials are curated to form a research workspace for the task.
Human Reproduction & Validation โ Human researchers independently reproduce the paper's results from the provided workspace and instructions, verifying that every rubric (checklist) item is achievable. This ensures the benchmark is fair and the rubric (checklist) is grounded in reality.
ResearchClawBench operates in two distinct stages:
flowchart LR
subgraph Stage1["Stage 1 — Auto Research"]
A["Raw Data\n+ Instructions"] --> B["AI Agent\n(autonomous)"]
B --> C["Code"]
C --> G["Figures\n+ Report"]
end
subgraph Stage2["Stage 2 — Evaluation"]
G --> D["LLM Judge"]
E["Target Paper"] --> H["Rubric (Checklist)"]
H --> D
D --> F["Reasoning\n+ Per-Item Scores"]
end
style Stage1 fill:#f0f4ff,stroke:#3b82f6,stroke-width:2px
style Stage2 fill:#fff7ed,stroke:#f59e0b,stroke-width:2px
The AI agent receives a workspace containing raw datasets, reference materials, and task instructions. It must independently:
report/report.md) with figures, methodology, results, and discussionNo hand-holding. No chain-of-thought hints. The agent works in its own sandboxed workspace with full tool access โ just like a real researcher.
Evaluation view โ target paper (left), AI report (center), scored rubric (checklist) panel (right)
Once the agent finishes, its report is evaluated against the original published paper using a fine-grained rubric (checklist). The judge receives the task instructions, the AI report, and the rubric (checklist) criteria โ then scores each item using a dual-mode rubric:
flowchart TD
subgraph Inputs
I["INSTRUCTIONS.md\n(task background)"]
R["Agent Report\n(text + figures)"]
TP["Target Paper"]
CL["Rubric (Checklist)"]
end
TP --> CL
I & R & CL --> J["Multimodal LLM Judge"]
J --> DET{"Determine\nEvaluation Mode"}
DET -->|"Quantitative\nresults"| OBJ["Mode A: Target Optimization\n(Comparison through quantitative indicators)"]
DET -->|"Qualitative\nreasoning"| SUB["Mode B: Diagnostic Analysis\n(Comparison through logical evidence)"]
OBJ --> SO["Score by metric\naccuracy vs paper"]
SUB --> SS["Score by evidence\nstrength vs paper"]
SO & SS --> T["Reasoning\n+ Per-Item Scores\nโ Weighted Total"]
style Inputs fill:#f0f4ff,stroke:#3b82f6,stroke-width:2px
style J fill:#fef3c7,stroke:#f59e0b,stroke-width:2px
style OBJ fill:#dbeafe,stroke:#3b82f6,stroke-width:2px
style SUB fill:#fce7f3,stroke:#ec4899,stroke-width:2px
style T fill:#f0fdf4,stroke:#22c55e,stroke-width:2px
Each rubric (checklist) item includes:
text for methodology/findings, image for figure comparison (multimodal vision)The judge automatically determines which evaluation mode applies to each item, then scores it with the corresponding rubric (see below).
For rubric (checklist) items involving specific numerical results, metrics, or quantitative outcomes:
| Score | Meaning |
|---|---|
| 0 | Criterion completely absent |
| 1โ10 | Mentioned but no quantitative results provided |
| 11โ20 | Results given but methodology has fundamental errors |
| 21โ30 | Significant methodological flaws; metrics deviate severely |
| 31โ40 | Methodology mostly correct but metrics notably worse than the paper |
| 41โ50 | Metrics roughly comparable to the paper |
| 51โ60 | Metrics slightly better than the paper |
| 61โ70 | Metrics clearly better than the paper |
| 71โ80 | Methodology and metrics both substantially improved |
| 81โ90 | Metrics dramatically surpass the paper |
| 91โ100 | Breakthrough results far exceeding the paper |
For rubric (checklist) items involving theoretical explanations, mechanistic insights, or interpretive analysis:
| Score | Meaning |
|---|---|
| 0 | Criterion completely absent |
| 1โ10 | Mentioned only with vague, generic statements |
| 11โ20 | Some description but no substantive analysis |
| 21โ30 | Analysis attempted but evidence insufficient or logic has gaps |
| 31โ40 | Correct direction but lacks depth; key arguments missing |
| 41โ50 | Analysis depth and rigor comparable to the paper |
| 51โ60 | More supporting evidence provided than the paper |
| 61โ70 | More complete logical chain and more rigorous argumentation |
| 71โ80 | Significantly deeper analysis with novel insights |
| 81โ90 | Analysis depth far exceeds the paper |
| 91โ100 | Original contributions with breakthrough insights |
Strict by design. The judge is highly skeptical of AI-generated content โ plausible-sounding claims must be backed by concrete evidence. Longer reports do not score higher. Substance over style.
Each domain contains 4 carefully curated tasks with complete experimental data from real published research:
| Domain | Example Topics | Data Types |
|---|---|---|
| Astronomy | Black hole superradiance, Bayesian stellar inference | .dat, .csv |
| Chemistry | GNN molecular prediction, protein-ligand docking | .pdb, .sdf, .csv |
| Earth | Glacier mass balance, climate datasets | .csv, multi-region series |
| Energy | Battery degradation, renewable energy modeling | .xlsx, time series |
| Information | NLP benchmarks, deep learning analysis | .pdf, .tex, .ipynb |
| Life | Nanopore sequencing, genomic analysis | .csv, .xlsx |
| Material | Materials property prediction, pretrained models | .pt, .csv |
| Math | Multi-agent pathfinding, optimization | .json, .npy, grid maps |
| Neuroscience | Neural decoding, brain signal processing | .csv, .h5, .yaml |
| Physics | Quantum geometry, superfluid stiffness | .h5, .json, .csv |
40 tasks total โ each a curated research challenge selected from high-quality human-authored publications, spanning the full spectrum from data analysis to novel scientific insight.
You can view the leaderboard on our Website, which is updated in real time.
The built-in dashboard aggregates the best score per (task, agent) pair and displays:
The frontier represents the state of the art โ every point above 50 is uncharted territory where AI surpasses human researchers on that specific task.
ResearchClawBench/
โโโ evaluation/ # Core evaluation framework
โ โโโ server.py # Flask API + SSE streaming
โ โโโ run_task.py # Workspace setup + agent subprocess
โ โโโ score.py # Multimodal LLM scoring engine
โ โโโ config.py # Paths, constants, loads agents.json
โ โโโ agents.json # Agent presets (add your own here)
โ โโโ instructions_tmpl.py # Unified prompt template for all agents
โ โโโ utils.py # File tree, path safety, discovery
โ โโโ static/app.js # Single-file frontend
โ โโโ templates/index.html # Entry point
โโโ tasks/ # 40 research tasks
โ โโโ Astronomy_000/
โ โ โโโ task_info.json # Task description + data manifest
โ โ โโโ data/ # Raw experimental datasets
โ โ โโโ related_work/ # Reference papers
โ โ โโโ target_study/ # Paper + rubric (checklist) + images
โ โโโ Chemistry_000/
โ โโโ ... # 10 domains x 4 tasks
โโโ workspaces/ # Generated at runtime (gitignored)
git clone https://github.com/InternScience/ResearchClawBench.git # If you only need to run evaluations, you can instead use: # git clone --depth 1 https://github.com/InternScience/ResearchClawBench.git cd ResearchClawBench pip install -r evaluation/requirements.txt
The Hugging Face dataset mirror at InternScience/ResearchClawBench currently includes 16 community-contributed tasks beyond the 40 tasks in this repository, all packaged in the same tasks/<TaskID>/... layout.
If you want to use these extra tasks directly in this repository, set --output-dir to your local tasks/ directory.
Download the helper script:
pip install huggingface_hub curl -L -o download_tasks.py https://huggingface.co/datasets/InternScience/ResearchClawBench/resolve/main/download_tasks.py
Download all mirrored Hugging Face tasks:
python download_tasks.py --all --output-dir /path/to/ResearchClawBench/tasks
Download one or more specific tasks:
python download_tasks.py --task Astronomy_004 --task Physics_004 --output-dir /path/to/ResearchClawBench/tasks
The downloaded files are placed directly under that tasks directory, for example /path/to/ResearchClawBench/tasks/Astronomy_004/....
Any task directory placed under tasks/ with a valid task_info.json will be discovered automatically by the evaluation UI/API.
Create evaluation/.env with your scoring model credentials:
JUDGE_API_KEY=sk-xxx JUDGE_API_BASE=https://api.openai.com/v1 JUDGE_MODEL_NAME=gpt-5.1
Install whichever agent(s) you plan to benchmark. You do not need every built-in preset.
| Agent | Official installation guide | Notes |
|---|---|---|
| Claude Code | Claude Code overview | Anthropic official docs |
| Codex CLI | Codex CLI | OpenAI official docs |
| ARIS Codex | Auto-claude-code-research-in-sleep | Imported runs are supported in the UI. The public preset is documentation-only for now; one-click launch is not yet supported. |
| OpenClaw | OpenClaw | Official website and setup entry |
| Nanobot | HKUDS/nanobot | Official GitHub repository |
| EvoScientist | EvoScientist/EvoScientist | Official GitHub repository |
| ResearchClaw | ymx10086/ResearchClaw | pip install |
| LingTai | Lingtai-AI/lingtai | Official GitHub repository |
| ResearchHarness | InternScience/ResearchHarness | Lightweight baseline harness for testing different LLMs; install with pip install researchharness. The Web UI preset still uses a local checkout path in agents.json. |
python -m evaluation
Open http://localhost:5000 โ browse tasks, pick an agent, hit Start Run, and watch the research happen live.
After a run completes, switch to the Evaluation tab and click Score. The multimodal LLM judge evaluates each rubric (checklist) item and returns per-item scores with reasoning.
For scripted LLM sweeps with ResearchHarness, use python3 -m evaluation.cli_eval or the repository-local python3 rcb-eval entrypoint. The CLI builds the same standard ResearchClawBench run workspaces used by the Web UI, runs the installed researchharness package, scores completed reports, and prints per-run and aggregate summary tables.
The CLI loads evaluation/.env and also respects environment variables set in the shell. The YAML intentionally separates the evaluated model from the judge model:
agent_model: name_env: AGENT_MODEL_NAME # model being evaluated api_base_env: AGENT_API_BASE api_key_env: AGENT_API_KEY judge_model: enabled: true name_env: JUDGE_MODEL_NAME # model used only for scoring api_base_env: JUDGE_API_BASE api_key_env: JUDGE_API_KEY
Real ResearchHarness batch runs also require tool credentials inherited by the installed researchharness package: SERPER_KEY for web/scholar search, JINA_KEY for web fetch, and MINERU_TOKEN for PDF reading. These can be placed in evaluation/.env; the CLI checks that they exist before creating any batch workspace and runs a lightweight tool preflight before each run. The preflight uses the exact same process environment as the real evaluation run, including any proxy-related variables; the CLI does not rewrite proxy settings. If tool calls fail in your local network environment, adjust or disable your proxy settings in the shell or .env, then rerun. Runs with failed tool preflight are marked as skipped and summarized in eval_report_<batch_id>.md.
Provider-specific OpenAI-compatible request options can be passed through agent_model.extra_body. For example, Qwen thinking mode can be toggled with enable_thinking; when thinking is enabled, keep researchharness.max_output_tokens high enough for the model to produce a final answer after thinking.
pip install -r evaluation/requirements.txt
# Edit agent_model, judge_model, tasks, repeats, and concurrency first.
cp eval_configs/researchharness_example_1_single_task.yaml eval_configs/my_eval.yaml
AGENT_MODEL_NAME=gpt-5.4 \
AGENT_API_BASE=https://your-agent-api.example/v1 \
AGENT_API_KEY=sk-agent-xxx \
JUDGE_MODEL_NAME=gpt-5.1 \
JUDGE_API_BASE=https://your-judge-api.example/v1 \
JUDGE_API_KEY=sk-judge-xxx \
SERPER_KEY=serper-xxx \
JINA_KEY=jina-xxx \
MINERU_TOKEN=mineru-xxx \
python3 -m evaluation.cli_eval eval_configs/my_eval.yamlProvided examples:
eval_configs/researchharness_example_1_single_task.yaml: one task with one repeat.eval_configs/researchharness_example_2_mixed_repeats.yaml: multiple tasks with different repeat counts.eval_configs/researchharness_example_3_all_tasks.yaml: all official tasks with tasks: all.eval_configs/researchharness_example_4_qwen_thinking.yaml: Qwen example with agent_model.extra_body.enable_thinking.Useful validation command:
python3 -m evaluation.cli_eval eval_configs/my_eval.yaml --dry-run --skip-secret-check --no-score
Each CLI batch is written under workspaces/cli_runs/cli_<timestamp>_<random>/, with individual runs stored as cli_<task_id>_<timestamp>_<random>/ subdirectories. The batch directory also contains eval_report_<batch_id>.md, which records the config metadata, runtime summary, run directories, trace directories, per-run results, per-task summary with duration columns, and overall statistics. Each run keeps the standard contents: _meta.json, _agent_output.jsonl, report/report.md, and _score.json when scoring is enabled. ResearchHarness events are captured in _agent_output.jsonl and also saved as sibling trace directories named <run_id>_trace/ in the same batch directory. The role prompt is loaded from the installed researchharness package, so this CLI does not require a separate ResearchHarness checkout. The shorter python3 rcb-eval eval_configs/my_eval.yaml entrypoint is also provided.
After CLI evaluation, python3 rcb-clear prints a dry-run summary of duplicated task inputs under workspaces/cli_runs/, including reclaimable size and affected run counts. Use python3 rcb-clear --yes to delete only duplicated CLI inputs (data/ and related_work/ inside CLI run directories). Web UI runs, INSTRUCTIONS.md, reports, code, outputs, scores, metadata, and ResearchHarness traces are preserved.
ResearchClawBench ships with built-in support for Claude Code, Codex CLI, ARIS Codex, OpenClaw, Nanobot, EvoScientist, ResearchClaw, LingTai, plus a lightweight ResearchHarness baseline:
Agent configuration is stored in evaluation/agents.json. To add a new agent, simply append an entry:
{
"my_agent": {
"label": "My Agent",
"icon": "M",
"logo": "/static/logos/my_agent.svg",
"cmd": "my-agent run -m <PROMPT> -w <WORKSPACE>"
}
}| Placeholder | Replaced With | Notes |
|---|---|---|
<PROMPT> |
Prompt content (via file path or $(cat ...)) |
Required. For -p style flags, replaced with file path; otherwise replaced with "$(cat 'path')" to pass content |
<WORKSPACE> |
Absolute path to the workspace directory | Optional. Only replaced if present in cmd |
The prompt injected into <PROMPT> is auto-generated from evaluation/instructions_tmpl.py, which combines a unified agent persona (autonomous execution guidelines, workspace sandbox rules) with task-specific instructions (description, data files, deliverables). All agents receive the exact same prompt โ no code changes required, just edit the JSON file and restart the server.
The GitHub benchmark repository stays focused on the base 40 tasks. New task submissions should go through the ResearchClawBench submission Space, which validates the uploaded zip and opens a PR against the Hugging Face dataset repository for maintainer review.
After a submission is merged into the dataset repository, you can download it into your local tasks/ directory with download_tasks.py, and the evaluation UI/API will discover it automatically.
We welcome contributions in several forms โ see CONTRIBUTING.md for detailed guidelines.
๐ง Email: xu_wanghan@sjtu.edu.cn
๐ฌ Community:
If you would like to cite our work, please use the following BibTeX.
@article{xu2026researchclawbench, title={ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research}, author={Xu, Wanghan and Li, Shuo and Ye, Tianlin and Cao, Qinglong and Chen, Yixin and Gao, Hengjian and Wang, Yiheng and Li, Qi and Li, Kun and Xu, Sheng and others}, journal={arXiv preprint arXiv:2606.07591}, year={2026} }
InternScience/ResearchClawBench is a public repo on GitHub by InternScience. ๐ฆ ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery.
It declares the MIT license on GitHub.
Its maker has not tagged a system, so it is unknown here. Its README says what it needs.
Open "Take it" on its page: Take My Repo writes an install plan for your machine and hands it to your AI agent (Claude Code, Codex or Cursor), which asks you before each step. Take My Repo has not tested it on any machine.
Its last change on GitHub was on 2026-10-28. GitHub counts 1 open issue and pull requests. Take My Repo does not judge from that whether it is maintained.
Nobody has reviewed it on Take My Repo yet, which says nothing about its quality.
Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.

The agent that grows with you
Turn any AI agent into an AI Scientist. The #1 Agent Skills library for science, used by 250,000+ scientists worldwide. 177 ready-to-use validated skills plus 100+ scientific databases covering biology, chemistry, medicine, and drug discovery. Compatible with Cursor, Claude Code, Codex, Pi, Antigravity, and the open Agent Skills standard.

Turn any PDF or image document into structured data for your AI. A powerful, lightweight OCR toolkit that bridges the gap between images/PDFs and LLMs. Supports 100+ languages.

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
Reverse engineer anything with agents, from app behavior down to native binaries.
InternAgent-1.5: A Unified Agentic Framework for Long-Horizon Autonomous Scientific Discovery
GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation

An open, curated collection of Agent Skills for scientific research โ clone it, use it, extend it!
MLEvolve is an open-source autonomous system for end-to-end machine learning algorithm design and optimization powered by progressive search and experience-driven memory.

A curated collection of papers, datasets, and resources on Scientific Datasets and Large Language Models (LLMs)
This page is also available as structured data, with the same sources and dates.
/api/v1/resources/gh/InternScience/ResearchClawBenchTake My Repo has not tested this on your machine. Requirements come from the project's own documentation.