OpenETA
English · 简体中文
📄 Paper ·
🌐 Project Page ·
💻 GitHub
A new agentic paradigm for moving intelligence from the digital world into the physical world.
Physical intelligence is fundamentally about closing a causal loop with the world: understanding the
current state, acting to change it, observing the real consequence, updating the underlying belief,
and accumulating experience through continued interaction. Frontier agentic models have already
begun to exhibit this capability in the digital domain, where they reason over goals, invoke tools,
inspect execution results, and complete long-horizon tasks through feedback. What they still lack is
a reliable way to extend that loop into the physical world.
OpenETA explores this transition from Digital Agent to Physical Agent as a new paradigm. Instead of
reducing embodied intelligence to a single observation-to-action model, it places the Agent inside
the world's causal chain so that perception, action, verification, and learning become one continuous
process. Intelligence moves beyond processing digital information and begins to understand and
change reality.
Architecture ·
Action Pipeline ·
Parallel Evaluation ·
Real-robot Deployment
What's New
-
2026-10-16 — RoboDojo + human-assisted harness RSI: The
robodojo-v112 branch
adds a Codex CLI harness for RoboDojo, versioned task clarifications,
coarse native completion feedback, and measured finger collision geometry.
Across 42 tasks with GPT-6 Astra / high, interim cumulative coverage grows
from 54.8% Pass@1 → 64.3% RSI Pass@2 → 66.7% RSI Pass@3.
These are cross-version, human-assisted results; unfinished/unfilled attempts
count as no success in this provisional curve, not final fixed-version pass@k.
Setup
· Protocol and per-task results

-
2026-09-06 — Stage 2 Agent Harness: We publish the Stage 2 closed-loop
embodiment harness, adding 35 verified tool contracts, bounded
multimodal context with durable rollouts, a universal evaluation entry point,
reachability- and collision-aware motion, and a Human VLM operator console.
We have canceled state machine in host, which once led to limited exploration and intelligence of agents.
-
2026-08-03 — OpenETA for Codex: We released the
openeta-light
branch, which connects the Codex TUI to LIBERO through six typed tools and a
versioned Operator context. With gpt-5.6-sol at medium reasoning effort,
Codex + OpenETA reaches 70.8% Pass@1 and 90.0% Pass@5 across
all 130 LIBERO tasks.
-
2026-07-27 — Real-Robot Deployment: We released the
dev/real-robot-deployment
branch with the real-robot deployment stack, hardware interfaces, and physical-agent tooling.
-
2026-07-25 — Physical Hello World: OpenETA's first public release, extending the agentic
loop from the digital domain into the physical world.
Core Features
| Capability |
OpenETA system boundary |
| Causal closed loop |
Every world-changing action creates a fresh-observation obligation before reasoning, recovery, or completion can continue. |
| Replaceable Planner |
Different LLMs and VLMs connect through an OpenAI-compatible interface with primary and fallback endpoints. |
| Composable physical capabilities |
Perception, grasping, placement, motion, and future native policies compose through stable Tool and MCP interfaces. |
| Tool / Skill separation |
Tools are host-owned atomic capabilities; Skills are readable and reviewable experience, never hidden execution. |
| Host-owned supervision |
Schemas, provenance, safety gates, approval modes, and reviewers remain outside the Agent's authority. |
| Auditable Memory and Rollouts |
Session working memory and immutable evidence are stored separately so success, failure, and uncertainty can be replayed. |
| Unified sim-to-real boundary |
Simulators and robots share observation, action, result, and MCP lifecycle contracts. |
| Verifiable experience evolution |
Experience begins as a session-local candidate and advances only after review, canary, and holdout validation. |
The core loop is:
observe -> reason -> act -> verify world change -> observe again -> adapt
Architecture
Three boundaries constrain the system:
- Cognition / execution boundary: the Agent proposes intent, the Host decides whether to execute
it, and the Simulator or Robot retains authority over environment truth.
- Action boundary: read-only capabilities may run in parallel, while world-mutating AtomActions
execute one at a time and must satisfy the fresh-observation obligation before the next action.
- Evidence boundary: Tool completion, world-state change, and task success are distinct claims.
Only trusted environment receipts, rewards, termination signals, or checkers can establish task
completion.
Feature Matrix
| Agentic Planner |
Tools |
Simulator |
Real World |
- OpenAI-compatible LLM / VLM ✅
- Custom Planner ✅
- Primary / fallback endpoint ✅
|
- Semantic Tools
- Perception
- SAM 3 ✅
- UniDepth V2 ✅
- MolmoPoint-8B ✅
- Grasp & Placement
- AnyGrasp ✅
- GraspGenX ✅
- AnyPlace ✅
- Contact-GraspNet
- Control Tools
- Policy Adapters
- OpenVLA
- OpenVLA-OFT
- openpi
- GR00T N1.6 / N1.7
- Controllers & Safety
- robosuite
OSC_POSE ✅
- cuRobo ✅
|
- Manipulation Environments
- MetaWorld / Sawyer ✅
- ManiSkill / multiple robots ✅
- LIBERO / Franka Panda ✅
- RoboCasa365 / PandaOmron ✅
- BEHAVIOR-1K / R1Pro ✅
- Genesis / Franka ✅
|
- Observation
- RealSense D400 / L515 ✅
- Webcam / RTSP ✅
- UR5e state ✅
- Motion
|
✅ Adapted. Unmarked items are not yet adapted. Simulator backends use isolated venvs; see
Simulation Layer for installation, environment counts, and action spaces.
Quick Start
Install and Run
Requires Python 3.10+ and uv.
uv sync --extra dev
uv run openeta
Use /provider in the TUI to configure an OpenAI-compatible Planner, then enter a task directly.
Common commands include /tools, /memory, /approvement, /run, /sessions, and /resume.
For a bounded single-task run:
uv run openeta --once "inspect the current scene" --max-turns 8
Interfaces
| Entry point |
Purpose |
uv run openeta |
Interactive Agent TUI for provider setup, Tool inspection, execution, and session recovery |
uv run openeta --once "..." |
Single-task execution with a bounded turn count |
uv run openeta-batch --manifest ... |
Parallel evaluation with isolated episodes and human recovery |
uv run openeta --command preflight/run/iterate/inspect ... |
Unattended experiments and experience iteration |
uv run openeta-replay EPISODE_DIR --output replay.mp4 |
Reconstruct a timeline and video from episode logs |
uv run python -m sim.mcp_server |
Simulator MCP, REST API, and Web Dashboard |
uv run openeta-robocasa-benchmark ... |
RoboCasa365 manifests, task suites, and benchmark utilities |
See uv run openeta --help, uv run openeta-batch --help, and each subcommand's --help for the
complete CLI reference.
Simulator and Dashboard
Install one simulator backend, then start the MCP service and live Dashboard:
bash scripts/setup_envs.sh libero
uv run python -m sim.mcp_server
# Open http://localhost:8765/
Evaluation
Validate a parallel-evaluation manifest without connecting to a model or MCP service:
uv run openeta-batch \
--manifest examples/parallel_libero_eval.json \
--validate-only
The repository includes LIBERO canary, train, holdout, and full-evaluation manifests. After
connecting a Planner and the required MCP services, run isolated episodes in parallel:
uv run openeta-batch \
--manifest examples/parallel_libero_eval.json \
--concurrency 10 \
--approvement reviewed_autonomy \
--output outputs/parallel-libero.json
Unattended experiments follow four stages: preflight -> run -> iterate -> inspect.
uv run openeta --command preflight \
--manifest examples/parallel_libero_eval.json
uv run openeta --command iterate \
--train-manifest examples/parallel_libero_six_train.json \
--validation-manifest examples/parallel_libero_six_holdout.json \
--experiment-id libero-skill-v1 \
--rounds 3 \
--approvement reviewed_autonomy \
--on-need-human fail
uv run openeta --command inspect \
--experiment-id libero-skill-v1
Process completion does not imply task success. Final outcomes are established by trusted
environment receipts, rewards, checkers, and retained rollout evidence.
Roadmap
Documentation
| Document |
Scope |
| Architecture |
Boundaries across the Agent, Adapter, Simulator, and MCP layers |
| Agent command pipeline |
Command schemas, Tool contracts, AtomActions, and safety gates |
| Simulation layer |
Installation, environment registry, MCP, REST, and Dashboard |
| Parallel simulator evaluation |
Concurrency, budgets, human recovery, and experience promotion |
| Rollout data contract |
Session data layers and immutable evidence |
| Code policy runtime |
Bounded code backends and sandbox boundaries |
| Calibration lifecycle |
Calibration proposal, review, and publication |
| Real-robot deployment |
Cameras, robot arms, real-environment MCP, and safety boundaries |
Contributing
OpenETA remains under active research and development. New Planners, Tools, simulators, robots, and
Skills should preserve the existing observation, command, and result contracts and include tests for
side effects, failure semantics, and resource cleanup. Changes to schemas, Tool contracts, task
success semantics, or safety gates must update the corresponding design documents.
uv sync --extra dev
uv run pytest
Acknowledgements
OpenETA's initial simulator integrations were migrated and adapted from
RLinf. The project builds on the open-source robotics ecosystems
around LIBERO, MetaWorld, ManiSkill, RoboCasa, BEHAVIOR, MuJoCo, SAPIEN, and OmniGibson. Its
perception and manipulation capabilities benefit from SAM 3, UniDepth, AnyGrasp, AnyPlace,
Contact-GraspNet, GraspGenX, and Molmo.
Community
Join the OpenETA community on WeChat (微信技术交流群). Scan the QR code to enter.
Citation and License
If you find OpenETA useful in your research, please cite:
@misc{chen2026etanewagenticparadigm,
title={ETA: A New Agentic Paradigm for Embodied Tasks},
author={Yitong Chen and Zezheng Huai and Sixian Li and Yubang Wang and Haozhe Zhang and Yifei Zhang and Hechang Chen and Jingjing Gong and Yu-Gang Jiang and Xipeng Qiu},
year={2026},
eprint={2608.03924},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2608.03924}
}
OpenETA is licensed
under the Apache License 2.0. Third-party components
and migrated code remain subject to their respective upstream licenses and retained source-file
notices.