AndroidEnv - The Android Interaction & Learning Platform
English · 简体中文
AndroidEnv is a Python and
Kotlin platform for interacting with Android devices
(both virtualized emulators and physical hardware). Built for foundation model
UI agents, automated testing, and Reinforcement Learning (RL) research,
AndroidEnv provides low-latency device telemetry and control over any Android
application.
In AndroidEnv v2, device interaction is decoupled from Reinforcement
Learning task definitions:
AndroidDevice (Direct Device Control & Telemetry): A task-free handle
that connects to an on-device Kotlin Core server over WebSockets to stream
screenshots, accessibility trees, system logs, and sensor data while
injecting touch gestures, key events, text, and ADB commands.
AndroidEnv & TaskWrapper (Reinforcement Learning): An optional
dm_env layer on top of
AndroidDevice that adds Markov Decision Process (MDP) semantics (step(),
reset(), rewards, and episode terminations) via
task definitions.
- AndroidEnv Playground & MCP Server: A responsive web workbench for live
screen interaction, accessibility tree inspection, and ADB command
execution, plus a
stdio Model Context Protocol (MCP) server
(android-env-mcp) for LLM agents.
Index
Platform Features
- Real-World Android Ecosystem: Agents interact with an unaltered Android
OS—on Android Studio Emulators or physical devices—exposing the exact same
apps, UI frameworks, and system services that billions of human users
interact with daily.
- Rich, Subscription-Based Telemetry (
DeviceState): Subscribe on demand
to the signals your agent needs:
- Screenshots: High-speed, decoded RGB frame buffers (
width,
height, raw_pixels).
- Accessibility Trees: Full
AndroidAccessibilityForest window and UI
node hierarchies streamed directly from the on-device
AccessibilityService.
- System & Sensor Streams: Foreground package/activity, accessibility
events, logcat streams, audio output, battery & power metrics, and
hardware sensors (accelerometer, gyroscope, magnetometer, barometer,
ambient temperature, light, proximity).
- Expressive Action Interface (
Action):
- Touch Gestures: Normalized
(x, y) coordinates in [0.0, 1.0] with
TOUCH_DOWN, TOUCH_MOVE, and TOUCH_UP for taps, swipes, scrolls,
and drags.
- Hardware Keys & Native Text Entry: Inject Android keycodes (
BACK,
HOME, ENTER), set text directly on focused UI nodes
(ACTION_TYPE_SET_TEXT), or trigger clipboard actions (CUT, COPY,
PASTE).
- Direct ADB Execution: Run structured or generic
AdbRequest
commands via device.execute_adb_call().
- Composable RL Layer: When training RL policies, wrap
AndroidDevice
with android_env.rl.environment.AndroidEnv and TaskWrapper to use custom
.textproto tasks, reward parsers, and dm_env / Gymnasium wrappers.
Getting Started
Installation
Install AndroidEnv from PyPI:
$ python3 -m pip install android-env
Or clone the repository from main and build/test with Bazel (ensure
ANDROID_HOME points to a local Android SDK installation):
$ git clone https://github.com/google-deepmind/android_env/
$ cd android_env
$ bazel test //...
To build the PyPI wheel locally using Bazel:
$ bazel build //:android_env_wheel
1. Set Up an Android Device
Connect a physical Android device via USB debugging (adb devices) or create an
Android Virtual Device (AVD) using Android Studio. See the
Emulator guide
for step-by-step instructions.
2. Direct Control with AndroidDevice
For UI agents, foundation model evaluations, and automation scripts that do not
need RL episode loops:
from android_env.components import android_device_loader
from android_env.components import config_classes
from android_env.proto import device_env_service_pb2
config = config_classes.AndroidDeviceConfig(serial='emulator-5554')
with android_device_loader.load_android_device(config) as device:
# Subscribe to screenshots and accessibility trees:
device.update_subscriptions({
device_env_service_pb2.DEVICE_SIGNAL_SCREENSHOT,
device_env_service_pb2.DEVICE_SIGNAL_ACCESSIBILITY_TREE,
})
# Read the latest device state:
state = device.get_state()
# Tap the center of the screen:
device.send_action(
device_env_service_pb2.Action(
action_type=device_env_service_pb2.ACTION_TYPE_TOUCH_DOWN,
touch_position=device_env_service_pb2.Point(x=0.5, y=0.5),
)
)
device.send_action(
device_env_service_pb2.Action(
action_type=device_env_service_pb2.ACTION_TYPE_TOUCH_UP,
)
)
3. Launch the Interactive Web Playground
Explore the device screen, inspect live accessibility trees, and run ADB
commands from your browser (after pip install "android-env[playground]"):
$ android-env-playground --serial=emulator-5554
Or launch a local AVD from a source checkout with Bazel:
$ bazel run //android_env/playground:playground -- \
--config=android_env/playground/configs/emulator_config.py
To expose the device as a stdio Model Context Protocol (MCP) server for LLM
clients (after pip install "android-env[mcp]"):
$ android-env-mcp --serial=emulator-5554
4. Reinforcement Learning (dm_env)
To define MDP tasks with rewards and episode resets (after
pip install "android-env[rl]"), see the
Tasks guide,
Example tasks,
and
detailed instructions.
About
This library is developed and maintained by
Google DeepMind. You can find the original
technical report on arXiv, as well as the
introductory
blog post.
If you use AndroidEnv in your research, you can cite the paper using the
following BibTeX:
@article{ToyamaEtAl2021AndroidEnv,
title = "{AndroidEnv}: A Reinforcement Learning Platform for Android",
author = {Daniel Toyama and Philippe Hamel and Anita Gergely and
Gheorghe Comanici and Amelia Glaese and Zafarali Ahmed and Tyler
Jackson and Shibl Mourad and Doina Precup},
year = {2021},
eprint = {2105.13231},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
volume = {abs/2105.13231},
url = {http://arxiv.org/abs/2105.13231},
}
Disclaimer: This is not an official Google product.