🧪 Google Hackathon with AskUI
Join us as we explore how to build your first AI-powered agent for automating software testing. With AskUI, you'll create agents that interact with interfaces visually—just like a human. No matter your experience level, this is your chance to experiment, learn, and build something innovative.
⚙️ Prerequisites
Before you can set up and run your agent, ensure you have the following installed:
- 🔄 AskUI Shell - The command line tool for AskUI Agents.
- 🖊️ A code editor of your choice (e.g., VSCode, PyCharm).
- 🔑 An Anthropic API key - Required for the VisionAgent to function properly.
🔧 Setup
1. Install AskUI Agent OS
Agent OS is a device controller that allows agents to take screenshots, move the mouse, click, and type on the keyboard across any operating system.
Windows
AMD64
AskUI Installer for AMD64
ARM64
AskUI Installer for ARM64
Linux
⚠️ Warning: Agent OS currently does not work on Wayland. Switch to XOrg to use it.
AMD64
curl -L -o /tmp/AskUI-Suite-Latest-User-Installer-Linux-AMD64-Web.run https://files.askui.com/releases/Installer/Latest/AskUI-Suite-Latest-User-Installer-Linux-AMD64-Web.run
bash /tmp/AskUI-Suite-Latest-User-Installer-Linux-AMD64-Web.run
ARM64
curl -L -o /tmp/AskUI-Suite-Latest-User-Installer-Linux-ARM64-Web.run https://files.askui.com/releases/Installer/Latest/AskUI-Suite-Latest-User-Installer-Linux-ARM64-Web.run
bash /tmp/AskUI-Suite-Latest-User-Installer-Linux-ARM64-Web.run
MacOS
curl -L -o /tmp/AskUI-Suite-Latest-User-Installer-MacOS-ARM64-Web.run https://files.askui.com/releases/Installer/Latest/AskUI-Suite-Latest-User-Installer-MacOS-ARM64-Web.run
bash /tmp/AskUI-Suite-Latest-User-Installer-MacOS-ARM64-Web.run
2. Install vision-agent in your Python environment
Note: Requires Python version >=3.10.
3a. Authenticate with an AI Model Provider
|
AskUI INFO |
Anthropic INFO |
| ENV Variables |
ASKUI_WORKSPACE_ID, ASKUI_TOKEN |
ANTHROPIC_API_KEY |
| Supported Commands |
click(), get(), locate(), mouse_move() |
act(), click(), get(), locate(), mouse_move() |
| Description |
Faster Inference, European Server, Enterprise Ready |
Supports complex actions |
To get started, set the environment variables required to authenticate with your chosen model provider.
How to set an environment variable?
- Create
.env file similar to .env.template and update required credentials.
3b. Test with 🤗 Hugging Face AI Models (Spaces API)
You can test the Vision Agent with Huggingface models via their Spaces API. Please note that the API is rate-limited so for production use cases, it is recommended to choose step 3a.
Note: Hugging Face Spaces host model demos provided by individuals not associated with Hugging Face or AskUI. Don't use these models on screens with sensible information.
Supported Models:
Example Code:
agent.click("search field", model="OS-Copilot/OS-Atlas-Base-7B")
3c. Host your own AI Models
UI-TARS
You can use Vision Agent with UI-TARS if you provide your own UI-TARS API endpoint.
-
Step: Host the model locally or in the cloud. More information about hosting UI-TARS can be found here.
-
Step: Provide the TARS_URL and TARS_API_KEY environment variables to Vision Agent.
-
Step: Use the model="tars" parameter in your click(), get() and act() etc. commands or when initializing the VisionAgent.
▶️ Run Your Agent
This agent runs automated UI tests using Pytest and AskUI's VisionAgent. When run, it:
- Initializes the VisionAgent
- Executes test cases defined in the
tests/ directory
- Performs UI interactions like clicks and assertions
- Reports test results
To run your agent locally:
�� Happy Coding! 🚀