AgentAccess
macOS Accessibility automation framework. Control any app via the Accessibility API — inspect elements, click buttons, type text, manage windows, capture screenshots, and more.
Installation
dependencies: [
.package(url: "https://github.com/AgentiLoop/AgentAccess.git", from: "2.10.24"),
]
Quick Start
import AgentAccess
let ax = AccessibilityService.shared
// Check permission first
guard AccessibilityService.hasAccessibilityPermission() else {
AccessibilityService.requestAccessibilityPermission()
return
}
Features
Inspect Elements
// Inspect element at screen coordinates
let info = ax.inspectElementAt(x: 500, y: 300, depth: 3)
// Find element by role and title
let result = ax.findElement(role: "AXButton", title: "OK", value: nil, appBundleId: nil, timeout: 5)
// Get focused element
let focused = ax.getFocusedElement(appBundleId: "com.apple.Safari")
// Get children of an element
let children = ax.getChildren(role: "AXGroup", title: nil, value: nil, appBundleId: nil, depth: 2)
// Get element properties
let props = ax.getElementProperties(role: "AXTextField", title: "Search", value: nil, appBundleId: nil)
Click, Type, Scroll
// Click at coordinates
ax.clickAt(x: 100, y: 200, button: "left", clicks: 1)
// Click an element by description
ax.clickElement(role: "AXButton", title: "Submit", value: nil, appBundleId: nil, timeout: 5, verify: true)
// Type text at coordinates
ax.typeText("Hello world", at: 100, y: 200)
// Type into a specific element
ax.typeTextIntoElement(role: "AXTextField", title: "Email", text: "user@example.com", appBundleId: nil, verify: true)
// Scroll
ax.scrollAt(x: 500, y: 400, deltaX: 0, deltaY: -5)
// Key press
ax.pressKey(virtualKey: 36, modifiers: []) // Return key
// Drag
ax.drag(fromX: 100, fromY: 200, toX: 300, toY: 400, button: "left")
Wait for Elements
// Wait with fixed polling
let found = ax.waitForElement(role: "AXButton", title: "Done", value: nil, appBundleId: nil, timeout: 10, pollInterval: 0.5)
// Wait with adaptive polling (starts fast, slows down)
let found = ax.waitForElementAdaptive(role: "AXSheet", title: nil, value: nil, appBundleId: nil, timeout: 10, initialDelay: 0.1, maxDelay: 1.0)
Window Management
// List all visible windows
let windows = ax.listWindows(limit: 50)
// Get window frame
let frame = ax.getWindowFrame(windowId: 12345)
// Resize and move a window
ax.setWindowFrame(appBundleId: "com.apple.Safari", x: 0, y: 0, width: 1200, height: 800)
// Highlight an element (visual overlay)
ax.highlightElement(role: "AXButton", title: "OK", value: nil, appBundleId: nil, x: nil, y: nil, duration: 2.0, color: "green")
Menu Bar
// Click a menu item by path
ax.clickMenuItem(appBundleId: "com.apple.Safari", menuPath: ["File", "New Window"])
// Show a menu
ax.showMenu(role: nil, title: "File", value: nil, appBundleId: "com.apple.Safari")
App Management
// Launch, activate, quit, list apps
ax.manageApp(action: "launch", bundleId: "com.apple.Safari", name: nil)
ax.manageApp(action: "activate", bundleId: "com.apple.Safari", name: nil)
ax.manageApp(action: "quit", bundleId: "com.apple.Safari", name: nil)
ax.manageApp(action: "list", bundleId: nil, name: nil)
Screenshots
// Capture a screen region
ax.captureScreenshot(x: 0, y: 0, width: 800, height: 600)
// Capture a specific window
ax.captureScreenshot(windowID: 12345)
// Capture all visible windows
ax.captureAllWindows()
Element Properties
// Set properties on an element
ax.setProperties(role: "AXSlider", title: nil, value: nil, appBundleId: nil, properties: ["AXValue": 0.75])
// Read focused element content
ax.readFocusedElement(appBundleId: "com.apple.TextEdit")
// Scroll to an element
ax.scrollToElement(role: "AXButton", title: "Save", appBundleId: nil)
Permission Gating
Users can disable specific AX actions for safety:
let perms = AccessibilityPermissions.shared
// Check if restricted
perms.isRestricted("AXPress") // false (enabled by default)
// Disable an action
perms.toggle("AXDelete") // now restricted
// Re-enable all
perms.enableAll()
AX Actions (30)
| Group |
Actions |
| Core |
AXPress, AXConfirm, AXActivate, AXCancel, AXShowMenu, AXDismiss |
| Values |
AXIncrement, AXDecrement |
| Disclosure |
AXExpand, AXCollapse, AXOpen |
| Window |
AXRaise, AXZoom, AXMinimize |
| Text |
AXCopy, AXCut, AXPaste, AXSelect, AXSelectAll |
| Scroll |
AXScrollToVisible, AXScrollPageUp/Down/Left/Right |
| Focus |
AXFocus |
| UI |
AXShowDefaultUI, AXShowAlternateUI |
| Content |
AXDelete, AXPick |
AX Roles Reference
Web Content (browser automation)
| Role |
Element |
| AXWebArea |
Root of web page content |
| AXLink |
Clickable hyperlink |
| AXImage |
Image |
| AXTable, AXRow, AXColumn, AXCell |
Table structure |
| AXHeading |
h1-h6 headings |
| AXList, AXListItem |
Lists |
| AXBlockQuote |
Blockquote |
| AXForm |
Form element |
App UI
| Role |
Element |
| AXApplication |
App root |
| AXWindow, AXSheet, AXDialog, AXPopover |
Windows |
| AXToolbar, AXTabGroup, AXSplitGroup |
Layout |
| AXScrollArea, AXGroup, AXOutline |
Containers |
| AXMenuBar, AXMenu, AXMenuItem |
Menus |
| AXButton, AXCheckBox, AXRadioButton |
Controls |
| AXTextField, AXTextArea, AXStaticText |
Text |
| AXSlider, AXStepper, AXProgressIndicator |
Values |
| AXPopUpButton, AXComboBox |
Dropdowns |
| AXColorWell, AXDateField |
Pickers |
| AXDisclosureTriangle |
Disclosure |
Audit Logging
All operations are logged via AgentAudit to os.log. View in Console.app under subsystem <host app bundle ID>.audit, category Accessibility.
Requirements
- macOS 14+
- Swift 6.4+
- Accessibility permission (System Settings > Privacy & Security > Accessibility)
Part of AgentiLoop Agent!
AgentAccess is one of the open-source building blocks of AgentiLoop Agent!, the native AI agent for macOS 14.6+ on Apple Silicon and Intel. Agent! codes in Xcode, drives any Mac app, runs shell as you or as root, and works with 23 LLM providers plus on-device Apple Intelligence.
🌐 agentiloop.ai · ⬇️ Download Agent! · 🍺 brew install --cask agentiloop-agent · 💻 CLIs: Rust / Go
More Agent! packages: AgentAudit · AgentColorSyntax · AgentD1F · AgentEventBridges · AgentLLM · AgentMCP · AgentSwift · AgentTerminalNeo · AgentTools · AgentScripts
Copyright © 2026 AgentiLoop.ai, a Logos InkPen LLC company. All rights reserved.
License
MIT — see LICENSE.