Python environment
To get started hacking, install Python 3.10+ and uv. Then run:
Run the unit tests with uv run --package master pytest.
Master configuration
The master is deployed via Docker Compose, which manages three services:
a PostgreSQL database, the Buildbot master, and a Caddy reverse proxy with
automatic HTTPS.
halide-llvm (and other) wheels used to be hosted here via a bundled
pypiserver instance at pypi.halide-lang.org, fronted by this same Caddy.
That's gone: the MIT-hosted VM this appliance runs on turned out to be subject
to intermittent border-security IP quarantines outside anyone's control
(shared cloud egress IPs occasionally get flagged for unrelated scanning
activity from other tenants, with no exception mechanism available). Package
hosting has moved to halide/pypi (GitHub Releases + Pages), which routes CI
traffic off that network path entirely. pypi.halide-lang.org now points
there via DNS; this repo no longer serves it.
Secrets
Five secrets control authentication with external users and servers. These
will need to be determined before starting up a new master.
- Obtain a GitHub personal access token
with at least the
repo scope enabled (other scopes that are not currently
used but might be later are write:packages and delete:packages). Call
this GITHUB_TOKEN.
- Generate a secret for the workers to authenticate with the master. Call this
WORKER_SECRET.
- Generate a secret to authenticate GitHub's webhook updates with the master.
Call this
WEBHOOK_SECRET.
- Choose a password for the
halidenightly user to authenticate with the web
interface. Call this WWW_PASSWORD.
- Generate a password for the PostgreSQL database. Call this
DB_PASSWORD.
This is only needed when using PostgreSQL (i.e., when
HALIDE_BB_MASTER_DB_URL contains {DB_PASSWORD}). The default SQLite
backend does not require it.
A convenient command for generating a secure secret is openssl rand -hex 20.
Write all the secrets to the corresponding files:
$ echo "$GITHUB_TOKEN" > secrets/github_token.txt
$ echo "$WORKER_SECRET" > secrets/halide_bb_pass.txt
$ echo "$WEBHOOK_SECRET" > secrets/webhook_token.txt
$ echo "$WWW_PASSWORD" > secrets/buildbot_www_pass.txt
$ echo "$DB_PASSWORD" > secrets/db_password.txt
GitHub configuration
Make your way to the Webhooks section of your repository settings. The url is
https://github.com/{owner}/{repo}/settings/hooks. The following settings are
the correct ones:
- Payload URL:
https://buildbot.halide-lang.org/master/change_hook/github
- Content type:
application/json
- Secret:
$WEBHOOK_SECRET
- SSL verification: Select Enable SSL verification
- Which events would you like to trigger this webhook?
a. Let me select individual events. Check "Pull requests"
and "Pushes".
Starting the master
$ docker compose up -d --build
The database is automatically initialized on first start. Caddy handles TLS
certificates, so there is no manual web server configuration needed. The
Buildbot web interface is available at https://buildbot.halide-lang.org/master/.
Port 9990 must be reachable by workers. Port 8012 is internal only; Caddy
proxies it on ports 80/443.
Running without Docker
By default, the master uses a SQLite database (sqlite:///state.sqlite), so
no external database is required. To use PostgreSQL instead, set
HALIDE_BB_MASTER_DB_URL:
$ export HALIDE_BB_MASTER_DB_URL="postgresql://buildbot:{DB_PASSWORD}@db/buildbot"
Then start the master:
The script automatically runs upgrade-master before starting. If the master
is already running, invoking ./master.sh with no arguments will reconfig it
instead.
Worker configuration
The master recognizes workers by their reported names, e.g. linux-worker-4
or win-worker-1.
Linux
Write the worker secret and set the worker name, then use the wrapper script:
$ echo "$WORKER_SECRET" > worker/halide_bb_pass.txt
$ export HALIDE_BB_WORKER_NAME=$WORKER_NAME
$ ./worker.sh
Windows
> Set-Content worker/halide_bb_pass.txt "$WORKER_SECRET"
> $env:HALIDE_BB_WORKER_NAME = "$WORKER_NAME"
> .\worker.ps1
Optional environment variables
| Variable |
Default |
Description |
HALIDE_BB_MASTER_ADDR |
buildbot.halide-lang.org |
Master hostname |
HALIDE_BB_MASTER_PORT |
9990 |
Master PB port |
Platform-specific installation
Automated installation scripts that set up system dependencies and configure
the worker to start automatically are provided under worker/:
- Windows:
worker/windows/install.ps1 — installs dependencies via winget,
sets up Visual Studio 2022, installs uv, and bootstraps vcpkg with the
required libraries.