e2e-lab
Bug regression comparison for Teable: give it a list of
teable-ee commits, it runs the same bug cases against every one of them and
answers, in a single table, "is this bug still there on this revision" — so you
can see at a glance which two revisions a fix landed between, and whether
anything that was fixed has come back.
| case | issue | status | `3f2a1c` | `8b4e77` | `a1c9f0` | transition |
| record/t1481-... | T1481 | fixed | ❌ | ❌ | ✅ | fixed 8b4e77..a1c9f0 |
| record/t1502-... | T1502 | open | ⬜ | ⬜ | ⬜ | |
| field/t1520-... | T1520 | fixed | ✅ | ✅ | ❌ | regressed 8b4e77..a1c9f0 |
How it works
The execution skeleton is teable-perf-lab's, proven in production there:
- Injection: this repository's
cases/, framework/, registry.ts, and
spec are copied into a teableio/teable-ee checkout and run through
teable-ee's own e2e harness (vitest, initApp(), seeded auth). Cases live
here and evolve independently; historical commits of the product need no
knowledge of them.
- Pinning: refs are resolved to SHAs once, up front; every job checks out
the pinned SHA.
- One job per commit: isolated database built from that commit's own
migrations, every selected case run once on v2, one JSON payload per case
written before any assertion throws — the payloads are the source of truth,
and failures carry the server's own error body.
- Fail-closed report: every planned (case × commit) cell must have exactly
one payload. Missing evidence fails the run; it never renders as an empty
cell someone might read as green.
One guarded engine. The active regression matrix runs v2 only, because that
is where fixes land. v1 is not currently executed by this workflow.
What is this repository's own: the verdict model. Each case declares the bug
it reproduces and its believed status (open / fixed); the run observes
(absent / present / error) and the comparison judges. Known-unfixed bugs
failing is expected and green; a fixed bug reproducing on the gating column
(the newest commit) is a regression and red; the same observation on an older
column is just the world before the fix. A case that errors is red everywhere,
because it observed nothing. Details: framework/verdict.ts
and docs/operations/e2e-lab.md.
Running
gh workflow run e2e-lab.yml \
--repo teableio/teable-e2e-lab \
--ref main \
-f teable_ee_commits="<sha-before-fix>,<sha-after-fix>,develop" \
-f case_filter=all
Commits go oldest-first; the last one is the gating column. The table lands in
the report job's GitHub summary; comparison.json in the
e2e-lab-comparison-* artifact. Local direction-finding runs follow the localrun skill, which is not published
with this repository; ask the team for it.
Browser dependency archives
Before runtime bootstrap, the workflow selects the Kernel.org Ubuntu archive
with canonical archive/security fallbacks in the runner's APT mirror list.
Kernel.org is a separately operated mirror, not a
Canonical host; its independent hostname avoids relying only on ubuntu.com
when that domain's resolution fails. The existing Ubuntu signing keyring,
repository suites/components and APT/TLS verification remain unchanged.
This bootstrap belongs to the Lab workflow, not to a helper introduced by a
new EE revision: historical comparison columns must remain runnable. Browser
cases still install Chromium with --with-deps and execute the original case
plan; dependency errors are never treated as a successful run.
Adding a case
Read .agents/README.md — the short version: one
declarative cases/<group>/<name>.case.ts (data only, typed against its
runner), one same-name .md documenting the bug and the checkpoint, one
registry entry. Execution logic lives in framework/runners/. pnpm check
validates everything that can be validated without a teable-ee checkout,
including a full type check against the runner/config contracts.
Security-sensitive bugs are not accepted until their fix has shipped — see
CONTRIBUTING.md.
Repository layout
cases/ declarative bug cases + per-case docs
framework/ types, verdict model, checkpoint seam, runners, artifacts
registry.ts explicit case registration
e2e-lab.e2e-spec.ts the single injected entry point
scripts/ planner, comparison table, acceptance gate, checks
.github/ e2e-lab.yml (the run), check.yml (secrets-free PR checks)
docs/ operations guide, cross-repo dispatch status