Benchmark e2e frameworks

packages/e2e-bench answers one question: how long does the same suite take in each framework, on the same build and the same device. It runs warden, Argent, agent-device and tester.army one at a time and writes a Markdown and an HTML report. No LLM is involved.

Build the app once

The benchmark installs one pinned app on every framework's run. It is built once and stored at .bench/app/<platform>/ (the .app or .apk plus a manifest.json with the warden fingerprint, sha256, source and commit). Every bench run reuses it. bench run never builds, fingerprints or calls EAS: it installs the pinned artifact (simctl install / adb install -r) after claiming the device, and builds it once only when it is missing.

bun run bench app build --platform ios            # build (warden cache, then a local Expo build) and pin it
bun run bench app build --platform ios --force    # rebuild even though one is pinned
bun run bench app build --platform android --eas  # also look the build up on EAS
bun run bench app status --platform ios           # hash, source, builtAt; fresh or stale, exit 1 when missing or stale
bun run bench app clean --platform ios            # remove the pinned app
bun run bench run --platform ios --app path/to/Example.app   # use a pinned dir or import a bare .app / .apk

app build resolves the build with warden app ensure --variant bench on apps/example. Without --force it reuses the pinned app without fingerprinting. app status compares the pinned hash with warden app fingerprint --variant bench.

Rebuild with --force after changing apps/example or its Expo and React Native dependencies. Changes to packages/e2e or warden do not need a rebuild.

Build on EAS

The app is the Expo project @delacour/warden (extra.eas.projectId in app.config.ts, GitHub base directory apps/example), so bench app build --eas also looks the bench profile up on EAS (bunx eas-cli, logged in to the delacour account). The bench variant in apps/example/warden.config.ts is the only place EAS is configured, and eas.json holds the Release simulator and APK profile. .eas/workflows/bench-build.yml builds both platforms on EAS: run it with eas workflow:run .eas/workflows/bench-build.yml, or it runs on pushes to main that touch apps/example/.

Run the baseline on EAS

.eas/workflows/bench-baseline.yml builds the bench iOS app once, then runs one job per framework, each on its own EAS macos-large worker with its own simulator, so a measurement never touches your Mac and nothing one framework leaves behind reaches the next. Dispatch it from apps/example:

bunx eas-cli workflow:run .eas/workflows/bench-baseline.yml -F runs=5                                  # all four frameworks
bunx eas-cli workflow:run .eas/workflows/bench-baseline.yml -F runs=5 -F frameworks=argent,agent-device

Inputs: runs (reps, default 5; rep 1 is cold), frameworks (comma list of warden,argent,agent-device,tester-army, default all four; a framework's job is skipped unless named) and max_minutes (default 55). Each framework job runs packages/e2e-bench/scripts/eas-bench.sh: install deps, build warden-simd, install argent or agent-device with bun add -g when that is its framework (tester-army comes with bun install), claim a headless simulator, record a 60 s idle control and run bench run against the downloaded .app. A watchdog stops the bench when the job reaches max_minutes; the results so far are kept. On the way out the job screenshots the simulator (sim-final.png), lists the apps it still runs and releases the lease, even when the run fails.

The workflow fingerprints the native project and reuses an existing bench build with the same fingerprint, so it only builds when apps/example changed. Each job uploads an eas-ios-<framework> artifact to the run page (expo.dev/accounts/delacour/projects/warden/workflows); bunx eas-cli workflow:view <run-id> prints the download URLs. Each archive unpacks to Users/expo/bench-results. Copy one to .bench/eas-ios-<framework> and re-render:

bun run bench report .bench/eas-ios-warden

A hosted macOS worker idles hotter than a Mac and shares its host, so EAS numbers compare only with EAS numbers from the same worker class (macos-large). Never compare them with local runs.

Run it

From the repo root:

bun run bench doctor                         # is every framework installed? prints install hints
bun run bench run --platform ios --runs 3    # full run on a device warden claims
bun run bench run --frameworks warden,argent --runs 2 --udid <leased-udid>
bun run bench list                           # frameworks, platforms, the 8 canonical test ids
bun run bench report .bench/<run-id>         # re-render the reports
bun run bench compare .bench/<run-a> .bench/<run-b>
bun run bench summary .bench/<run-id>        # slim summary.json for the docs page

A full iOS run takes about an hour. Keep the Mac awake and use a dedicated simulator. Ctrl-C tears down, releases the lease and still writes results.

Prerequisites

The harness never installs global tools. bench doctor prints the exact command for anything missing.

Read the report

A run writes .bench/<run-id>/:

bench-results.json   machine-readable, schema v1
summary.json         slim summary the docs comparison page reads
report.md            the tables below
report.html          one self-contained file with charts
logs/                one log per framework and phase
artifacts/<fw>/rep-<n>/   native reporter output and resource samples

• Summary table. One row per framework: warm suite median and p95, ratio to the fastest, cold rep 1, cold-start cost, prepare and teardown time, overhead, pass and flake rate, residue and footprint. Warm numbers exclude rep 1.
• Per-test matrix. Median milliseconds per canonical test and framework, with the ratio to the fastest in bold. Compare per-test totals first; step timings are not equally defined in every tool.
• Cold start. Rep 1 minus the warm median, plus the time spent in prepare. This is the cost of installing a runner app or starting a daemon, not steady-state speed.
• Overhead. Suite wall time minus the sum of test durations: what the framework spends outside the tests (process start, device attach, reporting).
• Residue. Processes, driver apps and directories left behind after teardown. A healthy run leaves none unless --keep-drivers is set.
• Footprint. Lines and bytes of suite source, size of package-local dependencies and binaries. A rough measure of what you maintain.
• Resources. Host and simulator CPU and RSS, sampled every 500 ms. The CPU headline is host CPU-seconds per warm rep (timed run only); %cpu is a decaying average on macOS, so do not judge a framework on it.
• Unsupported and failed. A framework that cannot drive the app lands here with its error. A native test name missing from the manifest is a parity breach and fails the framework.

Fairness rules

• Same pinned app for every framework, with its sha256 recorded.
• Zero LLM calls.
• Same assertions per canonical test id.
• Same retry helpers: dropped first tap, dropped deep link, system alerts.
• The iOS "Open in …?" sheet is answered once per boot by the harness, before every framework's prepare: iOS asks it for the first custom-scheme link opened from outside the app and not again, so without that the first framework on a shared simulator would pay for it and no other would see it, and on one simulator per framework a tool with no optional step would lose the deep-link test to it.
• Fresh app launch per test.
• retries=0.
• Frameworks run sequentially, with daemons stopped between them.
• caffeinate -dims held for the run.
• Warm medians exclude rep 1.
• Machine-readable reporters only.
• Host otherwise idle, with the load average recorded.
• Android sim counts the app and every framework's on-device helpers, matched case-insensitively.
• Host processes are tracked by pid and start time, so a reused pid is never counted.
• Android animation scales are reset to the device's original values before every framework and restored at the end.

Caveats per framework

Add a framework

1. Add the id to FRAMEWORK_IDS in adapter.types.ts.
2. Write src/adapters/<id>.ts exporting a BenchAdapter with doctor, prepare, run, teardown and residuePatterns. Spawn only through the injected ctx.exec.
3. Add a parser under src/adapters/parse/ and a real output sample under src/adapters/fixtures/, and test against it with fakeExec.
4. Register it in src/adapters/registry.ts.
5. Add a native.<id> name for every test in suites/manifest.json.
6. Create suites/<id>/ with all 8 tests and the same retries as apps/example/e2e/helpers/steps.ts. Document any gap in the caveats.

Add a canonical test

1. Add the id to CANONICAL_IDS in src/results/schema.ts.
2. Add an entry to suites/manifest.json with a title, the step labels and a native name per framework.
3. Write the test in every suites/<framework>/ with identical assertions and retries.
4. Run bun run --cwd packages/e2e-bench test and bun run bench doctor.

Results

The framework comparison page charts the published runs: warm suite time, per-test medians, step latency, cold start, resources, residue and footprint. The full Markdown reports live in docs/bench/ in the repo.

To publish a run, write its slim summary into the docs site and commit the report next to it:

bun run bench summary .bench/<run> --out apps/docs/content/docs/bench/data/<platform>-<date>.json
cp .bench/<run>/report.md docs/bench/<platform>-<date>.md

The comparison page picks up every *.json in apps/docs/content/docs/bench/data/, with a platform and date selector when there is more than one. bench run and bench report also write summary.json next to report.md.