e2e framework benchmark — ios 2026-10-25
Run eas-ios-warden, started 2026-10-25T12:35:52.050Z
Fairness rules
- Same
.appfor every framework (sha256 recorded above) - Zero LLM calls
- Same assertions per canonical test id
- Same retry helpers (dropped first tap, dropped deep link, system alerts)
- Fresh app launch per test
retries=0- Frameworks run sequentially
caffeinate -dimsheld for the run- Warm medians exclude rep 1 (cold)
- Machine-readable reporters only
- Host otherwise idle; load average recorded
- Android
simcounts the app and every framework's on-device helpers (uiautomator, instrumentation runners, agent-device, Argent, warden-uiad), matched case-insensitively - Host processes are tracked by pid and start time, so a reused pid is never counted
- Android animation scales are reset to the device's original values before every framework and restored at the end
Summary
| Framework | Status | Version | Suite median (warm) | p95 | Ratio | Cold rep 1 | Cold-start cost | Prepare | Teardown | Overhead | Pass rate | Flake | Residue | Footprint (LOC / deps) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| warden | ✓ ok | bun 1.3.14 + warden-e2e | 97.04 s | 98.63 s | 1.00× | 98.33 s | 1.94 s | 652 ms | 85 ms | 80 ms | 100% | 0% | 0 proc / 0 app / 0 path | 379 LOC / 0 B |
Warm suite wall time
- warden
Bar = warm median. Whisker = p95. Dot = cold rep 1. Lower is faster; ratios are against the fastest framework.
Per-test medians (ms)
| Test | warden |
|---|---|
| onboarding steps through to sign-in | 3136 (1.00×) |
| invalid email and short password show field errors | 17987 (1.00×) |
| a locked account shows the form alert | 11046 (1.00×) |
| valid credentials land on todos | 10937 (1.00×) |
| add, complete and filter todos | 16997 (1.00×) |
| open a todo, mark it done, then delete it | 12015 (1.00×) |
| deep link opens a todo by id | 11791 (1.00×) |
| settings shows the account, toggles switches and signs out | 13018 (1.00×) |
Median per test
- warden
Spread across warm reps
Whisker = min to max. Shaded box = min to p95. Tick = median. A distribution carries no quartiles, so the box is not an interquartile range.
Per-step latency
- warden
Whisker = min to max. Shaded box = min to p95. Tick = median. Only frameworks that report steps appear.
| Framework | Steps | Median | p95 | Max |
|---|---|---|---|---|
| warden | 504 | 170 ms | 3.93 s | 5.00 s |
Resources
- warden
CPU
Memory
| Framework | Scope | CPU-s / rep | CPU avg | CPU peak | RSS avg | RSS peak |
|---|---|---|---|---|---|---|
| warden | host | 9.12 s | 8.1% | 59.4% | 147.8 MB | 184.9 MB |
| warden | sim | — | 102.7% | 292.5% | 36.4 GB | 36.5 GB |
CPU-s / rep is the host CPU time of the timed run (warm median) and the headline. CPU avg and peak are ps %cpu, a decaying average on macOS: they move with suite length, do not judge on them.
Where the CPU goes
One table per framework, from the warm-median rep's timed run (the rep whose host CPU-seconds is the lower median). Rows are processes grouped by label (top 12 per scope, the rest in other); host is the framework's own process tree, sim the simulator's launchd_sim tree (iOS only). Share is of the scope's CPU-seconds. RSS is the label's summed RSS per 500 ms tick (0 where it had no process). Spawns counts distinct pids a tick saw: a process that lives between two ticks is only seen through the (exit charge) rows (CPU of a child the bench ran, booked at its exit), so many short-lived xcrun simctl calls read low in Spawns and in CPU.
warden
| Process | Scope | CPU-s | Share | RSS avg | RSS peak | Spawns |
|---|---|---|---|---|---|---|
| warden-simd | host | 7.03 s | 77.2% | 55.9 MB | 56.0 MB | 1 |
| bun test | host | 1.15 s | 12.6% | 92.2 MB | 107.1 MB | 1 |
| bun test (exit charge) | host | 0.69 s | 7.5% | 0 B | 0 B | 0 |
| simctl | host | 0.24 s | 2.7% | 277.4 KB | 20.7 MB | 3 |
| (PlistBuddy) | host | 0.00 s | 0.0% | 0 B | 0 B | 0 |
| (bash) | host | 0.00 s | 0.0% | 0 B | 0 B | 0 |
| (simctl) | host | 0.00 s | 0.0% | 0 B | 0 B | 0 |
| bun cli.ts | host | 0.00 s | 0.0% | 0 B | 0 B | 0 |
| iOS | sim | 75.30 s | 64.7% | 34.8 GB | 34.8 GB | 286 |
| wardenexample | sim | 30.57 s | 26.3% | 374.9 MB | 432.8 MB | 8 |
| SimulatorFrameworkBridge-iOS serve | sim | 6.04 s | 5.2% | 197.7 MB | 210.2 MB | 12 |
| bun cli.ts | sim | 2.01 s | 1.7% | 235.0 MB | 241.7 MB | 1 |
| CoreSimulatorBridge | sim | 0.94 s | 0.8% | 83.7 MB | 83.9 MB | 1 |
| launchd_sim | sim | 0.39 s | 0.3% | 52.2 MB | 52.2 MB | 1 |
| xpcproxy_sim | sim | 0.26 s | 0.2% | 390.5 KB | 11.4 MB | 8 |
| mlhostd | sim | 0.24 s | 0.2% | 129.7 KB | 14.4 MB | 2 |
| ps | sim | 0.15 s | 0.1% | 1.9 MB | 1.9 MB | 191 |
| (launchctl) | sim | 0.14 s | 0.1% | 93.2 KB | 17.4 MB | 1 |
| storekitd | sim | 0.14 s | 0.1% | 127.9 MB | 127.9 MB | 1 |
| triald | sim | 0.13 s | 0.1% | 96.6 MB | 96.6 MB | 1 |
| other | sim | 0.00 s | 0.0% | 443.9 MB | 443.9 MB | 6 |
Residue
- warden: none
Unsupported / failed
None.