Same .app for every framework (sha256 recorded above)
Zero LLM calls
Same assertions per canonical test id
Same retry helpers (dropped first tap, dropped deep link, system alerts)
Fresh app launch per test
retries=0
Frameworks run sequentially
caffeinate -dims held for the run
Warm medians exclude rep 1 (cold)
Machine-readable reporters only
Host otherwise idle; load average recorded
Android sim counts the app and every framework's on-device helpers (uiautomator, instrumentation runners, agent-device, Argent, warden-uiad), matched case-insensitively
Host processes are tracked by pid and start time, so a reused pid is never counted
Android animation scales are reset to the device's original values before every framework and restored at the end
Summary
Framework
Status
Version
Suite median (warm)
p95
Ratio
Cold rep 1
Cold-start cost
Prepare
Teardown
Overhead
Pass rate
Flake
Residue
Footprint (LOC / deps)
warden
✓ ok
bun 1.3.14 + warden-e2e
97.04 s
98.63 s
1.06×
98.33 s
1.94 s
652 ms
85 ms
80 ms
100%
0%
0 proc / 0 app / 0 path
379 LOC / 0 B
argent
✓ ok
0.30.1
131.81 s
132.14 s
1.44×
132.43 s
2.35 s
1.73 s
511 ms
168 ms
100%
0%
0 proc / 0 app / 1 path
161 LOC / 0 B
agent-device
✓ ok
agent-device 0.21.31
91.77 s
91.93 s
1.00×
92.86 s
18.41 s
17.32 s
322 ms
223 ms
100%
0%
0 proc / 1 app / 1 path
158 LOC / 0 B
tester-army
✓ ok
e2e 0.20.0
132.34 s
133.43 s
1.44×
151.03 s
18.69 s
0 ms
523 ms
2.21 s
100%
0%
0 proc / 1 app / 1 path
307 LOC / 0 B
Warm suite wall time
warden
argent
agent-device
tester-army
Bar = warm median. Whisker = p95. Dot = cold rep 1. Lower is faster; ratios are against the fastest framework.
Per-test medians (ms)
Test
warden
argent
agent-device
tester-army
onboarding steps through to sign-in
3136 (1.00×)
8220 (2.62×)
3867 (1.23×)
6032 (1.92×)
invalid email and short password show field errors
17987 (1.09×)
17962 (1.09×)
16549 (1.00×)
20867 (1.26×)
a locked account shows the form alert
11046 (1.14×)
14040 (1.45×)
9671 (1.00×)
12799 (1.32×)
valid credentials land on todos
10937 (1.12×)
14982 (1.53×)
9771 (1.00×)
14889 (1.52×)
add, complete and filter todos
16997 (1.10×)
20867 (1.35×)
15503 (1.00×)
21196 (1.37×)
open a todo, mark it done, then delete it
12015 (1.00×)
18881 (1.57×)
12491 (1.04×)
18171 (1.51×)
deep link opens a todo by id
11791 (1.13×)
17367 (1.67×)
10417 (1.00×)
18056 (1.73×)
settings shows the account, toggles switches and signs out
13018 (1.00×)
19116 (1.47×)
13228 (1.02×)
18070 (1.39×)
Median per test
warden
argent
agent-device
tester-army
Spread across warm reps
Whisker = min to max. Shaded box = min to p95. Tick = median. A distribution carries no quartiles, so the box is not an interquartile range.
Per-step latency
warden
argent
agent-device
tester-army
Whisker = min to max. Shaded box = min to p95. Tick = median. Only frameworks that report steps appear.
Framework
Steps
Median
p95
Max
warden
504
170 ms
3.93 s
5.00 s
argent
1192
318 ms
1.82 s
2.82 s
agent-device
632
479 ms
1.62 s
1.90 s
tester-army
592
494 ms
3.18 s
3.98 s
Resources
warden
argent
agent-device
tester-army
CPU
Memory
Framework
Scope
CPU-s / rep
CPU avg
CPU peak
RSS avg
RSS peak
warden
host
9.12 s
8.1%
59.4%
147.8 MB
184.9 MB
warden
sim
—
102.7%
292.5%
36.4 GB
36.5 GB
argent
host
2.79 s
1.7%
39.7%
182.0 MB
261.8 MB
argent
sim
—
131.3%
788.2%
36.3 GB
36.5 GB
agent-device
host
10.30 s
12.9%
532.8%
745.9 MB
2.4 GB
agent-device
sim
—
125.8%
380.0%
36.8 GB
37.0 GB
tester-army
host
17.01 s
13.9%
508.1%
599.5 MB
2.0 GB
tester-army
sim
—
118.9%
560.4%
36.8 GB
37.0 GB
CPU-s / rep is the host CPU time of the timed run (warm median) and the headline. CPU avg and peak are ps %cpu, a decaying average on macOS: they move with suite length, do not judge on them.
Where the CPU goes
One table per framework, from the warm-median rep's timed run (the rep whose host CPU-seconds is the lower median). Rows are processes grouped by label (top 12 per scope, the rest in other); host is the framework's own process tree, sim the simulator's launchd_sim tree (iOS only). Share is of the scope's CPU-seconds. RSS is the label's summed RSS per 500 ms tick (0 where it had no process). Spawns counts distinct pids a tick saw: a process that lives between two ticks is only seen through the (exit charge) rows (CPU of a child the bench ran, booked at its exit), so many short-lived xcrun simctl calls read low in Spawns and in CPU.