Framework comparison

The same 8-test suite on one build and one device in warden, Argent, agent-device and tester.army. Suite time, per-test medians, cold start, resources and footprint.

Every framework runs the same 8 parity tests against the same .app, with no LLM involved. Every run happens on EAS Workflows, one macos-large worker and one fresh simulator per framework (see iOS run 2026-10-25 on EAS for how, and for every artifact). See Benchmark e2e frameworks to run it yourself.

iOS run of 2026-10-25 · full report

Fastest, all tests passing

91.77 s

agent-device, warm suite median

Ratio spread

1× to 1.44×

agent-device to tester.army

Passed every test

4 of 4

every framework that ran passed every test

Suite

8 tests

5 runs each, run 1 cold

warden
Argent
agent-device
tester.army

Suite time

Warm suite wall time, measured by the harness around the whole suite process. Warm numbers leave out the first, cold run.

Bar: warm median. Whisker: p95. Dot: cold first run. The ratio is the median over the fastest framework that passed every test. A ✗ marks a framework with failing tests, whose time is not comparable.

Per test

Warm median per canonical test. Compare these before the step numbers.

Warm median per test and framework
Test	warden	Argent	agent-device	tester.army
onboarding steps through to sign-in	3.14 s1.00×	8.22 s2.62×	3.87 s1.23×	6.03 s1.92×
invalid email and short password show field errors	17.99 s1.09×	17.96 s1.09×	16.55 s1.00×	20.87 s1.26×
a locked account shows the form alert	11.05 s1.14×	14.04 s1.45×	9.67 s1.00×	12.80 s1.32×
valid credentials land on todos	10.94 s1.12×	14.98 s1.53×	9.77 s1.00×	14.89 s1.52×
add, complete and filter todos	17.00 s1.10×	20.87 s1.35×	15.50 s1.00×	21.20 s1.37×
open a todo, mark it done, then delete it	12.02 s1.00×	18.88 s1.57×	12.49 s1.04×	18.17 s1.51×
deep link opens a todo by id	11.79 s1.13×	17.37 s1.67×	10.42 s1.00×	18.06 s1.73×
settings shows the account, toggles switches and signs out	13.02 s1.00×	19.12 s1.47×	13.23 s1.02×	18.07 s1.39×

Bold: fastest in the row. ✗: the test failed in at least one run, so the time is not comparable.

Step latency

Whisker: min to max. Box: min to p95. Tick: median. Step timings are not defined the same way in every tool, so compare per-test totals first.

Cold start, prepare, teardown

Cold-start cost is the first run minus the warm median, plus the time spent in prepare: what installing a runner app or starting a daemon costs, not steady-state speed. Overhead is suite wall time minus the sum of test durations.

A negative cold-start cost is not a duration. It means the cold first run finished faster than the warm median (run-to-run noise on a framework with little startup cost), so the table shows "none (rep 1 not slower, ...)" and leaves it out of the fastest-cell highlight.

Suite time, cold start, prepare, teardown and reliability
Metric	warden	Argent	agent-device	tester.army
Warm suite median	97.04 s	131.81 s	91.77 s	132.34 s
Warm suite p95	98.63 s	132.14 s	91.93 s	133.43 s
Cold first run	98.33 s	132.43 s	92.86 s	151.03 s
Cold-start cost	1.94 s	2.35 s	18.41 s	18.69 s
Prepare	652 ms	1.73 s	17.32 s	0 ms
Teardown	85 ms	511 ms	322 ms	523 ms
Overhead outside tests	80 ms	168 ms	223 ms	2.21 s
Run-to-run spread (cv)	1.3%	0.3%	0.2%	0.5%
Pass rate	100%	100%	100%	100%
Flake rate	0%	0%	0%	0%

Resources

CPU is judged in CPU-seconds per rep: the CPU time the framework's own processes used during one timed run (warm median), from per-process deltas of ps time. The sampled %cpu is shown last as a secondary number. Memory is the framework's own processes, sampled every 500 ms.

CPU time the framework's own processes used during one timed run (warm median), from per-process deltas of ps time. Host means the framework's own processes.

Sampled every 500 ms while the suite runs. Host means the framework's own processes, summed per process, so shared pages can be counted twice.

Host CPU-seconds, memory and %cpu
Metric	warden	Argent	agent-device	tester.army
Host CPU-s / rep	9.1 s	2.8 s	10.3 s	17.0 s
Host memory avg	147.8 MB	182.0 MB	745.9 MB	599.5 MB
Host memory peak	184.9 MB	261.8 MB	2.4 GB	2.0 GB
Host %cpu avg	8.1%	1.7%	12.9%	13.9%
Host %cpu peak	59%	40%	533%	508%
Simulator %cpu avg	102.7%	131.3%	125.8%	118.9%
Simulator %cpu peak	293%	788%	380%	560%

Host CPU-s / rep: warm median of the timed run. %cpu is a decaying average, inflated by a short suite; compare CPU-seconds.

Where the CPU goes

The same CPU-seconds split by process name, from the warm-median rep. A name keeps its colour in every bar, the six biggest names are shown and the rest are folded into other. The sampler ticks every 500 ms, so a process that lives between two ticks only shows up through the exit charge booked when its parent reaps it.

Host processes (the framework's own tree)

xcodebuild
warden-simd
simulator-server
bun cli.ts
snapshot-bridge
bun test
other

warden9.1 s
Argent2.7 s
agent-device10.2 s
tester.army17.0 s

Table

warden
Process	CPU-s	Share	RSS avg
warden-simd	7.0 s	77.2%	55.9 MB
bun test	1.1 s	12.6%	92.2 MB
bun test (exit charge)	0.7 s	7.5%	0 B
simctl	0.2 s	2.7%	277.4 KB
(PlistBuddy)	0.0 s	0.0%	0 B
(bash)	0.0 s	0.0%	0 B
(simctl)	0.0 s	0.0%	0 B
bun cli.ts	0.0 s	0.0%	0 B

Argent
Process	CPU-s	Share	RSS avg
simulator-server	2.5 s	92.0%	96.6 MB
node argent	0.2 s	7.8%	85.7 MB
node argent (exit charge)	0.0 s	0.2%	0 B
(node)	0.0 s	0.0%	0 B
node tool-server.cjs	0.0 s	0.0%	0 B
simctl	0.0 s	0.0%	0 B
system_profiler	0.0 s	0.0%	0 B

agent-device
Process	CPU-s	Share	RSS avg
xcodebuild	7.5 s	73.3%	218.0 MB
bun cli.ts	1.8 s	17.3%	268.7 MB
snapshot-bridge	0.8 s	7.5%	116.3 MB
node agent-device	0.1 s	1.5%	61.0 MB
ps	0.0 s	0.2%	1.9 MB
simctl	0.0 s	0.2%	20.4 MB
node agent-device (exit charge)	0.0 s	0.1%	0 B
(simctl)	0.0 s	0.0%	96.9 KB
DTServiceHub	0.0 s	0.0%	16.7 MB
caffeinate	0.0 s	0.0%	7.3 MB
(bash)	0.0 s	0.0%	0 B
(clang)	0.0 s	0.0%	0 B
other	0.0 s	0.0%	0 B

tester.army
Process	CPU-s	Share	RSS avg
xcodebuild	14.3 s	84.0%	217.0 MB
node entry.js	1.1 s	6.2%	84.9 MB
snapshot-bridge	0.6 s	3.7%	116.2 MB
(node) (exit charge)	0.6 s	3.7%	0 B
node e2e	0.3 s	2.1%	109.4 MB
(node)	0.0 s	0.2%	0 B
other (exit charge)	0.0 s	0.2%	0 B
(ps)	0.0 s	0.1%	27.9 KB
simctl	0.0 s	0.0%	20.5 MB
DTServiceHub	0.0 s	0.0%	16.8 MB
(pgrep)	0.0 s	0.0%	0 B
(simctl)	0.0 s	0.0%	0 B
other	0.0 s	0.0%	0 B

Simulator processes

iOS
wardenexample
AgentDeviceRunnerUITests-Runner
bun cli.ts
SimulatorFrameworkBridge-iOS serve
CoreSimulatorBridge
other

warden116.3 s
Argent133.5 s
agent-device136.0 s
tester.army178.1 s

Table

warden
Process	CPU-s	Share	RSS avg
iOS	75.3 s	64.7%	34.8 GB
wardenexample	30.6 s	26.3%	374.9 MB
SimulatorFrameworkBridge-iOS serve	6.0 s	5.2%	197.7 MB
bun cli.ts	2.0 s	1.7%	235.0 MB
CoreSimulatorBridge	0.9 s	0.8%	83.7 MB
launchd_sim	0.4 s	0.3%	52.2 MB
xpcproxy_sim	0.3 s	0.2%	390.5 KB
mlhostd	0.2 s	0.2%	129.7 KB
ps	0.1 s	0.1%	1.9 MB
(launchctl)	0.1 s	0.1%	93.2 KB
storekitd	0.1 s	0.1%	127.9 MB
triald	0.1 s	0.1%	96.6 MB
other	0.0 s	0.0%	443.9 MB

Argent
Process	CPU-s	Share	RSS avg
iOS	98.8 s	74.0%	34.8 GB
wardenexample	29.5 s	22.1%	375.5 MB
bun cli.ts	2.8 s	2.1%	298.6 MB
simctl	0.4 s	0.3%	363.8 KB
CoreSimulatorBridge	0.4 s	0.3%	117.8 MB
launchd_sim	0.4 s	0.3%	52.2 MB
ps	0.3 s	0.2%	1.9 MB
(mlhostd)	0.3 s	0.2%	222.5 KB
triald	0.2 s	0.1%	96.7 MB
storekitd	0.2 s	0.1%	128.4 MB
(managedappdistri)	0.1 s	0.1%	214.5 KB
mlhostd	0.1 s	0.1%	119.3 KB
other	0.1 s	0.0%	444.5 MB

agent-device
Process	CPU-s	Share	RSS avg
iOS	89.3 s	65.7%	35.5 GB
wardenexample	35.8 s	26.3%	371.9 MB
AgentDeviceRunnerUITests-Runner	9.1 s	6.7%	268.2 MB
CoreSimulatorBridge	0.5 s	0.4%	80.6 MB
simctl	0.5 s	0.4%	676.7 KB
launchd_sim	0.3 s	0.2%	52.4 MB
triald	0.1 s	0.1%	96.7 MB
storekitd	0.1 s	0.1%	128.3 MB
xpcproxy_sim	0.1 s	0.1%	176.0 KB
mlhostd	0.1 s	0.1%	34.2 KB
(wardenexample)	0.0 s	0.0%	2.4 MB
assetsd	0.0 s	0.0%	163.2 MB
other	0.0 s	0.0%	274.3 MB

tester.army
Process	CPU-s	Share	RSS avg
iOS	107.5 s	60.4%	35.1 GB
wardenexample	49.9 s	28.0%	380.4 MB
AgentDeviceRunnerUITests-Runner	15.9 s	8.9%	271.0 MB
bun cli.ts	2.6 s	1.4%	254.0 MB
CoreSimulatorBridge	0.6 s	0.3%	80.6 MB
simctl	0.5 s	0.3%	534.1 KB
launchd_sim	0.5 s	0.3%	51.8 MB
mlhostd	0.2 s	0.1%	135.3 KB
triald	0.2 s	0.1%	96.6 MB
storekitd	0.1 s	0.1%	128.2 MB
launchctl	0.1 s	0.1%	36.7 KB
ps	0.1 s	0.0%	1.9 MB
other	0.1 s	0.0%	446.4 MB

Residue and footprint

Residue is what is left on the machine after teardown. Footprint is how much suite source you maintain and how large the runner binaries are.

Footprint and residue
Metric	warden	Argent	agent-device	tester.army
Suite source (lines)	379	161	158	307
Suite source (size)	17.5 KB	7.1 KB	5.4 KB	13.8 KB
Runner binaries on disk	6.1 MB	77.8 MB	6.9 MB	—
Leftover processes	0	0	0	0
Leftover driver apps	0	0	1	1
Leftover directories	0	1	1	1

Versions and run details

Framework versions and status
Framework	Version	Status
warden	bun 1.3.14 + warden-e2e	ran, all tests passed
Argent	0.30.1	ran, all tests passed
agent-device	agent-device 0.21.31	ran, all tests passed
tester.army	e2e 0.20.0	ran, all tests passed

Host: VirtualMac2,1, Apple M4 Pro (Virtual), 10 cores, 40.0 GB, macOS 26.5.2
Device: warden-iphone-17-1 iOS 26.5
App: nz.co.delacour.warden.example, 96.4 MB, build import:2d356, sha256 2d35643eda49
Tools: bun 1.3.14, warden 0.1.1
Run: 5 runs per framework, 0 retries, load average 3.85 at start
Report: docs/bench/ios-2026-10-25.md

Fairness rules

Same .app for every framework, with its sha256 recorded.
Zero LLM calls.
Same assertions per canonical test id.
Same retry helpers: dropped first tap, dropped deep link, system alerts.
The iOS "Open in …?" sheet is answered once per boot by the harness before every framework's prepare (iOS asks it for the first custom-scheme link after a boot, not again).
Fresh app launch per test, retries=0.
Frameworks run sequentially with daemons stopped between them, or, on EAS, each on its own worker and simulator.
caffeinate -dims held for the run, host otherwise idle, load average recorded.
Warm medians exclude rep 1.
Machine-readable reporters only.
A framework with any failing test never sets the pace, so it is never the fastest.

Caveats per framework

Framework	Gap
warden	Copied from the example app, retries written with warden's own options: `tap({ until })`, a deep-link retry, `systemAlerts: "dismiss"`. The hand-rolled helpers it replaced ran the published headline numbers; the idiomatic run is reported beside them.
Argent	Retries are nested `when:` guards capped at two retaps. Flow selectors read only the app's view hierarchy, never SpringBoard, so the flow cannot answer the "Open in …?" sheet itself. A checkbox's checked state is not readable, so the remaining-count assertion covers the effect.
agent-device	`.ad` scripts have no retry or conditional, only `wait` and `press`, so the deep-link script cannot answer a sheet that may or may not show. Needs a runner prepare step and a daemon.
tester.army	Has its own copy of the hand-rolled helpers warden used before. Runs with a private agent-device state dir and Node 24 on `PATH`.

Each run is a snapshot on one machine class with few warm repetitions: read the ratios as a guide, not a verdict. The full Markdown report for each run is in docs/bench/ in the repo; the EAS runs also publish every worker's artifact on their run page.

Framework comparison

Host processes (the framework's own tree)

Simulator processes

On this page