warden

iOS run 2026-10-25 on EAS

The four-framework iOS run of 2026-10-25, one EAS macos-large worker per framework, 5 runs each, with how it was run, the full tables and every artifact.

The run behind the 2026-10-25 iOS dataset on the comparison page. Everything below comes from the four workers' artifacts, merged into one report; nothing was measured on a developer machine.

How it was run

Workflowapps/example/.eas/workflows/bench-baseline.yml, EAS Workflows run 01a172f5-fdee-7992-b66c-a2173338eb32, inputs runs=5, frameworks=warden,argent,agent-device,tester-army, max_minutes=55
Codewarden commit 5ffd64f2 (branch feature/devboxes), every suite under packages/e2e-bench/suites/
WorkersOne EAS macos-large worker per framework, four in all, two at a time on this plan. VM template macos-tahoe-26.5-xcode-26.6: VirtualMac2,1, Apple M4 Pro (Virtual), 10 cores, 40 GB, macOS 26.5.2 (25F84), Xcode 26.6 (17F113)
SimulatorEach worker claims its own headless simulator with warden claim ios --headless (iPhone 17, iOS 26.5, named warden-iphone-17-1), created fresh for the job and shut down at the end. Nothing one framework leaves behind can reach another
AppThe same build on every worker: nz.co.delacour.warden.example, EAS bench profile (Release, simulator), sha256 2d35643eda49, 101.1 MB, reused by fingerprint (build_ios skipped), installed with simctl install before the bench
Runs5 runs of the 8-test suite per framework. Run 1 is cold; warm numbers are the median of runs 2 to 5. retries=0, fresh app launch per test, caffeinate -dims held
Before the benchbun install, warden-simd built from source (Swift release), the framework installed (bun add -g @swmansion/argent@0.30.1 / agent-device@0.21.31; tester.army's e2e 0.20.0 comes with bun install, Node 22.23.1), then the job waits for the 1-minute load average to drop under the core count and records a 60 s idle control of the simulator with the app in the foreground
Deep-link sheetiOS shows "Open in …?" for the first custom-scheme link after a boot and never again, so the harness opens one link and answers the sheet before every framework's prepare. No suite handles the sheet itself
Hard stopA watchdog stops the bench at max_minutes; it did not fire on any worker (the longest job ran 28 min)
Isolation per frameworkHelpers of every framework stopped, driver apps uninstalled before the run (cold cost measured), residue listed after teardown
SamplingHost and simulator processes sampled every 500 ms from ps; the CPU headline is CPU-seconds of the framework's own process tree during the timed run, warm median
Evidence on exitEach job screenshots the simulator and lists the apps still running, releases the lease even when a step failed, and uploads an eas-ios-<framework> artifact
Versionsbun 1.3.14, warden 0.1.1, Argent 0.30.1, agent-device 0.21.31, tester.army e2e 0.20.0 (bundled agent-device 0.21.25), Node 22.23.1

Hosted workers idle hotter than a Mac and share their host, so these numbers compare only with other macos-large runs, never with runs on other hardware.

Results

Summary

FrameworkStatusVersionSuite median (warm)p95RatioCold rep 1Cold-start costPrepareTeardownOverheadPass rateFlakeResidueFootprint (LOC / deps)
wardenokbun 1.3.14 + warden-e2e97.04 s98.63 s1.06×98.33 s1.94 s652 ms85 ms80 ms100%0%0 proc / 0 app / 0 path379 LOC / 0 B
argentok0.30.1131.81 s132.14 s1.44×132.43 s2.35 s1.73 s511 ms168 ms100%0%0 proc / 0 app / 1 path161 LOC / 0 B
agent-deviceokagent-device 0.21.3191.77 s91.93 s1.00×92.86 s18.41 s17.32 s322 ms223 ms100%0%0 proc / 1 app / 1 path158 LOC / 0 B
tester-armyoke2e 0.20.0132.34 s133.43 s1.44×151.03 s18.69 s0 ms523 ms2.21 s100%0%0 proc / 1 app / 1 path307 LOC / 0 B

Per-test medians (ms)

Testwardenargentagent-devicetester-army
onboarding steps through to sign-in3136 (1.00×)8220 (2.62×)3867 (1.23×)6032 (1.92×)
invalid email and short password show field errors17987 (1.09×)17962 (1.09×)16549 (1.00×)20867 (1.26×)
a locked account shows the form alert11046 (1.14×)14040 (1.45×)9671 (1.00×)12799 (1.32×)
valid credentials land on todos10937 (1.12×)14982 (1.53×)9771 (1.00×)14889 (1.52×)
add, complete and filter todos16997 (1.10×)20867 (1.35×)15503 (1.00×)21196 (1.37×)
open a todo, mark it done, then delete it12015 (1.00×)18881 (1.57×)12491 (1.04×)18171 (1.51×)
deep link opens a todo by id11791 (1.13×)17367 (1.67×)10417 (1.00×)18056 (1.73×)
settings shows the account, toggles switches and signs out13018 (1.00×)19116 (1.47×)13228 (1.02×)18070 (1.39×)

Per-step latency

FrameworkStepsMedianp95Max
warden504170 ms3.93 s5.00 s
argent1192318 ms1.82 s2.82 s
agent-device632479 ms1.62 s1.90 s
tester-army592494 ms3.18 s3.98 s

Untraced time

FrameworkUntraced / repShare of test timeLargest (test)
warden3.35 s3.5%settings/account-switches-sign-out (718 ms)

Test wall time that no step and no span of the framework's own trace covers (warden: trace.jsonl; nested steps and spans count once). Runner overhead, hooks, sleeps and calls nobody wrapped end up here.

Resources

FrameworkScopeCPU-s / repCPU-s (whole)CPU avgCPU peakRSS avgRSS peak
wardenhost9.12 s46.25 s8.1%59.4%147.8 MB184.9 MB
wardensim—563.94 s102.7%292.5%36.4 GB36.5 GB
argenthost2.79 s14.69 s1.7%39.7%182.0 MB261.8 MB
argentsim—956.48 s131.3%788.2%36.3 GB36.5 GB
agent-devicehost10.30 s73.33 s12.9%532.8%745.9 MB2.4 GB
agent-devicesim—683.96 s125.8%380.0%36.8 GB37.0 GB
tester-armyhost17.01 s107.19 s13.9%508.1%599.5 MB2.0 GB
tester-armysim—914.59 s118.9%560.4%36.8 GB37.0 GB

Sampled every 500 ms from ps. host = the framework's own process tree; sim = every process of the simulator's launchd_sim tree, or on Android the app under test plus every framework's on-device helpers from top (every 2 s). RSS is a per-process sum, so shared framework pages are counted once per process: compare the numbers across frameworks, not against physical memory.

CPU-s / rep (the CPU headline) is the CPU time of the timed run only, median over the warm reps (over every rep, cold included, when a framework has no warm rep, as with --runs 1), from per-process deltas of ps time; the other columns are whole-framework (prepare, every rep, the per-rep housekeeping and teardown), so their averages move with suite length. CPU avg / CPU peak are ps %cpu, a decaying average on macOS, kept for comparison with older runs. Android sim has no CPU-seconds (top only). CPU-seconds undercount: CPU a process uses after the last 500 ms tick, and processes living between ticks, are only recovered for children the bench ran itself (their rusage at exit); daemons and processes a framework spawns and never waits for are not, so frameworks that fan out to many short-lived tools read low. An exit charge is booked to its root's group, so in the per-group split the unseen CPU of a descendant of another group (a bun test under a warden root) shows under the root. procps (Linux) ps time has 1 s resolution.

Host by process group

FrameworkGroupCPU-s (whole)RSS avgRSS peak
wardenbun-test10.27 s92.0 MB129.3 MB
wardenwarden-simd35.39 s55.7 MB56.2 MB
wardenwarden-cli0.13 s82.2 KB77.7 MB
wardenother0.47 s21.7 KB20.5 MB
argentother14.69 s182.0 MB261.8 MB
agent-deviceother73.33 s745.9 MB2.4 GB
tester-armyother107.19 s599.5 MB2.0 GB

bun-test the spec process, warden-simd the resident iOS helper, warden-cli the warden commands the bench spawned, adb / emulator the Android tools, other the rest (another framework's runner, xcrun, java). Each group's RSS average counts 0 for ticks without a process, so the groups add up to the host.

Where the CPU goes

warden

ProcessScopeCPU-sShareRSS avgRSS peakSpawns
warden-simdhost7.03 s77.2%55.9 MB56.0 MB1
bun testhost1.15 s12.6%92.2 MB107.1 MB1
bun test (exit charge)host0.69 s7.5%0 B0 B0
simctlhost0.24 s2.7%277.4 KB20.7 MB3
(PlistBuddy)host0.00 s0.0%0 B0 B0
(bash)host0.00 s0.0%0 B0 B0
(simctl)host0.00 s0.0%0 B0 B0
bun cli.tshost0.00 s0.0%0 B0 B0
iOSsim75.30 s64.7%34.8 GB34.8 GB286
wardenexamplesim30.57 s26.3%374.9 MB432.8 MB8
SimulatorFrameworkBridge-iOS servesim6.04 s5.2%197.7 MB210.2 MB12
bun cli.tssim2.01 s1.7%235.0 MB241.7 MB1
CoreSimulatorBridgesim0.94 s0.8%83.7 MB83.9 MB1
launchd_simsim0.39 s0.3%52.2 MB52.2 MB1
xpcproxy_simsim0.26 s0.2%390.5 KB11.4 MB8
mlhostdsim0.24 s0.2%129.7 KB14.4 MB2
pssim0.15 s0.1%1.9 MB1.9 MB191
(launchctl)sim0.14 s0.1%93.2 KB17.4 MB1
storekitdsim0.14 s0.1%127.9 MB127.9 MB1
trialdsim0.13 s0.1%96.6 MB96.6 MB1
othersim0.00 s0.0%443.9 MB443.9 MB6

argent

ProcessScopeCPU-sShareRSS avgRSS peakSpawns
simulator-serverhost2.50 s92.0%96.6 MB96.7 MB1
node argenthost0.21 s7.8%85.7 MB87.9 MB1
node argent (exit charge)host0.01 s0.2%0 B0 B0
(node)host0.00 s0.0%0 B0 B0
node tool-server.cjshost0.00 s0.0%0 B0 B0
simctlhost0.00 s0.0%0 B0 B0
system_profilerhost0.00 s0.0%0 B0 B0
iOSsim98.77 s74.0%34.8 GB34.9 GB287
wardenexamplesim29.50 s22.1%375.5 MB435.1 MB8
bun cli.tssim2.78 s2.1%298.6 MB308.0 MB1
simctlsim0.41 s0.3%363.8 KB20.5 MB7
CoreSimulatorBridgesim0.40 s0.3%117.8 MB118.2 MB1
launchd_simsim0.37 s0.3%52.2 MB52.2 MB1
pssim0.29 s0.2%1.9 MB2.0 MB260
(mlhostd)sim0.26 s0.2%222.5 KB40.5 MB2
trialdsim0.20 s0.1%96.7 MB96.7 MB1
storekitdsim0.19 s0.1%128.4 MB128.4 MB1
(managedappdistri)sim0.14 s0.1%214.5 KB54.5 MB1
mlhostdsim0.13 s0.1%119.3 KB15.7 MB2
othersim0.05 s0.0%444.5 MB453.2 MB9

agent-device

ProcessScopeCPU-sShareRSS avgRSS peakSpawns
xcodebuildhost7.50 s73.3%218.0 MB218.7 MB1
bun cli.tshost1.77 s17.3%268.7 MB273.3 MB1
snapshot-bridgehost0.77 s7.5%116.3 MB116.4 MB1
node agent-devicehost0.15 s1.5%61.0 MB66.8 MB1
pshost0.02 s0.2%1.9 MB1.9 MB182
simctlhost0.02 s0.2%20.4 MB20.4 MB1
node agent-device (exit charge)host0.01 s0.1%0 B0 B0
(simctl)host0.00 s0.0%96.9 KB17.2 MB1
DTServiceHubhost0.00 s0.0%16.7 MB16.7 MB1
caffeinatehost0.00 s0.0%7.3 MB7.3 MB1
(bash)host0.00 s0.0%0 B0 B0
(clang)host0.00 s0.0%0 B0 B0
otherhost0.00 s0.0%0 B0 B0
iOSsim89.31 s65.7%35.5 GB35.5 GB291
wardenexamplesim35.81 s26.3%371.9 MB438.3 MB8
AgentDeviceRunnerUITests-Runnersim9.14 s6.7%268.2 MB268.2 MB1
CoreSimulatorBridgesim0.51 s0.4%80.6 MB80.7 MB1
simctlsim0.48 s0.4%676.7 KB20.5 MB7
launchd_simsim0.31 s0.2%52.4 MB52.4 MB1
trialdsim0.13 s0.1%96.7 MB96.7 MB1
storekitdsim0.13 s0.1%128.3 MB128.3 MB1
xpcproxy_simsim0.09 s0.1%176.0 KB9.1 MB4
mlhostdsim0.08 s0.1%34.2 KB6.1 MB1
(wardenexample)sim0.00 s0.0%2.4 MB433.2 MB1
assetsdsim0.00 s0.0%163.2 MB163.2 MB1
othersim0.00 s0.0%274.3 MB274.3 MB4

tester-army

ProcessScopeCPU-sShareRSS avgRSS peakSpawns
xcodebuildhost14.26 s84.0%217.0 MB218.2 MB1
node entry.jshost1.05 s6.2%84.9 MB92.8 MB1
snapshot-bridgehost0.63 s3.7%116.2 MB116.3 MB1
(node) (exit charge)host0.62 s3.7%0 B0 B0
node e2ehost0.35 s2.1%109.4 MB132.3 MB1
(node)host0.03 s0.2%0 B0 B0
other (exit charge)host0.03 s0.2%0 B0 B0
(ps)host0.01 s0.1%27.9 KB1.7 MB5
simctlhost0.00 s0.0%20.5 MB20.5 MB1
DTServiceHubhost0.00 s0.0%16.8 MB16.8 MB1
(pgrep)host0.00 s0.0%0 B0 B0
(simctl)host0.00 s0.0%0 B0 B0
otherhost0.00 s0.0%0 B0 B0
iOSsim107.48 s60.4%35.1 GB35.2 GB289
wardenexamplesim49.85 s28.0%380.4 MB441.8 MB9
AgentDeviceRunnerUITests-Runnersim15.86 s8.9%271.0 MB271.2 MB1
bun cli.tssim2.56 s1.4%254.0 MB264.4 MB1
CoreSimulatorBridgesim0.57 s0.3%80.6 MB80.6 MB1
simctlsim0.54 s0.3%534.1 KB20.8 MB8
launchd_simsim0.46 s0.3%51.8 MB51.8 MB1
mlhostdsim0.24 s0.1%135.3 KB14.3 MB3
trialdsim0.16 s0.1%96.6 MB96.7 MB1
storekitdsim0.15 s0.1%128.2 MB128.2 MB1
launchctlsim0.10 s0.1%36.7 KB9.4 MB1
pssim0.06 s0.0%1.9 MB1.9 MB263
othersim0.07 s0.0%446.4 MB870.8 MB8

One table per framework, from the warm-median rep's timed run (the rep whose host CPU-seconds is the lower median). Rows are processes grouped by label (top 12 per scope, the rest in other); host is the framework's own process tree, sim the simulator's launchd_sim tree (iOS only). Share is of the scope's CPU-seconds. RSS is the label's summed RSS per 500 ms tick (0 where it had no process). Spawns counts distinct pids a tick saw: a process that lives between two ticks is only seen through the (exit charge) rows (CPU of a child the bench ran, booked at its exit), so many short-lived xcrun simctl calls read low in Spawns and in CPU.

Residue

  • warden: none
  • argent: paths: /Users/expo/.argent (328.0 KB)
  • agent-device: driver apps: com.callstack.agentdevice.runner.uitests.xctrunner; paths: /Users/expo/.agent-device-bench/agent-device (2.9 MB)
  • tester-army: driver apps: com.callstack.agentdevice.runner.uitests.xctrunner; paths: /Users/expo/.agent-device-bench/tester-army (4.9 MB)

Artifacts

Each worker uploaded its results directory as one archive. The viewable files are copied here; the archives hold everything, including the per-rep resources.ndjson ticks and the 60 s idle control. To re-render one: download the archive, tar -xzf it, then bun run bench report Users/expo/bench-results from the warden repo.

Merged run

The four results merged into one run, bench-2026-10-25-ios-eas, ratios recomputed across the four: report.html, report.md, bench-results.json, summary.json (the comparison page's dataset).

warden

Worker job bench_warden, bun 1.3.14 + warden-e2e (this commit). Download the whole artifact (the EAS eas-ios-warden archive, unpacks to Users/expo/bench-results).

The warden worker's simulator at exit

Argent

Worker job bench_argent, @swmansion/argent 0.30.1. Download the whole artifact (the EAS eas-ios-argent archive, unpacks to Users/expo/bench-results).

The Argent worker's simulator at exit

agent-device

Worker job bench_agent_device, agent-device 0.21.31. Download the whole artifact (the EAS eas-ios-agent-device archive, unpacks to Users/expo/bench-results).

The agent-device worker's simulator at exit

tester.army

Worker job bench_tester_army, e2e 0.20.0, bundling agent-device 0.21.25. Download the whole artifact (the EAS eas-ios-tester-army archive, unpacks to Users/expo/bench-results).

The tester.army worker's simulator at exit

Follow-up: warden after the helper fix

The profile in docs/bench/cpu-profile-2026-10-25.md put 74 % of warden-simd's CPU in idb's generic Codable decode of every accessibility frame. The helper now links idb with a single-pass frame decoder (fork commit 1768a0d on v1.7.2, library package idb-v1.7.2-warden.1). Same workflow, same worker type and simulator, warden only, 5 runs: run 01a1739d at commit a87b04cc. The bench app was rebuilt by EAS for this run (sha256 c283eae41c7a, same source as the run above), so the comparison page keeps the four-framework run above as its dataset and this run stands beside it.

warden, per rep (warm)Run above (01a172f5)After (01a1739d)Delta
Host CPU-seconds9.12 s3.79 s-5.33 s (-58 %)
of which warden-simd7.03 s1.83 s-5.20 s (-74 %)
Host RSS peak184.9 MB177.6 MB-7.3 MB
Suite median97.04 s91.89 s-5.15 s (-5 %)
Pass rate / flakes100 % / 0100 % / 0

The simulator's own CPU over the whole job (every process under launchd_sim, prepare and all five reps) read higher on this worker (821 CPU-s against 564 CPU-s) while both simulators idled alike before the run (idle control in the artifacts); the host is now out of the way, so the in-simulator readers are the next thing to profile.

Worker job bench_warden. Download the whole artifact (the EAS eas-ios-warden archive, unpacks to Users/expo/bench-results).

The warden worker's simulator at exit, after the helper fix

On this page