Field notes: optimizing the fleet

Seven findings from a week of measuring and fixing the fleet. All numbers measured, no estimates.

The push daemon that ate the fleet

Idle iOS 26.5 simulators were burning 50–213% CPU each. The culprit was apsd stuck in a failing APNs connect-retry loop. Disabling apsd and diagnosticd inside the sim took idle CPU to 1–6% — roughly 50x less — and is what makes a warm pool affordable.

Pause, don't reboot

SIGSTOP a simulator's whole launchd_sim process tree and it parks at 0% CPU; SIGCONT thaws it in 26 ms on an M3 (~225 ms on Intel) with a cached PID list, vs 20 s–2 min boots. A 2 h 50 m park thawed perfectly. Never run simctl against a parked sim — it hangs instead of erroring.

The golden image that silently lost its slimming

launchctl disable state lives outside the simulator data directory, so data-dir-swap stamping dropped all 141 disabled services and the sims booted effectively stock (~204 processes vs 77 slim). The fix: boot, re-apply, and verify after every stamp and erase — 141/141 disables, 65 services vs ~330 stock, 4.6 s boot-to-ready.

Boot storms, not steady state

Every load catastrophe (load averages past 500 on 8 cores) was concurrent simulator boots, never steady-state work. Boots now go through a semaphore — 1 at a time on Intel, 2 on Apple Silicon — with a load gate at 2x core count and a 5 GiB free-disk gate.

Designing for a 40 ms link

Agents reach the Macs over a 40–65 ms relayed link at 5–20 Mbit/s. JPEG q70 downscaled to 800 px cut screenshots 26x (574,479 B → 22,252 B; end-to-end 2.6 s → 1.2 s). tap_element does observe→find→tap in one request, and actions:batch ships up to 32 actions in one round trip.

The workstation that never throttles

A 2013 Mac Pro locks 92 W and 3.6 GHz on all cores indefinitely; 2017 MacBook Pros hit their thermal wall in ~2 minutes, cap near 30 W, and hold ~73% of starting speed. Clean build: 13 m 10 s vs 15 m 26 s and 16 m 31 s — and the gap widens on back-to-back CI builds.

Warm sims, golden images, and a broker

Everything else that landed since v0.1: simbridge, a resident helper that takes taps from 3.05 s to 30 ms on Intel (~100x); golden images that stamp pre-slimmed sims in seconds instead of ~2 minutes of slimming; copy-on-write simctl-clone snapshots that survive restore; manzanas-broker fronting N daemons behind one address without proxying frames; and manzanas-eval, a harness that reports determinism rate and per-step latency percentiles in CI.

By the numbers