iOS simulator input through the HID injector #1019

Part of #1013.

Scope

After this PR, a person holding an iOS simulator lease taps, swipes, scrolls, presses Home, and types hardware keys on an iOS 26 or older runtime, through WS /v1/leases/{id}/input. A physical iPhone lease answers inputUnavailable: true. The app switcher, Lock, rotation and keys on iOS 27 runtimes are dropped; #1024 holds them.

In short

An iOS simulator's device input is the vendored capture program run in a new input mode. It loads serve-sim's HID injector, which Simlock now vendors, and reads input messages on stdin. Home brings SpringBoard forward with simctl, the way serve-sim does it on Xcode 26 and later. On an iOS 27 runtime the guest ignores injected keys, so Simlock drops keys there and says so in 0x82.

flowchart LR
  S[input module, #1017] --> P[device input port, #1017]
  P --> I[iOS device input NEW]
  I -- spawn --> H[simlock-screen-capture --input NEW]
  I -- stdin: len, tag, JSON --> H
  H -- HIDInjector, re-vendored --> SIM[(simulator)]
  I -- simctl --set launch springboard --> SIM
  P --> X[physical iPhone: unavailable]

• input mode: simlock-screen-capture <udid> --input, with SIMLOCK_IOS_DEVICE_SET set as for a capture. It writes no picture.
• On stdin, each message is the stream's envelope: 4-byte big-endian length, one tag byte, then the serve-sim JSON, for example tag 0x03 with {"type":"begin","x":0.5,"y":0.25}. End of stdin stops the helper.
• On stdout, the helper writes one envelope with tag 0x82 and {"width":<px>,"height":<px>} once its HID client is set up. The size is the device type's native screen size (F19).
• 0x82 on iOS always says "orientation":"portrait". It adds "keyboardUnavailable":true when the simulator's runtime is iOS 27 or later, or cannot be read (F10).
• Failed, not unavailable: a missing binary, no 0x82 within 10 s, or a helper that exits early makes the open reject, and Input socket for a lease #1017 closes with 1013. A helper that exits later calls onFailed.

If this goes wrong, a person sees taps do nothing, or a helper left running after the socket closed.

Technical spec

Facts

Commit e1f2f43.

Modules touched

Searches: screenCapture, createIosScreenCapture, findSimDevice, nativeScreenSizes, SIMLOCK_IOS_DEVICE_SET, HIDInjector, build-capture, BytePipeProcess, closeStdin in src/, vendor/, scripts/; snapshot in e2e/.

• vendor/screen-capture/Sources/simlock-screen-capture/: HIDInjector.swift (re-vendored, patched), an input mode in main.swift, FrameCapture.nativeScreenSizes made internal.
• vendor/screen-capture/NOTICE.md, docs/internal/VENDORED-CODE.md, vendor/README.md: the file taken and two new deviations.
• src/ports/byte-pipe.ts, src/ports/byte-pipe.test.ts, src/ports/AGENTS.md: BytePipeProcess.writeStdin(bytes); the fake records stdinWrites.
• src/drivers/ios/input.ts (new): the iOS device input.
• src/drivers/ios/index.ts: sets the driver's input (Input socket for a lease #1017's Driver.input) next to screenCapture.
• src/drivers/ios/AGENTS.md, website/src/content/docs/docs/using-devices/watching-the-screen.mdx.
• e2e/slow-agent-device.test.ts: turns on, for iOS, the tap case Android input through scrcpy's control channel #1018 adds (Android input through scrcpy's control channel #1018 skips it on iOS).

flowchart LR
  core[core driver catalog, #1017] --> drv[ios/index.ts *]
  drv --> inp[ios/input.ts NEW]
  inp --> bp[ports/byte-pipe * writeStdin]
  bp --> bin[simlock-screen-capture --input *]
  bin --> hid[HIDInjector.swift re-vendored]
  bin --> fc[FrameCapture *]

sequenceDiagram
  participant M as input module
  participant I as iOS device input
  participant H as helper --input
  M->>I: openInput(simulator)
  I->>H: spawn udid --input, SIMLOCK_IOS_DEVICE_SET
  H-->>I: 0x82 {width, height}
  I-->>M: device input; onConfig (+ keyboardUnavailable on iOS 27)
  M->>I: 0x03 touch begin
  I->>H: writeStdin [len][0x03]{...}
  Note over I,H: no 0x82 in 10 s
  I-->>M: open rejects (1013)
  Note over H: helper exits later
  I-->>M: onFailed (1013)

Surface and extension points

createIosDeviceInput(options) returns the driver's input, the implementation of #1017's device input port (feature Shared values, "Device input port"). Its options mirror IosScreenCaptureOptions: launcher, command, deviceSet, baseEnvironment, clock, plus a process runner for simctl. A physical iPhone gets {unavailable: "Input is not supported on physical iPhones"}. It is one of two implementations beside #1018's; the port and Driver.input are #1017's.

BytePipeProcess.writeStdin(bytes): Promise<void> (new): writes bytes to the child's stdin and rejects when the write fails. NodeBytePipeLauncher still ignores EPIPE after closeStdin() (F12), but a write before that surfaces its error.

Contract and event changes

none. Searched src/contract/ for input shapes: the messages are #1017's core/input.ts. No event.

Other code on the same state

Searches: simlock-screen-capture, SIMLOCK_IOS_DEVICE_SET, simctl launch, springboard, closeStdin in src/ and vendor/ (7 hits).

• A running capture of the same simulator: both are separate processes of the same binary. The input mode opens a HID client, the capture a framebuffer; neither reads the other.
• The capture's own use of BytePipeProcess: it never calls writeStdin; its closeStdin and EPIPE handling are unchanged.
• agent-device on the same simulator: both inject input. Whichever lands last wins; that is the take-over the feature asks for.
• home relaunches SpringBoard. An app the agent was driving goes to the background, as a Home press does.
• The health monitor rebooting a crashed simulator under the lease: the helper keeps a HID client bound to the old boot, so taps may vanish while it stays alive. A known gap, as the capture's own "a screen stream can freeze after a reboot under its lease" (src/drivers/ios/AGENTS.md, Known gaps).

Failure modes

A worker restart mid-session leaves a helper whose stdin closed, so it exits (F5). #1017 shares one device input per lease, so two helpers never run for one lease.

Rules in play

• vendor-and-third-party-code.md rule 6, "names its origin per file, never by a count": NOTICE.md's file table gains HIDInjector.swift.
• vendor-and-third-party-code.md rule 11, "updates the notice and the inventory in the same change": NOTICE.md, the VENDORED-CODE.md row and vendor/README.md.
• VENDORED-CODE.md "How a new vendored thing gets here", step 4: src/source-hygiene.test.ts checks the shape.
• NOTICE.md "Deviations": two new ones. HIDInjector drops the Duo, Device Hub, pose, Home, app switcher and Lock paths (F16, F18): Home is the TS device input's simctl --set … launch … springboard, so it has one owner (architecture rule 10); FrameCapture.nativeScreenSizes is internal, not private (F19).
• architecture rule 2, "iOS and Android are encapsulated in their own driver modules": HID and the runtime rule stay in src/drivers/ios/.
• architecture rule 11, "Every wait that crosses a process boundary is bounded": spawn, ready, simctl and close have limits.

Tests

Seam: createIosDeviceInput with a FakeBytePipeLauncher and a scripted helper, on a FakeClock; model on src/drivers/ios/screen-capture.test.ts. The port change extends src/ports/byte-pipe.test.ts.

• Touches, buttons, scrolls and keys on an iOS simulator reach the HID helper: touch begin/move/end written to stdin as envelopes; home runs simctl --set <set> launch <udid> com.apple.springboard; lock and app_switcher dropped; scroll written to stdin; a key on an iOS 26 runtime written to stdin; a key on iOS 27 or an unknown runtime dropped, with keyboardUnavailable: true in the config; helper missing → rejected; no 0x82 from the helper in 10 s → rejected (at 9.9 s still waiting); helper exits → failed.
• A physical iPhone's device input is unavailable: physical device; no process is started.

Done when

• The two Tests lines and the byte-pipe test pass in pnpm test, and src/source-hygiene.test.ts passes.
• pnpm build:capture builds simlock-screen-capture with the input mode where Xcode is selected (toolchain: Swift is built only there).
• On a simulator, e2e/slow-agent-device.test.ts opens a lease and the Appduct fixture through agent-device, opens WS /v1/leases/{id}/input, gets 0x83 and 0x82, taps the centre of tab-status as agent-device's snapshot places it (as a fraction of the screen size in the same snapshot --json), and the next snapshot shows the Status tab's screen (slow lane, iOS simulator). This task turns on, for iOS, the tap case Android input through scrcpy's control channel #1018 adds.
• watching-the-screen.mdx lists what input does on iOS: touch, scroll, Home, keys on iOS 26 runtimes; not the app switcher, Lock, rotation or keys on iOS 27 (iOS simulator input on Xcode 27: hardware keys, app switcher, Lock and rotation #1024).

Out of scope

• Rotation, the app switcher, Lock, and keys on iOS 27 runtimes (iOS simulator input on Xcode 27: hardware keys, app switcher, Lock and rotation #1024).
• A 0x82 that follows the simulator's real orientation. The HID takes positions as fractions of the device's native frame, and today's capture keeps that frame's size after a rotation, so picture and taps stay in the same frame (Assumption for the implementer to confirm on the slow lane).
• Duo devices and their hinge.
• Input on physical iPhones.

Depends on

• Input socket for a lease #1017
• Android input through scrcpy's control channel #1018 (adds the shared tap case in e2e/slow-agent-device.test.ts this task turns on for iOS)

Written by an agent.