Measure and cut a hosted job's time to its first step

Time to a hosted job's first step, measured

The last 25 CI and 25 Deploy runs (2026-10-20 22:15 to 2026-10-21 14:30 UTC): 280 hosted jobs (132 CI, 148 Deploy). Timestamps are from the jobs API (started_at is set when the scheduler claims the job, just before it asks the runner for a sandbox) and from each job's "Set up" log. A job is queued at its run's created_at, or when the last job it needs finished. For Deploy, the time a run waited behind the previous one in deploy-production (by design) is left out: 10 of 148 jobs.

By workflow (median / p90): CI queued to first step 17.4 s / 46.4 s, of which queued to assigned 7.7 s / 36.1 s; Deploy 13.0 s / 33.4 s, of which queued to assigned 3.9 s / 22.0 s.

What the numbers say:

1. Queued to assigned is the largest phase, and most of it is self-inflicted. The scheduler (start_queued_except) claimed a job, waited for its sandbox to start, then claimed the next. The median gap between two starts in one batch was 2.4 s, so by start order the median wait was 1.7 s for a batch's first job, 4.5 s second, 7.4 s third, 9.9 s fourth, 12.7 s fifth, 15.7 s sixth.
2. The tail (p90 32 s, max 346 s) comes from "no machine free". The runner logged many There is no container instance that can be provided to this Durable Object refusals on Sandbox4Core (2026-10-21 00:32 to 00:34, 13:04, and 14:07 to 14:16, among others; the telemetry query stops at 100 events). Each refused job goes back to the queue and is only tried again at the next one-minute cron sweep. Jobs whose wait overlapped a refusal: median 13.5 s, p90 120 s. Other jobs: 5.2 s / 22 s.
3. Container start and image pull are fast: 5.9 s median. The 4.7 GB image is evidently pre-pulled, so a smaller image would gain little here. Without container-level timestamps I can't split sandbox start from image pull and runner boot; neither the jobs API nor runner telemetry has them.
4. Set up job's actions took 3.3 s, about 1.7 s per action, fetched one after another (look up, then archive download).

What PR #291 changes

• The scheduler still claims jobs one at a time, so each sees the room the ones before it took. It then starts them all together (join_all). Expected: every job in a batch is assigned in about one start time (~2 s), up to ~13 s sooner for a run's sixth job. This alone should bring the median queued to assigned down to about 2 s.
• The runner looks up all of a job's actions, and downloads their archives, in parallel. Expected: about 1.7 s instead of 3.3 s per job.

Together: a median of about 15.5 s down to about 9 s from queued to first step. No cost change: nothing is kept warm.

Not done here (follow-ups): #292 retries a job refused for lack of a machine sooner than the one-minute sweep, and #293 looks into why the "no container instance" refusals happen at all. A warm sandbox per busy repository isn't justified by these numbers. Container start is ~6 s, and a warm 4-core sandbox would cost ~$0.11/hour, ~$80/month, to save ~4 s per job.