Find why Sandbox4Core refuses starts with no container instance well below max_instances

Follows #266. Sandbox4Core (max_instances 30) often refuses starts with There is no container instance that can be provided to this Durable Object, try again later, even though flagon-io runs at most 12 jobs at once (BURST_PER_WORKSPACE). On 2026-10-21 there were 100+ refusals: bursts at 00:32 to 00:34, 13:04 and 14:07 to 14:16, across 24 sandboxes in the first 14 hours. Find out why:

Are stopped or finished sandboxes still counted against max_instances? (sleepAfter is 100m; does a container whose process exited release its instance at once?)
Does a deploy's rollout (rollout_active_grace_period 7200) hold old instances while new ones start?
Or is it Cloudflare's per-location capacity?

scripts/ops/runner-errors.mjs already sorts these into the "no container free" bucket; correlate it with the live instance count. Each refusal costs a job at least one cron sweep (up to a minute) before it is tried again.

Findings for 2026-10-21 (UTC). All from read-only queries: Workers Observability events, the containers API, and the g1t-actions D1.

Not max_instances. At every refusal burst, 5 to 19 jobs were running across all runner classes (jobs with started_at before and finished_at after the minute). Sandbox4Core's own share is smaller still, against its 30.

Not rollouts. Sandbox4Core rollouts started at 01:03, 02:03, 10:59, 11:59, 13:13, 13:26, 14:31, 15:08 and 16:08. The bursts came at 00:33–35, 01:04–05, 11:57–58, 12:28, 12:57–58, 13:04, 14:07–08, 14:14–16, 15:00–02, 15:33–34, 16:04–05 and 18:04–06. Most bursts are nowhere near a rollout, and the ones beside 13:13 and 14:31 came before them. Even if grace-period instances do count, ≤19 jobs can't hold 30.

Not finished sandboxes holding instances. With no jobs running, the containers API shows g1t-runner-sandbox4core at active 0, assigned 1, healthy 6, so exited sandboxes do release their instances.

What it is: start bursts outrun the capacity Cloudflare has prepared. Each burst lines up with a cluster of job starts (8 to 18 in about 2 minutes), which follow merges and pushes to main (CI fan-out): #244/#245 at 00:33, #275 at 13:06, #285/#286 at 14:13/14:17, #291 at 14:59, #288 at 16:03, #302 at 18:04. Every runner application shows instances: 7 (6 healthy and idle) whatever its max_instances. Starts beyond that prepared pool wait for Cloudflare to schedule a new instance, and until it does, a start fails with "no container instance … try again later". The refusals are transient: the worst Sandbox4Core sandbox (74f254b8…) was refused from 12:57:07 to 12:58:28 and then started. In all, 47 Sandbox4Core sandboxes were refused that day; GitEdge is hit the same way.

The counts are inflated. A refused sandbox's alarm retries the start about every 0.6 s, so one sandbox logs 30 to 85 refusals. The 777 events that day come from about 50 sandboxes.

What's left to fix. #303 already retries a refused job within seconds instead of waiting a cron sweep, which removes most of the cost. What remains:

Back off the alarm's start retry, so one refusal doesn't log dozens of events and churn the alarm.
Have runner-errors.mjs count distinct sandboxes per burst, not events.
If start latency in a burst matters, ask Cloudflare whether the prepared pool can be raised, or stagger starts slightly.

Work

Repositories and pull requests

On it

Sessions

Activity