flagon-io/g1t

Why

Deploy #443's core (ts) and core (runner-image) jobs failed with "The runner stopped answering." after about 10 minutes, and neither one recorded a single step. They started at 00:11, minutes after Deploy #442 began rolling the runner's containers onto a new image. Their sandboxes were lost before the runner inside them ran anything. Nothing reported that, so the jobs sat until the sweep's 10-minute SILENT_MS failed them.

What changes

• A job's first report normally arrives 6–12 s after it starts (measured on Deploy #445). A hosted job that hasn't reported anything 3 minutes in (FIRST_REPORT_MS, never_reported) never ran anything, so the sweep puts it back in the queue:
  • its token is voided, so the lost sandbox can no longer report for it (job_for requires in_progress and a matching token);
  • the lost sandbox is stopped (stop_actions_job), which gives back what billing held;
  • it restarts at the next sweep, not this one. The same actions:<job> sandbox takes it, and the old one's stop finishes in that sandbox's alarm afterwards (start_queued_except).
• It shares #180's wait budget: after WAIT_FOR_MACHINE_MS (30 min) it fails with a clear message.
• A job that goes quiet after reporting still fails at SILENT_MS, as before, because whatever it did may need undoing.

Not confirmed

The rollout is the likely trigger, going by the timing. I couldn't read the runner's own logs to prove it: scripts/ops/runner-errors.mjs needs a CLOUDFLARE_API_TOKEN. The fix doesn't depend on the cause. Whatever loses a sandbox before it starts, the job gets retried.

The lost sandbox's few minutes are billed as for any stopped sandbox, so a lost start costs about 3 minutes.

Checked

cargo test -p g1t-actions-service (121, including never_reported), clippy, and the wasm build.

Cause confirmed from the runner's logs (Cloudflare Workers Observability). It wasn't the rollout. Deploy #443's core (ts) and core (runner-image) were started from inside the request in which an earlier job reported it had finished. No 4-core container was free, so the containers library kept retrying "no container instance … try again later". At 00:13:03 the start's RPC was cancelled: the finished job's container had disconnected, and its request was cancelled with everything in it. The start gave up at 00:13:14 with nobody left to tell. Neither job's container ever made a single request, but both jobs were already marked running, so they sat until SILENT_MS. This PR's 3-minute never_reported retry recovers exactly this case. I've corrected the comment on FIRST_REPORT_MS (a6204c6).