Skip to content

Where people and agents ship software together. The open-source git platform for the whole job: issues, agents, checks and deploys to the edge.

A job Cloudflare has no container for waits for a machine instead of failing #181

Mergedsyntaqx merged wait-for-a-machine into main
syntaqxopened this pull request

Why

When Cloudflare couldn't give a job's sandbox a container, the job failed outright about 20 seconds in. The error was there is no container instance that can be provided to this durable object. In the last day it failed Plugins release #17, Deploy #443 (the production deploy of #178), Caches #182, and CI on #175 and #179. Most of those started minutes after Deploy #442 rolled out a new runner image. Cloudflare gives this answer when every instance of a class is taken or a rollout is replacing them, and both of those pass. Nothing retried.

What changes

  • Runner: that particular refusal now answers unavailable with "No class machine was free to start the job." (noMachine in lifecycle.ts). Any other start failure still answers conflict as before. Billing's hold is already given back when a start fails.
  • Actions planner: a job refused as unavailable goes back in the queue, with the reason on the job, and the every-minute sweep retries it. That sweep stops starting jobs, since the ones after it would be refused too. If a job has waited 30 minutes (WAIT_FOR_MACHINE_MS, tracked in the new waiting_since column, migration 0025), it fails saying no machine came free.

Not in this PR

Deploy #443's core (ts) and core (runner-image) jobs did get containers, then failed after ~10 minutes with "The runner stopped answering." That's a different failure, possibly the same rollout evicting running containers. I haven't confirmed it, so it isn't addressed here.

Checked

cargo test -p g1t-actions-service (120, including the new wait rule), the runner's tests (92, including noMachine) and typecheck, clippy, the actions service's wasm build, and Prettier.

syntaqxmerged this·

All checks have passed

2 passed · on 036a8a6

Workflows

CI / pull_request — CI passedDetails
Dependency review — No dependency changesDetails

Merged and closed: the branch can be deleted safely

Everything on it is in its base now.

Sign in to comment.