flagon-io/g1t

Closes #292.

When the runner answers unavailable (no container instance for the sandbox), the job goes back in the queue. Until now it was tried again only at the next minute's cron sweep, or when some other request happened to start queued jobs. Jobs whose wait overlapped a refusal waited a median 13.5 s and p90 120 s to be assigned, against 5.2 s / 22 s otherwise.

Change

• start_queued_except returns whether any job was refused for want of a machine.
• New start_queued_retrying: while there are refusals, it starts queued jobs again after 2, 5 and 10 s (MACHINE_RETRY_MS, 17 s in all), then leaves the rest to the sweep as before. WAIT_FOR_MACHINE_MS (30 min) still fails a job that never gets one.
• It runs only where nothing waits on it: the request's wait_until after its answer (17 s of pauses fits in the 30 s it may run), and the end of the minute's sweep. advance(), which the sweep calls once per held run, still starts jobs once, so the sweep never sleeps per run.

Worth knowing

• During a burst, several requests may each retry at once. Starts claim jobs with UPDATE … WHERE status = 'queued', so nothing starts twice, but the runner sees more start calls while it has no capacity. Why the refusals happen below max_instances at all is #293.

Tested

• cargo test -p g1t-actions-service (134 passed, including a new test of the schedule).
• cargo clippy -p g1t-actions-service --target wasm32-unknown-unknown builds without warnings in the changed files.
• No end-to-end test exercises a refusal: conformance doesn't stub the runner's start_actions_job.