Sign in

flagon-io/g1t

public

Where people and agents ship software together. The open-source git platform for the whole job: issues, agents, checks and deploys to the edge.

gitagentscloudflarerustforgemcp

Commit

The runner's base image, pushed as Wrangler builds images, with pushes that fail loudly

syntaqxcommitted Parentedad7daBrowse files
5 files+80−120/5 viewed
+8−0
346346 so a runner change pushes about 5 MB (`scripts/deploy/image.mjs`).
347347 - **Ask.** Document `--cache-to type=registry` support on `registry.cloudflare.com`, or a hosted build
348348 cache.
349+- **Also hit (2026-10-06).** The first push of the 2.7 GB base to `registry.cloudflare.com` failed after
350+ several layers with `error from registry: blob unknown to registry`, from `docker push` and from
351+ `wrangler containers push` alike. The same push, run again an hour later, found every layer there and
352+ finished in 17 s. The image was an OCI index carrying a BuildKit provenance attestation (an
353+ `unknown/unknown` manifest), the default for `docker buildx build --load`; Wrangler's own builds pass
354+ `--provenance=false`. We now build as Wrangler does (one manifest, no provenance or SBOM) and push up
355+ to three times, failing loudly when no digest comes back. Ask: say whether manifests may reference
356+ just-uploaded blobs at once, and whether indexes with attestation manifests are supported.
349357
350358 ## Tooling and account
351359
+22−1
188188 image uploads only its own layer (about 5 MB): the base's layers are
189189 already there.
190190
191+Both are built as Wrangler builds images (`--platform linux/amd64
192+--provenance=false --sbom=false`): one manifest, not an OCI index with a
193+BuildKit attestation beside it. A push is tried up to three times and counts
194+only when Docker reports the digest; the first push of the base once failed
195+with `blob unknown to registry` and went through when run again (see
196+`docs/CLOUDFLARE_FEEDBACK.md`, C3). `build-base` and `image` exit non-zero
197+when a push fails, and `base.json` is written only after the push.
198+
191199 **The base** is recorded in `services/runner/base.json`: its reference, its
192200 digest, a hash of its folder (`inputs`), when it was built, its size and
193201 each toolchain's version. `node scripts/deploy.mjs build-base` builds it,
306314 `preserve_order` from `api` and `actions`, `digest` features from
307315 `secrets`), so each worker-build afterwards compiled its own variant
308316 again.
317+- **The runner's images**, measured on the same machine on 2026-10-06
318+ (Docker Desktop, 8 vCPUs; its disk was busy with other containers, so
319+ the cold figures are slow and noisy):
320+
321+ | | Before (one image) | Now |
322+ | --- | --- | --- |
323+ | Size, unpacked / compressed (what a machine pulls) | 3.08 GB / 819 MB | 2.68 GB / 686 MB |
324+ | A change to the runner | Docker rebuilds the image's Rust stage and pushes the image (1198 s in the first deploy after a prune) | binary 46–53 s (6 s unchanged), image 7 s, push one 5 MB layer (1.4 s to a local registry) |
325+ | The base from nothing | 431 s (whole image, cold) | 758 s cold, rarely: weekly or when its folder changes |
326+ | The base after `docker builder prune` | as from nothing | 68 s, its layers pulled from the registry it was pushed to |
327+ | The runner binary, cold (builder container) | | 99 s |
328+ | A Rust CI job's build (events, search, repos; 4 vCPUs) | | 51 s cold, 13 s with the Cargo target restored (107 MB zstd entry) |
329+
309330 - **Only what changed** is the largest saving: a change to one service
310331 deploys one service.
311332
373394 `jobHosts`). Under flagon-io/g1t's **Settings → Guardrails**
374395 (Maintain role or higher), **Workflow-only domains**:
375396
376−```
397+```text
377398 api.cloudflare.com | deploy.yml | production
378399 registry.cloudflare.com | deploy.yml, runner-base.yml | production
379400 ```
+13−0
1414 baseTag,
1515 contentHash,
1616 lockFor,
17+ pushImage,
18+ pushedDigest,
1719 readBaseLock,
1820 registryHas,
1921 removeDeployConfig,
215217 await assert.rejects(registryHas("not-a-reference", { fetchImpl, credentials }));
216218 });
217219
220+test("a push counts only when it ends with a digest, and is tried again", async () => {
221+ const digest = "sha256:" + "a".repeat(64);
222+ assert.equal(pushedDigest({ code: 0, out: `base: digest: ${digest} size: 856` }), digest);
223+ assert.equal(pushedDigest({ code: 0, out: "error from registry: blob unknown to registry" }), null);
224+ assert.equal(pushedDigest({ code: 1, out: `digest: ${digest}` }), null);
225+ assert.equal(pushedDigest({ code: 0, out: "Pushed" }), null);
226+ const answers = [{ code: 1, out: "error from registry: blob unknown to registry" }, { code: 0, out: `x: digest: ${digest} size: 1` }];
227+ assert.equal(await pushImage("r/x:y", { run: async () => answers.shift() }), digest);
228+ await assert.rejects(pushImage("r/x:y", { attempts: 2, run: async () => ({ code: 0, out: "blob unknown" }) }), /docker push r\/x:y failed/);
229+});
230+
218231 test("a static Linux binary is told from a dynamic one", () => {
219232 const dir = mkdtempSync(join(tmpdir(), "g1t-elf-"));
220233 try {
+33−7
137137 if (creds && creds.until > Date.now() && (!push || creds.push)) return creds;
138138 const args = ["containers", "registries", "credentials", REGISTRY, "--pull", "--json", "--expiration-minutes", "60"];
139139 if (push) args.push("--push");
140− const got = await wrangler(args, { cwd: ROOT });
140+ // In a unit's folder, not the root, whose .env may hold a token for something else.
141+ const got = await wrangler(args, { cwd: join(ROOT, "services/runner") });
141142 if (got.code !== 0) throw new Error(`could not get registry credentials:\n${lastLines(got.out)}`);
142143 const { username, password } = jsonFrom(got.out);
143144 creds = { username, password, push, until: Date.now() + 50 * 60 * 1000 };
195196 */
196197 export async function buildBase(unit, { tag, previous, onLine, account = ACCOUNT_ID, noCache = false }) {
197198 const ref = `${repositoryOf(unit, account)}:${tag}`;
198− const args = ["buildx", "build", "--platform", "linux/amd64", "--progress", "plain", "--load", "--build-arg", "BUILDKIT_INLINE_CACHE=1", "-t", ref];
199+ const args = ["buildx", "build", ...PLAIN_IMAGE, "--progress", "plain", "--load", "--build-arg", "BUILDKIT_INLINE_CACHE=1", "-t", ref];
199200 if (previous && !noCache) args.push("--cache-from", `type=registry,ref=${previous}`);
200201 if (noCache) args.push("--no-cache");
201202 args.push(join(ROOT, unit.image.base.context));
222223 }
223224
224225 /** Pushes `ref`; returns its digest in the registry. */
225−export async function pushImage(ref, { onLine } = {}) {
226− const pushed = await exec("docker", ["push", ref], { onLine });
227− if (pushed.code !== 0) throw new Error(`docker push ${ref} failed:\n${lastLines(pushed.out)}`);
228− return /digest: (sha256:[0-9a-f]{64})/.exec(pushed.out)?.[1] ?? null;
226+export async function pushImage(ref, { onLine = () => {}, attempts = 3, run = exec } = {}) {
227+ let last = "";
228+ for (let attempt = 1; attempt <= attempts; attempt++) {
229+ const pushed = await run("docker", ["push", ref], { onLine });
230+ last = pushed.out;
231+ const digest = pushedDigest(pushed);
232+ if (digest) return digest;
233+ // Cloudflare's registry can answer "blob unknown" for a layer it has
234+ // just taken; pushing again finds the layers there and finishes.
235+ if (attempt < attempts) onLine(`docker push ${ref} did not finish (attempt ${attempt} of ${attempts}); trying again`);
236+ }
237+ throw new Error(`docker push ${ref} failed:\n${lastLines(last)}`);
238+}
239+
240+/**
241+ * The digest a `docker push` ended with, or null if it did not end with
242+ * one, whatever its exit code: an error from the registry is a failure
243+ * even when Docker exits 0.
244+ */
245+export function pushedDigest({ code, out }) {
246+ if (code !== 0 || /error from registry|blob unknown|unknown blob|denied|unauthorized/i.test(out)) return null;
247+ return /digest: (sha256:[0-9a-f]{64})/.exec(out)?.[1] ?? null;
229248 }
230249
231250 /**
251+ * Build flags for an image Cloudflare Containers takes as Wrangler builds
252+ * it: one manifest for linux/amd64, with no provenance or SBOM attestation,
253+ * which would make it an index with an `unknown/unknown` manifest.
254+ */
255+export const PLAIN_IMAGE = ["--platform", "linux/amd64", "--provenance=false", "--sbom=false"];
256+
257+/**
232258 * Builds the runner's image: the base and the binary, from a build
233259 * context holding only the binary (target/runner-image).
234260 */
235261 export async function buildRunnerImage(unit, { ref, base, binaryDir, onLine }) {
236262 const built = await exec(
237263 "docker",
238− ["build", "--platform", "linux/amd64", "--progress", "plain", "--build-arg", `BASE=${base}`, "-f", join(ROOT, unit.image.dockerfile), "-t", ref, binaryDir],
264+ ["buildx", "build", ...PLAIN_IMAGE, "--progress", "plain", "--load", "--build-arg", `BASE=${base}`, "-f", join(ROOT, unit.image.dockerfile), "-t", ref, binaryDir],
239265 { onLine },
240266 );
241267 if (built.code !== 0) throw new Error(`docker build of the runner's image failed:\n${lastLines(built.out, 30)}`);
+4−4
11 {
22 "image": "registry.cloudflare.com/1e6f2cffa3f445920836e8ebe446bb58/g1t-runner:base-20261006-809f7651f6d9",
3− "digest": null,
4− "pushed": false,
3+ "digest": "sha256:17899aa70d29de209c1fc5bed77e6e090d338a1158b911ea8673ab3b811ef006",
4+ "pushed": true,
55 "inputs": "809f7651f6d994e790729cd365be7100eb00e50b637bef313bb04361ab39f394",
6− "built_at": "2026-10-06T07:43:55.123Z",
7− "size_bytes": 2683372742,
6+ "built_at": "2026-10-06T08:22:38.137Z",
7+ "size_bytes": 2683372502,
88 "versions": {
99 "node": "v24.21.0",
1010 "npm": "11.19.0",