Skip to content

Where people and agents ship software together. The open-source git platform for the whole job: issues, agents, checks and deploys to the edge.

g1t.sh
338 linesCodeBlameRaw

Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.

Inbox: a bell in the top bar opens it beside the page, and /inbox holds it all1// The runner's Containers images, as the deploy tool builds and ships them
The docs folder is gone, and what it held lives where people read it: how a self-hosted g1t runs and how to deploy g1t to Cloudflare are pages on docs.g1t.sh under Run g1t yourself, and speed, rate limits and operating g1t.sh are sections of CONTRIBUTING.md; code that cited a file in docs/ now points to the page or section that covers it, or says what it means itself, and applied migrations and the runner images are left as they were.2// (docs.g1t.sh/guides/deploy-to-cloudflare/#the-runners-images):
Inbox: a bell in the top bar opens it beside the page, and /inbox holds it all3//
4// base g1t-runner:base-<date>-<inputs> services/runner/base/Dockerfile:
5// the OS, toolchains and the Claude Code CLI. Rebuilt only when
6// its folder changes, or weekly (.g1t/workflows/runner-base.yml),
7// and recorded in services/runner/base.json.
8// runner g1t-runner:<content> services/runner/Dockerfile: the
9// base plus the runner binary (scripts/build-runner.mjs). Its tag
10// is a hash of everything it is built from, so the same source
11// always names the same image, and an image already in the
A deploy makes the runner's image in the registry, without Docker: it reads the base's manifest and config from Cloudflare's registry, uploads one gzipped layer holding the runner binary at /usr/local/bin/g1t-runner with mode 0755, and writes a config (the entrypoint g1t-runner, the base's command cleared, as the Dockerfile's ENTRYPOINT does) and a manifest that name the base's layers and that one, under the runner's tag, for linux/amd64 (scripts/deploy/registry.mjs, the registry's own upload calls with the credentials Wrangler gives). Deploy #256 ran out of disk because docker build pulled the 4.5 GB base into the job's own Docker Engine, compressed and unpacked, on a 20 GB sandbox whose own filesystem is that same base; now nothing of the base is downloaded and the job's Docker Engine is never started. Docker is asked for only when it is needed: a dry run (build, or image --no-push) still builds services/runner/Dockerfile with it, and so does a deploy whose registry step fails, logging df -h and docker system df before and after. The image job removes npm's downloads, which it only restores, once its install has read them, and logs the disk before and after the deploy. Tests cover the layer's tar, the config and manifest, the calls against a registry kept in memory (the base's layers are never fetched), the token flow, and that the Dockerfile says the same path, mode and entrypoint; the image was also checked against a local registry, where it runs and has the same config as a docker build of the Dockerfile.12// registry is never made again.
Inbox: a bell in the top bar opens it beside the page, and /inbox holds it all13//
A deploy makes the runner's image in the registry, without Docker: it reads the base's manifest and config from Cloudflare's registry, uploads one gzipped layer holding the runner binary at /usr/local/bin/g1t-runner with mode 0755, and writes a config (the entrypoint g1t-runner, the base's command cleared, as the Dockerfile's ENTRYPOINT does) and a manifest that name the base's layers and that one, under the runner's tag, for linux/amd64 (scripts/deploy/registry.mjs, the registry's own upload calls with the credentials Wrangler gives). Deploy #256 ran out of disk because docker build pulled the 4.5 GB base into the job's own Docker Engine, compressed and unpacked, on a 20 GB sandbox whose own filesystem is that same base; now nothing of the base is downloaded and the job's Docker Engine is never started. Docker is asked for only when it is needed: a dry run (build, or image --no-push) still builds services/runner/Dockerfile with it, and so does a deploy whose registry step fails, logging df -h and docker system df before and after. The image job removes npm's downloads, which it only restores, once its install has read them, and logs the disk before and after the deploy. Tests cover the layer's tar, the config and manifest, the calls against a registry kept in memory (the base's layers are never fetched), the token flow, and that the Dockerfile says the same path, mode and entrypoint; the image was also checked against a local registry, where it runs and has the same config as a docker build of the Dockerfile.14// Both live in one repository of Cloudflare's registry. A deploy makes the
15// runner's image there (registry.mjs): it uploads one layer with the
16// binary and a manifest naming the base's layers and that one, and pulls
17// nothing, so it needs neither Docker nor room for the base. Building it
18// with Docker is for dry runs, and the fallback. `wrangler deploy` is
19// given the runner's registry reference (a generated config), so a deploy
20// never builds an image itself.
Inbox: a bell in the top bar opens it beside the page, and /inbox holds it all21
22import { createHash } from "node:crypto";
23import { execFileSync } from "node:child_process";
24import { existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
25import { dirname, join } from "node:path";
26
27import { ACCOUNT_ID, exec, jsonFrom, lastLines, wrangler } from "./cloudflare.mjs";
28import { parseCargoLock } from "./lockfiles.mjs";
A deploy makes the runner's image in the registry, without Docker: it reads the base's manifest and config from Cloudflare's registry, uploads one gzipped layer holding the runner binary at /usr/local/bin/g1t-runner with mode 0755, and writes a config (the entrypoint g1t-runner, the base's command cleared, as the Dockerfile's ENTRYPOINT does) and a manifest that name the base's layers and that one, under the runner's tag, for linux/amd64 (scripts/deploy/registry.mjs, the registry's own upload calls with the credentials Wrangler gives). Deploy #256 ran out of disk because docker build pulled the 4.5 GB base into the job's own Docker Engine, compressed and unpacked, on a 20 GB sandbox whose own filesystem is that same base; now nothing of the base is downloaded and the job's Docker Engine is never started. Docker is asked for only when it is needed: a dry run (build, or image --no-push) still builds services/runner/Dockerfile with it, and so does a deploy whose registry step fails, logging df -h and docker system df before and after. The image job removes npm's downloads, which it only restores, once its install has read them, and logs the disk before and after the deploy. Tests cover the layer's tar, the config and manifest, the calls against a registry kept in memory (the base's layers are never fetched), the token flow, and that the Dockerfile says the same path, mode and entrypoint; the image was also checked against a local registry, where it runs and has the same config as a docker build of the Dockerfile.29import { appendRunnerImage } from "./registry.mjs";
Inbox: a bell in the top bar opens it beside the page, and /inbox holds it all30import { ROOT, parseJsonc } from "./stack.mjs";
31
32export const REGISTRY = "registry.cloudflare.com";
33
34/** The generated config a runner deploy uses, beside its wrangler.jsonc. */
35export const DEPLOY_CONFIG = "wrangler.deploy.json";
36
37const posix = (path) => path.replaceAll("\\", "/");
38
39/** Tracked and untracked (not ignored) files under `paths`, sorted. */
40export function filesUnder(paths, root = ROOT) {
41 if (!paths.length) return [];
42 const out = execFileSync("git", ["ls-files", "-co", "--exclude-standard", "-z", "--", ...paths], { cwd: root, encoding: "utf8", maxBuffer: 64 << 20 });
43 return [...new Set(out.split("\0").filter(Boolean).map(posix))].filter((file) => existsSync(join(root, file))).sort();
44}
45
46/**
47 * A hash of files' paths and contents, line endings made LF so a Windows
48 * checkout and a Linux one agree.
49 */
50export function contentHash(files, root = ROOT, read = (file) => readFileSync(join(root, file))) {
51 const hash = createHash("sha256");
52 for (const file of [...files].sort()) {
53 const bytes = read(file);
54 const text = bytes.includes(0) ? bytes : Buffer.from(bytes.toString("utf8").replaceAll("\r\n", "\n"));
55 hash.update(`${file}\0${text.length}\0`);
56 hash.update(text);
57 }
58 return hash.digest("hex");
59}
60
61/** The repository both images are pushed to, for an account. */
62export function repositoryOf(unit, account = ACCOUNT_ID) {
63 return `${REGISTRY}/${account}/${unit.image.repository}`;
64}
65
66/** What services/runner/base.json says, or null. */
67export function readBaseLock(unit, root = ROOT) {
68 const file = join(root, unit.image.base.lock);
69 return existsSync(file) ? JSON.parse(readFileSync(file, "utf8")) : null;
70}
71
72/** A hash of everything the base is built from: its folder. */
73export function baseInputs(unit, root = ROOT) {
74 return contentHash(filesUnder([unit.image.base.context], root), root);
75}
76
77/** The base's tag for its inputs, built on `date`. */
78export function baseTag(inputs, date = new Date()) {
79 return `base-${date.toISOString().slice(0, 10).replaceAll("-", "")}-${inputs.slice(0, 12)}`;
80}
81
82/**
83 * Whether the base recorded in base.json is the one its folder builds:
84 * { lock, inputs, current, pushed, ref } where `ref` is what the runner's
85 * image is built FROM (by digest, once pushed).
86 */
87export function baseState(unit, root = ROOT) {
88 const lock = readBaseLock(unit, root);
89 const inputs = baseInputs(unit, root);
90 // Pinned by digest once pushed, so the tag moving cannot change what is built.
91 const ref = lock ? (lock.pushed && lock.digest ? `${lock.image}@${lock.digest}` : lock.image) : null;
92 return { lock, inputs, current: Boolean(lock && lock.inputs === inputs), pushed: Boolean(lock?.pushed), ref };
93}
94
95/**
96 * The runner image's tag: a hash of its Dockerfile, the base it is built
97 * on, and every file the binary is built from (the crates in
98 * `unit.image.dirs`, the workspace's Cargo files, the build script).
99 */
100export function runnerTag(unit, root = ROOT) {
101 const base = readBaseLock(unit, root);
102 // Cargo.lock counts only for the packages the binary is built from, so a
103 // dependency bumped for another service names the same image.
104 const files = filesUnder([...unit.image.dirs, ...unit.image.files.filter((f) => f !== unit.image.base.lock && f !== "Cargo.lock")], root);
105 const lock = existsSync(join(root, "Cargo.lock")) ? readFileSync(join(root, "Cargo.lock"), "utf8") : "";
106 const hash = createHash("sha256");
107 hash.update(contentHash(files, root));
108 hash.update(`\0lock\0${lockFor(lock, unit.image.crate)}`);
109 hash.update(`\0base\0${base?.image ?? ""}\0${base?.digest ?? ""}`);
110 return hash.digest("hex").slice(0, 16);
111}
112
113/** The Cargo.lock entries `crate` is built from, in a stable order. */
114export function lockFor(text, crate) {
115 const packages = parseCargoLock(text);
116 const seen = new Set();
117 const stack = [crate];
118 while (stack.length) {
119 const name = stack.pop();
120 if (seen.has(name)) continue;
121 seen.add(name);
122 for (const entry of packages.get(name) ?? []) stack.push(...entry.deps);
123 }
124 return [...seen]
125 .sort()
126 .map((name) => `${name} ${JSON.stringify((packages.get(name) ?? []).map((e) => [e.version, e.source, e.checksum]))}`)
127 .join("\n");
128}
129
130/** The runner image's full reference for this checkout. */
131export function runnerRef(unit, root = ROOT, account = ACCOUNT_ID) {
132 return `${repositoryOf(unit, account)}:${runnerTag(unit, root)}`;
133}
134
135// ── The registry ──────────────────────────────────────────────────────────
136
137let creds = null;
138
139/** Short-lived registry credentials, from Wrangler (your login or the token). */
140export async function registryCredentials({ push = false } = {}) {
141 if (creds && creds.until > Date.now() && (!push || creds.push)) return creds;
142 const args = ["containers", "registries", "credentials", REGISTRY, "--pull", "--json", "--expiration-minutes", "60"];
143 if (push) args.push("--push");
144 // In a unit's folder, not the root, whose .env may hold a token for something else.
145 const got = await wrangler(args, { cwd: join(ROOT, "services/runner") });
146 if (got.code !== 0) throw new Error(`could not get registry credentials:\n${lastLines(got.out)}`);
147 const { username, password } = jsonFrom(got.out);
148 creds = { username, password, push, until: Date.now() + 50 * 60 * 1000 };
149 return creds;
150}
151
152/** Logs Docker in to Cloudflare's registry. */
153export async function dockerLogin({ push = false } = {}) {
154 const { username, password } = await registryCredentials({ push });
155 const done = await exec("docker", ["login", "--username", username, "--password-stdin", REGISTRY], { input: password });
156 if (done.code !== 0) throw new Error(`docker login ${REGISTRY} failed:\n${lastLines(done.out)}`);
157}
158
159const MANIFEST_TYPES = [
160 "application/vnd.oci.image.index.v1+json",
161 "application/vnd.oci.image.manifest.v1+json",
162 "application/vnd.docker.distribution.manifest.list.v2+json",
163 "application/vnd.docker.distribution.manifest.v2+json",
164].join(", ");
165
166/** Whether `ref` (registry/account/repo:tag) is in the registry. Needs no Docker. */
167export async function registryHas(ref, { fetchImpl = fetch, credentials = registryCredentials } = {}) {
168 const match = /^([^/]+)\/(.+):([^:/]+)$/.exec(ref);
169 if (!match) throw new Error(`not an image reference with a tag: ${ref}`);
170 const [, host, name, tag] = match;
171 const { username, password } = await credentials();
172 const response = await fetchImpl(`https://${host}/v2/${name}/manifests/${tag}`, {
173 method: "HEAD",
174 headers: { authorization: `Basic ${Buffer.from(`${username}:${password}`).toString("base64")}`, accept: MANIFEST_TYPES },
175 });
176 if (response.status === 404) return false;
177 if (!response.ok) throw new Error(`the registry answered ${response.status} for ${ref}`);
178 return true;
179}
180
181// ── Docker ────────────────────────────────────────────────────────────────
182
183/** Whether Docker has `ref` locally. */
184export async function dockerHas(ref) {
185 return (await exec("docker", ["image", "inspect", ref, "--format", "{{.Id}}"])).code === 0;
186}
187
188/** Bytes of an image as Docker keeps it unpacked, and its layers' compressed total when known. */
189export async function imageSize(ref) {
190 const found = await exec("docker", ["image", "inspect", ref, "--format", "{{.Size}}"]);
191 return found.code === 0 ? Number(found.out.trim()) : null;
192}
193
194/**
195 * Builds the base. Its cache comes from the base it replaces: the image
196 * carries its own build cache (BUILDKIT_INLINE_CACHE), so a machine with
197 * none, or one just pruned, pulls the layers that did not change instead
198 * of building them again. `--cache-from` an image that cannot be pulled
199 * (no login, a first build) is skipped with a warning.
200 */
201export async function buildBase(unit, { tag, previous, onLine, account = ACCOUNT_ID, noCache = false }) {
202 const ref = `${repositoryOf(unit, account)}:${tag}`;
203 const args = ["buildx", "build", ...PLAIN_IMAGE, "--progress", "plain", "--load", "--build-arg", "BUILDKIT_INLINE_CACHE=1", "-t", ref];
204 if (previous && !noCache) args.push("--cache-from", `type=registry,ref=${previous}`);
205 if (noCache) args.push("--no-cache");
206 args.push(join(ROOT, unit.image.base.context));
207 const built = await exec("docker", args, { onLine });
208 if (built.code !== 0) throw new Error(`docker build of the base failed:\n${lastLines(built.out, 30)}`);
209 return ref;
210}
211
212/** What the base has, for base.json: each toolchain's version. */
213export async function baseVersions(ref) {
214 const script = [
215 'echo "node=$(node --version)"',
216 'echo "npm=$(npm --version)"',
217 'echo "python=$(python3 --version | cut -d" " -f2)"',
218 'echo "go=$(go version | cut -d" " -f3)"',
219 'echo "rust=$(rustc --version | cut -d" " -f2)"',
Merge runner image: Java 21, .NET 8, Ruby 3.3220 'echo "java=$(java --version | head -n1 | cut -d" " -f2)"',
221 'echo "dotnet=$(dotnet --version)"',
222 'echo "ruby=$(ruby --version | cut -d" " -f2)"',
Inbox: a bell in the top bar opens it beside the page, and /inbox holds it all223 'echo "git=$(git --version | cut -d" " -f3)"',
224 'echo "claude_code=$(claude --version | cut -d" " -f1)"',
Merge branch 'main' into actions-toolkit-oidc-artifacts225 'echo "docker=$(dockerd --version | cut -d" " -f3 | tr -d ,)"',
Inbox: a bell in the top bar opens it beside the page, and /inbox holds it all226 'echo "debian=$(cat /etc/debian_version)"',
227 ].join("; ");
228 const found = await exec("docker", ["run", "--rm", "--platform", "linux/amd64", "--entrypoint", "bash", ref, "-c", script]);
229 if (found.code !== 0) return {};
230 return Object.fromEntries(found.out.trim().split(/\r?\n/).map((line) => line.split("=")).filter((pair) => pair.length === 2));
231}
232
233/** Pushes `ref`; returns its digest in the registry. */
234export async function pushImage(ref, { onLine = () => {}, attempts = 3, run = exec } = {}) {
235 let last = "";
236 for (let attempt = 1; attempt <= attempts; attempt++) {
237 const pushed = await run("docker", ["push", ref], { onLine });
238 last = pushed.out;
239 const digest = pushedDigest(pushed);
240 if (digest) return digest;
241 // Cloudflare's registry can answer "blob unknown" for a layer it has
242 // just taken; pushing again finds the layers there and finishes.
243 if (attempt < attempts) onLine(`docker push ${ref} did not finish (attempt ${attempt} of ${attempts}); trying again`);
244 }
245 throw new Error(`docker push ${ref} failed:\n${lastLines(last)}`);
246}
247
248/**
249 * The digest a `docker push` ended with, or null if it did not end with
250 * one, whatever its exit code: an error from the registry is a failure
251 * even when Docker exits 0.
252 */
253export function pushedDigest({ code, out }) {
254 if (code !== 0 || /error from registry|blob unknown|unknown blob|denied|unauthorized/i.test(out)) return null;
255 return /digest: (sha256:[0-9a-f]{64})/.exec(out)?.[1] ?? null;
256}
257
258/**
259 * Build flags for an image Cloudflare Containers takes as Wrangler builds
260 * it: one manifest for linux/amd64, with no provenance or SBOM attestation,
261 * which would make it an index with an `unknown/unknown` manifest.
262 */
263export const PLAIN_IMAGE = ["--platform", "linux/amd64", "--provenance=false", "--sbom=false"];
264
265/**
A deploy makes the runner's image in the registry, without Docker: it reads the base's manifest and config from Cloudflare's registry, uploads one gzipped layer holding the runner binary at /usr/local/bin/g1t-runner with mode 0755, and writes a config (the entrypoint g1t-runner, the base's command cleared, as the Dockerfile's ENTRYPOINT does) and a manifest that name the base's layers and that one, under the runner's tag, for linux/amd64 (scripts/deploy/registry.mjs, the registry's own upload calls with the credentials Wrangler gives). Deploy #256 ran out of disk because docker build pulled the 4.5 GB base into the job's own Docker Engine, compressed and unpacked, on a 20 GB sandbox whose own filesystem is that same base; now nothing of the base is downloaded and the job's Docker Engine is never started. Docker is asked for only when it is needed: a dry run (build, or image --no-push) still builds services/runner/Dockerfile with it, and so does a deploy whose registry step fails, logging df -h and docker system df before and after. The image job removes npm's downloads, which it only restores, once its install has read them, and logs the disk before and after the deploy. Tests cover the layer's tar, the config and manifest, the calls against a registry kept in memory (the base's layers are never fetched), the token flow, and that the Dockerfile says the same path, mode and entrypoint; the image was also checked against a local registry, where it runs and has the same config as a docker build of the Dockerfile.266 * Makes the runner's image in the registry, without Docker: the base's
267 * layers, named by its manifest, and one layer with the binary at `binary`
268 * (scripts/deploy/registry.mjs). What deploys use; the Docker build below
269 * is for a dry run, and the fallback. Returns its digest.
270 */
271export async function makeRunnerImage({ ref, base, binary, onLine = () => {}, credentials = () => registryCredentials({ push: true }) }) {
272 return appendRunnerImage({ ref, base, binary: readFileSync(binary), credentials, onLine });
273}
274
275/**
276 * Logs how full the disk is, and what Docker keeps on it: before and after
277 * a Docker build, so a build that runs out of room says what took it.
278 * Never fails; on a machine without `df`, it logs what it can.
279 */
280export async function diskReport(onLine, when, { run = exec } = {}) {
281 if (process.platform === "win32") return;
282 onLine(`disk ${when}:`);
283 for (const [command, args] of [
284 ["df", ["-h", "/", "/tmp", process.env.HOME ?? "/root"]],
285 ["docker", ["system", "df"]],
286 ]) {
287 try {
288 const done = await run(command, args);
289 for (const line of done.out.trim().split(/\r?\n/)) onLine(` ${line}`);
290 } catch (error) {
291 onLine(` ${command}: ${error.message ?? error}`);
292 }
293 }
294}
295
296/**
Inbox: a bell in the top bar opens it beside the page, and /inbox holds it all297 * Builds the runner's image: the base and the binary, from a build
298 * context holding only the binary (target/runner-image).
299 */
300export async function buildRunnerImage(unit, { ref, base, binaryDir, onLine }) {
301 const built = await exec(
302 "docker",
303 ["buildx", "build", ...PLAIN_IMAGE, "--progress", "plain", "--load", "--build-arg", `BASE=${base}`, "-f", join(ROOT, unit.image.dockerfile), "-t", ref, binaryDir],
304 { onLine },
305 );
306 if (built.code !== 0) throw new Error(`docker build of the runner's image failed:\n${lastLines(built.out, 30)}`);
307 return ref;
308}
309
310// ── The Worker's config ───────────────────────────────────────────────────
311
312/**
313 * Writes the config a runner deploy uses: its wrangler.jsonc with every
314 * container's image set to `ref`, beside it so its relative paths hold.
315 * Returns the file's name (pass it as --config). Remove it after.
316 */
317export function writeDeployConfig(unit, ref, root = ROOT) {
318 const config = parseJsonc(readFileSync(join(root, unit.path, "wrangler.jsonc"), "utf8"));
319 delete config.$schema;
320 for (const container of config.containers ?? []) {
321 container.image = ref;
322 delete container.image_build_context;
323 delete container.image_vars;
324 }
325 const file = join(root, unit.path, DEPLOY_CONFIG);
326 mkdirSync(dirname(file), { recursive: true });
327 writeFileSync(file, `${JSON.stringify(config, null, 2)}\n`);
328 return DEPLOY_CONFIG;
329}
330
331export function removeDeployConfig(unit, root = ROOT) {
332 rmSync(join(root, unit.path, DEPLOY_CONFIG), { force: true });
333}
334
335/** Writes services/runner/base.json. */
336export function writeBaseLock(unit, lock, root = ROOT) {
337 writeFileSync(join(root, unit.image.base.lock), `${JSON.stringify(lock, null, 2)}\n`);
338}

This file's history is long; its oldest lines are credited to the oldest commit read.