Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.
| Money in billing's messages reads to the cent, and the deploy reads migrations again after a failure | 1 | #!/usr/bin/env node |
| 2 | // Releases of the self-hosted runner, `g1t-runner` (crates/runner): built | |
| 3 | // for every platform, checksummed, signed, and published to the | |
| 4 | // g1t-downloads R2 bucket that g1t.sh serves at /downloads/runner/ | |
| 5 | // (apps/web/app/routes/downloads-runner.ts). | |
| 6 | // | |
| 7 | // node scripts/runner-release.mjs keygen # once: the release key | |
| 8 | // node scripts/runner-release.mjs build [--targets linux-x64,windows-x64] | |
| Merge the runner sandbox: g1t-runner sandbox up makes a person's own computer a self-hosted runner that nothing a job runs can get out of. Linux jobs run in a Linux VM of the runner's own (a WSL 2 distribution with no host mounts, interop or Docker socket on Windows; vfkit on macOS; Sysbox, or QEMU and KVM, on Linux), each in a fresh container under Sysbox inside it; on Windows 11 Pro and up, Windows jobs each run in a Windows Sandbox discarded after the job. Runners report their isolation, and the Runners page shows it. | 9 | // node scripts/runner-release.mjs manifest # again, with the sandbox's images |
| Money in billing's messages reads to the cent, and the deploy reads migrations again after a failure | 10 | // node scripts/runner-release.mjs sign # latest.json + .sig |
| 11 | // node scripts/runner-release.mjs verify # checks the signature | |
| 12 | // node scripts/runner-release.mjs publish [--dry-run] | |
| 13 | // | |
| 14 | // What a release is, at runner/<version>/ in the bucket: | |
| 15 | // | |
| 16 | // g1t-runner-linux-x64, -linux-arm64, -macos-x64, -macos-arm64, | |
| 17 | // -windows-x64.exe the binaries | |
| 18 | // SHA256SUMS `sha256 name`, one line each | |
| Merge the runner sandbox: g1t-runner sandbox up makes a person's own computer a self-hosted runner that nothing a job runs can get out of. Linux jobs run in a Linux VM of the runner's own (a WSL 2 distribution with no host mounts, interop or Docker socket on Windows; vfkit on macOS; Sysbox, or QEMU and KVM, on Linux), each in a fresh container under Sysbox inside it; on Windows 11 Pro and up, Windows jobs each run in a Windows Sandbox discarded after the job. Runners report their isolation, and the Runners page shows it. | 19 | // manifest.json { version, published_at, agent_image, files: { platform: { name, sha256 } }, |
| 20 | // sandbox: { runner_image, files: { "wsl-x64": { name, sha256, size }, … } } } | |
| 21 | // g1t-sandbox-* the sandbox's images (scripts/sandbox-image.mjs), when built | |
| Money in billing's messages reads to the cent, and the deploy reads migrations again after a failure | 22 | // |
| 23 | // and at runner/: latest.json (the newest release's manifest) and | |
| 24 | // latest.json.sig, its Ed25519 signature in base64. A runner trusts a | |
| 25 | // release only when the signature checks out against the public key built | |
| 26 | // into it (G1T_RUNNER_RELEASE_KEY at build time) and every file's SHA-256 | |
| 27 | // is the manifest's (crates/runner/src/selfhosted/update.rs). | |
| 28 | // | |
| 29 | // Environment: | |
| 30 | // RUNNER_RELEASE_KEY the private key (keygen prints it): a g1t Actions secret | |
| 31 | // G1T_RUNNER_RELEASE_KEY the public key, built into the binaries | |
| 32 | // RUNNER_AGENT_IMAGE the image agent work runs in, named in the manifest | |
| 33 | // | |
| 34 | // Every target is cross-compiled with cargo-zigbuild (`cargo install | |
| 35 | // cargo-zigbuild`, and zig on PATH), so one Linux machine builds them all. | |
| 36 | ||
| 37 | import { spawnSync } from "node:child_process"; | |
| 38 | import { createHash, createPrivateKey, createPublicKey, generateKeyPairSync, sign, verify } from "node:crypto"; | |
| 39 | import { copyFileSync, existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from "node:fs"; | |
| 40 | import { dirname, join } from "node:path"; | |
| 41 | import { fileURLToPath } from "node:url"; | |
| 42 | ||
| 43 | const ROOT = join(dirname(fileURLToPath(import.meta.url)), ".."); | |
| 44 | export const TARGETS = { | |
| 45 | "linux-x64": { triple: "x86_64-unknown-linux-musl", file: "g1t-runner-linux-x64" }, | |
| 46 | "linux-arm64": { triple: "aarch64-unknown-linux-musl", file: "g1t-runner-linux-arm64" }, | |
| 47 | "macos-x64": { triple: "x86_64-apple-darwin", file: "g1t-runner-macos-x64" }, | |
| 48 | "macos-arm64": { triple: "aarch64-apple-darwin", file: "g1t-runner-macos-arm64" }, | |
| 49 | "windows-x64": { triple: "x86_64-pc-windows-gnu", file: "g1t-runner-windows-x64.exe", exe: true }, | |
| 50 | }; | |
| 51 | const BUCKET = "g1t-downloads"; | |
| 52 | ||
| 53 | export function version() { | |
| 54 | const toml = readFileSync(join(ROOT, "crates/runner/Cargo.toml"), "utf8"); | |
| 55 | const found = /^version\s*=\s*"([^"]+)"/m.exec(toml); | |
| 56 | if (!found) throw new Error("crates/runner/Cargo.toml has no version"); | |
| 57 | return found[1]; | |
| 58 | } | |
| 59 | ||
| 60 | const outDir = (v = version()) => join(ROOT, "target", "runner-release", v); | |
| 61 | ||
| 62 | /** The 32 raw bytes of an Ed25519 public key, in base64, as the runner takes it. */ | |
| 63 | export function rawPublicKey(publicKey) { | |
| 64 | const der = publicKey.export({ type: "spki", format: "der" }); | |
| 65 | return der.subarray(der.length - 32).toString("base64"); | |
| 66 | } | |
| 67 | ||
| 68 | export function keygen() { | |
| 69 | const { privateKey, publicKey } = generateKeyPairSync("ed25519"); | |
| 70 | return { | |
| 71 | private: privateKey.export({ type: "pkcs8", format: "der" }).toString("base64"), | |
| 72 | public: rawPublicKey(publicKey), | |
| 73 | }; | |
| 74 | } | |
| 75 | ||
| 76 | function privateKey(text = process.env.RUNNER_RELEASE_KEY) { | |
| 77 | if (!text) throw new Error("RUNNER_RELEASE_KEY is not set"); | |
| 78 | return createPrivateKey({ key: Buffer.from(text, "base64"), format: "der", type: "pkcs8" }); | |
| 79 | } | |
| 80 | ||
| 81 | /** Signs `bytes`: the signature in base64. */ | |
| 82 | export function signBytes(bytes, key) { | |
| 83 | return sign(null, bytes, key).toString("base64"); | |
| 84 | } | |
| 85 | ||
| 86 | /** Whether `signature` is the key's over `bytes`, given the raw public key in base64. */ | |
| 87 | export function verifyBytes(bytes, signature, publicRaw) { | |
| 88 | const der = Buffer.concat([Buffer.from("302a300506032b6570032100", "hex"), Buffer.from(publicRaw, "base64")]); | |
| 89 | const key = createPublicKey({ key: der, format: "der", type: "spki" }); | |
| 90 | return verify(null, bytes, key, Buffer.from(signature, "base64")); | |
| 91 | } | |
| 92 | ||
| 93 | export const sha256 = (bytes) => createHash("sha256").update(bytes).digest("hex"); | |
| 94 | ||
| Merge the runner sandbox: g1t-runner sandbox up makes a person's own computer a self-hosted runner that nothing a job runs can get out of. Linux jobs run in a Linux VM of the runner's own (a WSL 2 distribution with no host mounts, interop or Docker socket on Windows; vfkit on macOS; Sysbox, or QEMU and KVM, on Linux), each in a fresh container under Sysbox inside it; on Windows 11 Pro and up, Windows jobs each run in a Windows Sandbox discarded after the job. Runners report their isolation, and the Runners page shows it. | 95 | /** |
| 96 | * The manifest of the files in a release folder. Its `sandbox` section, the | |
| 97 | * images `g1t-runner sandbox` runs and their SHA-256, comes from | |
| 98 | * sandbox.json there when scripts/sandbox-image.mjs made one, and is signed | |
| 99 | * with the rest. | |
| 100 | */ | |
| Money in billing's messages reads to the cent, and the deploy reads migrations again after a failure | 101 | export function manifest(dir, v, agentImage = process.env.RUNNER_AGENT_IMAGE || null) { |
| 102 | const files = {}; | |
| 103 | for (const [platform, target] of Object.entries(TARGETS)) { | |
| 104 | const path = join(dir, target.file); | |
| 105 | if (existsSync(path)) files[platform] = { name: target.file, sha256: sha256(readFileSync(path)) }; | |
| 106 | } | |
| Merge the runner sandbox: g1t-runner sandbox up makes a person's own computer a self-hosted runner that nothing a job runs can get out of. Linux jobs run in a Linux VM of the runner's own (a WSL 2 distribution with no host mounts, interop or Docker socket on Windows; vfkit on macOS; Sysbox, or QEMU and KVM, on Linux), each in a fresh container under Sysbox inside it; on Windows 11 Pro and up, Windows jobs each run in a Windows Sandbox discarded after the job. Runners report their isolation, and the Runners page shows it. | 107 | const m = { version: v, published_at: new Date().toISOString(), agent_image: agentImage, files }; |
| 108 | const sandboxPath = join(dir, "sandbox.json"); | |
| 109 | if (existsSync(sandboxPath)) { | |
| 110 | const sandbox = JSON.parse(readFileSync(sandboxPath, "utf8")); | |
| 111 | for (const file of Object.values(sandbox.files ?? {})) { | |
| 112 | if (!existsSync(join(dir, file.name))) throw new Error(`sandbox.json names ${file.name}, which is not in ${dir}`); | |
| 113 | } | |
| 114 | m.sandbox = sandbox; | |
| 115 | } | |
| 116 | return m; | |
| 117 | } | |
| 118 | ||
| 119 | /** Writes manifest.json and SHA256SUMS for what is in the folder now. */ | |
| 120 | function writeManifest(dir, v) { | |
| 121 | const m = manifest(dir, v); | |
| 122 | writeFileSync(join(dir, "manifest.json"), `${JSON.stringify(m, null, 2)}\n`); | |
| 123 | const all = [...Object.values(m.files), ...Object.values(m.sandbox?.files ?? {})]; | |
| 124 | writeFileSync(join(dir, "SHA256SUMS"), all.map((f) => `${f.sha256} ${f.name}`).join("\n") + "\n"); | |
| 125 | return m; | |
| Money in billing's messages reads to the cent, and the deploy reads migrations again after a failure | 126 | } |
| 127 | ||
| 128 | function run(command, args, options = {}) { | |
| 129 | const done = spawnSync(command, args, { stdio: "inherit", cwd: ROOT, shell: process.platform === "win32", ...options }); | |
| 130 | if (done.status !== 0) throw new Error(`${command} ${args.join(" ")} failed`); | |
| 131 | } | |
| 132 | ||
| 133 | function build(targets) { | |
| 134 | const v = version(); | |
| 135 | const dir = outDir(v); | |
| 136 | mkdirSync(dir, { recursive: true }); | |
| 137 | if (!process.env.G1T_RUNNER_RELEASE_KEY) console.error("warning: G1T_RUNNER_RELEASE_KEY is not set; these builds will not update themselves"); | |
| 138 | for (const platform of targets) { | |
| 139 | const target = TARGETS[platform]; | |
| 140 | if (!target) throw new Error(`unknown target ${platform}; one of ${Object.keys(TARGETS).join(", ")}`); | |
| 141 | console.error(`building ${platform} (${target.triple})`); | |
| 142 | run("rustup", ["target", "add", target.triple]); | |
| 143 | run("cargo", ["zigbuild", "--release", "--locked", "--package", "g1t-runner", "--target", target.triple]); | |
| 144 | const built = join(ROOT, "target", target.triple, "release", target.exe ? "g1t-runner.exe" : "g1t-runner"); | |
| 145 | copyFileSync(built, join(dir, target.file)); | |
| 146 | } | |
| Merge the runner sandbox: g1t-runner sandbox up makes a person's own computer a self-hosted runner that nothing a job runs can get out of. Linux jobs run in a Linux VM of the runner's own (a WSL 2 distribution with no host mounts, interop or Docker socket on Windows; vfkit on macOS; Sysbox, or QEMU and KVM, on Linux), each in a fresh container under Sysbox inside it; on Windows 11 Pro and up, Windows jobs each run in a Windows Sandbox discarded after the job. Runners report their isolation, and the Runners page shows it. | 147 | const m = writeManifest(dir, v); |
| Money in billing's messages reads to the cent, and the deploy reads migrations again after a failure | 148 | console.log(`built ${Object.keys(m.files).length} binaries of ${v} into ${dir}`); |
| 149 | } | |
| 150 | ||
| 151 | function signRelease() { | |
| 152 | const v = version(); | |
| 153 | const dir = outDir(v); | |
| 154 | const bytes = readFileSync(join(dir, "manifest.json")); | |
| 155 | writeFileSync(join(dir, "latest.json"), bytes); | |
| 156 | writeFileSync(join(dir, "latest.json.sig"), signBytes(bytes, privateKey())); | |
| 157 | console.log(`signed ${v}`); | |
| 158 | } | |
| 159 | ||
| 160 | function verifyRelease() { | |
| 161 | const dir = outDir(); | |
| 162 | const publicRaw = process.env.G1T_RUNNER_RELEASE_KEY; | |
| 163 | if (!publicRaw) throw new Error("G1T_RUNNER_RELEASE_KEY is not set"); | |
| 164 | const ok = verifyBytes(readFileSync(join(dir, "latest.json")), readFileSync(join(dir, "latest.json.sig"), "utf8"), publicRaw); | |
| 165 | if (!ok) throw new Error("latest.json's signature is not the release key's"); | |
| 166 | const m = JSON.parse(readFileSync(join(dir, "latest.json"), "utf8")); | |
| Merge the runner sandbox: g1t-runner sandbox up makes a person's own computer a self-hosted runner that nothing a job runs can get out of. Linux jobs run in a Linux VM of the runner's own (a WSL 2 distribution with no host mounts, interop or Docker socket on Windows; vfkit on macOS; Sysbox, or QEMU and KVM, on Linux), each in a fresh container under Sysbox inside it; on Windows 11 Pro and up, Windows jobs each run in a Windows Sandbox discarded after the job. Runners report their isolation, and the Runners page shows it. | 167 | const all = [...Object.values(m.files), ...Object.values(m.sandbox?.files ?? {})]; |
| 168 | for (const file of all) { | |
| Money in billing's messages reads to the cent, and the deploy reads migrations again after a failure | 169 | if (sha256(readFileSync(join(dir, file.name))) !== file.sha256) throw new Error(`${file.name}'s SHA-256 is not the manifest's`); |
| 170 | } | |
| Merge the runner sandbox: g1t-runner sandbox up makes a person's own computer a self-hosted runner that nothing a job runs can get out of. Linux jobs run in a Linux VM of the runner's own (a WSL 2 distribution with no host mounts, interop or Docker socket on Windows; vfkit on macOS; Sysbox, or QEMU and KVM, on Linux), each in a fresh container under Sysbox inside it; on Windows 11 Pro and up, Windows jobs each run in a Windows Sandbox discarded after the job. Runners report their isolation, and the Runners page shows it. | 171 | console.log(`verified ${m.version}: ${all.length} files${m.sandbox ? `, the sandbox's runner image ${m.sandbox.runner_image ?? "(none)"}` : ""}`); |
| Money in billing's messages reads to the cent, and the deploy reads migrations again after a failure | 172 | } |
| 173 | ||
| 174 | function publish(dryRun) { | |
| 175 | const v = version(); | |
| 176 | const dir = outDir(v); | |
| 177 | const put = (key, file, type) => { | |
| 178 | const args = ["wrangler", "r2", "object", "put", `${BUCKET}/${key}`, "--file", file, "--remote", "--content-type", type]; | |
| 179 | if (dryRun) console.log(`would put ${key}`); | |
| 180 | else run("npx", args, { cwd: join(ROOT, "apps/web") }); | |
| 181 | }; | |
| 182 | // The version's files first; latest.json last, so no runner is pointed | |
| 183 | // at files that are not there yet. | |
| 184 | for (const name of readdirSync(dir)) { | |
| Merge the runner sandbox: g1t-runner sandbox up makes a person's own computer a self-hosted runner that nothing a job runs can get out of. Linux jobs run in a Linux VM of the runner's own (a WSL 2 distribution with no host mounts, interop or Docker socket on Windows; vfkit on macOS; Sysbox, or QEMU and KVM, on Linux), each in a fresh container under Sysbox inside it; on Windows 11 Pro and up, Windows jobs each run in a Windows Sandbox discarded after the job. Runners report their isolation, and the Runners page shows it. | 185 | // sandbox.json is in manifest.json already. |
| 186 | if (name.startsWith("latest.json") || name === "sandbox.json") continue; | |
| Money in billing's messages reads to the cent, and the deploy reads migrations again after a failure | 187 | put(`runner/${v}/${name}`, join(dir, name), name.endsWith(".json") ? "application/json" : "application/octet-stream"); |
| 188 | } | |
| 189 | put("runner/latest.json", join(dir, "latest.json"), "application/json"); | |
| 190 | put("runner/latest.json.sig", join(dir, "latest.json.sig"), "text/plain"); | |
| 191 | } | |
| 192 | ||
| 193 | if (process.argv[1]?.replaceAll("\\", "/").endsWith("scripts/runner-release.mjs")) { | |
| 194 | const [command, ...rest] = process.argv.slice(2); | |
| 195 | const option = (name) => { | |
| 196 | const at = rest.indexOf(`--${name}`); | |
| 197 | return at >= 0 ? rest[at + 1] : undefined; | |
| 198 | }; | |
| 199 | try { | |
| 200 | switch (command) { | |
| 201 | case "keygen": { | |
| 202 | const keys = keygen(); | |
| 203 | console.log(`RUNNER_RELEASE_KEY=${keys.private}\nG1T_RUNNER_RELEASE_KEY=${keys.public}`); | |
| 204 | console.error("Keep RUNNER_RELEASE_KEY secret (a g1t Actions secret); G1T_RUNNER_RELEASE_KEY is public and goes into every build."); | |
| 205 | break; | |
| 206 | } | |
| 207 | case "build": | |
| 208 | build((option("targets") ?? Object.keys(TARGETS).join(",")).split(",")); | |
| 209 | break; | |
| Merge the runner sandbox: g1t-runner sandbox up makes a person's own computer a self-hosted runner that nothing a job runs can get out of. Linux jobs run in a Linux VM of the runner's own (a WSL 2 distribution with no host mounts, interop or Docker socket on Windows; vfkit on macOS; Sysbox, or QEMU and KVM, on Linux), each in a fresh container under Sysbox inside it; on Windows 11 Pro and up, Windows jobs each run in a Windows Sandbox discarded after the job. Runners report their isolation, and the Runners page shows it. | 210 | case "manifest": { |
| 211 | // Again, once the sandbox's images have joined the binaries. | |
| 212 | const m = writeManifest(outDir(), version()); | |
| 213 | console.log(`manifest of ${m.version}: ${Object.keys(m.files).length} binaries, ${Object.keys(m.sandbox?.files ?? {}).length} sandbox images`); | |
| 214 | break; | |
| 215 | } | |
| Money in billing's messages reads to the cent, and the deploy reads migrations again after a failure | 216 | case "sign": |
| 217 | signRelease(); | |
| 218 | break; | |
| 219 | case "verify": | |
| 220 | verifyRelease(); | |
| 221 | break; | |
| 222 | case "publish": | |
| 223 | publish(rest.includes("--dry-run")); | |
| 224 | break; | |
| 225 | default: | |
| 226 | console.error("usage: node scripts/runner-release.mjs keygen|build|sign|verify|publish"); | |
| 227 | process.exit(2); | |
| 228 | } | |
| 229 | } catch (error) { | |
| 230 | console.error(String(error.message ?? error)); | |
| 231 | process.exit(1); | |
| 232 | } | |
| 233 | } |
This file's history is long; its oldest lines are credited to the oldest commit read.