Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 1 | # Deploying g1t |
| 2 | ||
| 3 | How g1t.sh gets to Cloudflare: one manifest that lists every deployable | |
| 4 | unit, one tool that deploys only what changed, and a g1t Actions workflow | |
| 5 | that runs that tool on every push to `main`. Internal: the public | |
| 6 | self-hosting guide is `apps/docs/src/content/docs/guides/self-hosting.md`. | |
| 7 | ||
| 8 | | Piece | Where | | |
| 9 | | --- | --- | | |
| 10 | | The manifest | `deploy/stack.jsonc` | | |
| 11 | | The tool | `scripts/deploy.mjs` (library and tests in `scripts/deploy/`) | | |
| 12 | | Rust Worker builds | `scripts/build-rust-worker.mjs`, every Rust unit's build command | | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 13 | | The runner's images | `services/runner/base/Dockerfile`, `services/runner/Dockerfile`, `services/runner/base.json`, `scripts/build-runner.mjs`, `scripts/deploy/image.mjs` | |
| 14 | | The workflows | `.g1t/workflows/deploy.yml`, `.g1t/workflows/runner-base.yml` | | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 15 | | The old entry point | `scripts/deploy.sh`, now a wrapper | |
| 16 | ||
| 17 | ## The manifest | |
| 18 | ||
| 19 | `deploy/stack.jsonc` names every unit: each folder with a `wrangler.jsonc`. | |
| 20 | It is JSONC rather than TOML so that Node reads it with no dependency, | |
| 21 | with the same parser as the Wrangler configs, and comments stay possible. | |
| 22 | ||
| 23 | | Field | | | |
| 24 | | --- | --- | | |
| 25 | | `path` | The unit's folder. | | |
| 26 | | `kind` | `rust-worker` (built by worker-build), `ts-worker` (Wrangler bundles it), `react-router` (`vite build` first), `astro` (`astro build` first). | | |
| 27 | | `worker` | The Worker's name. Must match its `wrangler.jsonc`. | | |
| 28 | | `d1` | `{ database, migrations }`, when it has a database. Must match its `wrangler.jsonc`. | | |
| 29 | | `stage` | `core`, `edge` or `front` (below). | | |
| 30 | | `secrets` | The Wrangler secrets it needs, by name. `node scripts/deploy.mjs doctor` checks they are set. | | |
| 31 | | `setup` | One-time steps no config can say, for a first deploy. | | |
| 32 | | `inputs` | Files outside its folder it is built from that no workspace metadata names. A test finds such imports. | | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 33 | | `image` | A Containers image: `dockerfile` (the image deployed), `crate` (the binary it adds), `base` (`{ context, lock }`: the base image's folder and the file recording the base that was pushed) and `repository` (where both are pushed). See [the runner's images](#the-runners-images). | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 34 | | `self_host` | `run`, `off`, `separate` or `none`: what `deploy/self-host/configs.mjs` does with it. | |
| 35 | ||
| 36 | What a unit is **built from** is never listed by hand. The tool reads it: | |
| 37 | Rust path dependencies from `cargo metadata`, workspace packages from each | |
| 38 | `package.json`, closed transitively. `node scripts/deploy.mjs manifest` | |
| 39 | prints the result: | |
| 40 | ||
| 41 | ``` | |
| 42 | unit stage kind worker d1 built from (besides its folder) | |
| 43 | events core rust-worker g1t-events g1t-events crates/contracts crates/kit | |
| 44 | repos core rust-worker g1t-repos g1t-repos crates/contracts crates/kit crates/scan crates/secrets | |
| 45 | runner core ts-worker g1t-runner crates/actions crates/runner packages/contracts | |
| 46 | og core ts-worker g1t-og packages/contracts | |
| 47 | web front react-router g1t packages/contracts packages/theme | |
| 48 | ... | |
| 49 | ``` | |
| 50 | ||
| 51 | Root files count too: `Cargo.toml`, `Cargo.lock` and | |
| 52 | `scripts/build-rust-worker.mjs` for Rust units; `package-lock.json` and | |
| 53 | `tsconfig.base.json` for the others; `package.json` for all. A lockfile | |
| 54 | change counts for a unit only if a package in that unit's graph changed, | |
| 55 | read from the lockfile itself (`scripts/deploy/lockfiles.mjs`), so bumping | |
| 56 | `sharp` for the docs does not redeploy the Rust services. | |
| 57 | ||
| 58 | ### Stages | |
| 59 | ||
| 60 | | Stage | What | Units | | |
| 61 | | --- | --- | --- | | |
| 62 | | `migrations` | Every pending D1 migration, in parallel, before any code | each unit's `d1` | | |
| 63 | | `core` | Services reached through bindings | the services, `og` | | |
| 64 | | `edge` | Public endpoints other than the site | `api`, `models`, `pages`, `status` | | |
| 65 | | `front` | The site, sudo, the docs | `web`, `sudo`, `docs` | | |
| 66 | ||
| 67 | A stage starts only when the one before it succeeded. Inside a stage units | |
| 68 | deploy in parallel. The rule the tests enforce: **a unit binds only to units | |
| 69 | in its own stage or an earlier one**, so new code never calls a service | |
| 70 | that has not shipped. (Services in `core` bind to each other in cycles, | |
| 71 | which is why they share a stage.) | |
| 72 | ||
| 73 | ### What reads the manifest | |
| 74 | ||
| 75 | - `scripts/deploy.mjs`: everything below. | |
| 76 | - `deploy/self-host/configs.mjs`: which Workers a self-hosted installation | |
| 77 | runs (`self_host: "run"`, the site first) and which are bound to the off | |
| 78 | Worker (`"off"`). Its output is identical to before, apart from the order | |
| 79 | of `workers.txt` after the site. | |
| 80 | - Tests (`npm run test:deploy`) check that: every `wrangler.jsonc` in the | |
| 81 | repository has a unit; `worker`, `d1` and `image` match the configs; | |
| 82 | every KV id is named under `resources.kv`; stages follow bindings; the | |
| 83 | derived dependencies agree with Cargo's own resolved graph; every import | |
| 84 | that leaves a unit's folder is covered; `deploy/self-host/Dockerfile` | |
| 85 | builds exactly the Rust units self-hosting runs; and the service table in | |
| 86 | `docs/SELF_HOSTING.md` names every unit. | |
| 87 | ||
| 88 | ## The tool | |
| 89 | ||
| 90 | ```sh | |
| 91 | node scripts/deploy.mjs plan # what would deploy, and why (read-only) | |
| 92 | node scripts/deploy.mjs deploy # migrations, then every changed unit | |
| 93 | node scripts/deploy.mjs deploy --only web,api # just these, if they changed | |
| 94 | node scripts/deploy.mjs deploy --only web --force # just this, changed or not | |
| 95 | node scripts/deploy.mjs deploy --all # everything | |
| 96 | node scripts/deploy.mjs build --only events # build as a deploy would; upload nothing | |
| 97 | node scripts/deploy.mjs migrate # pending migrations only | |
| 98 | node scripts/deploy.mjs manifest [--check|--json] | |
| 99 | node scripts/deploy.mjs doctor # secrets each unit lacks | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 100 | node scripts/deploy.mjs build-base # build and push the runner's base image (Docker) |
| 101 | node scripts/deploy.mjs image # build and push the runner's image for this checkout (Docker) | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 102 | scripts/deploy.sh [units...] # the old entry point: all, or those named, always |
| 103 | ``` | |
| 104 | ||
| 105 | | Flag | | | |
| 106 | | --- | --- | | |
| 107 | | `--only a,b` / `--skip a,b` | Units by short name, folder or Worker name. | | |
| 108 | | `--all` | Every unit, changed or not. | | |
| 109 | | `--force` | Deploy the selected units even if unchanged. | | |
| 110 | | `--concurrency N` | Units at once inside a stage, and migrations at once (default 4). | | |
| 111 | | `--stage core` | One stage only. | | |
| 112 | | `--no-migrations` | Skip the migrations step (the workflow runs it as its own job). | | |
| 113 | | `--allow-dirty` | Deploy with uncommitted changes in what deploys. The version records no commit, so the next plan deploys it again. | | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 114 | | `--rebuild-image` | Build the runner's image even if one for this source is already in the registry. | |
| 115 | | `--rebuild-base` | Build and push a new base first (needs Docker), then the runner's image on it. Writes `services/runner/base.json`: commit it. | | |
| 116 | | `--no-push` | `build-base` and `image`: build locally, push nothing. | | |
| 117 | | `--no-cache` | `build-base`: build every layer again. | | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 118 | | `--since REV` | Treat Workers with no recorded commit as running `REV`. Used once to adopt Workers deployed before this tool. | |
| 119 | | `--json`, `--out FILE`, `--github-output` | The plan as data, for the workflow. | | |
| 120 | ||
| 121 | ### Where the deployed commit is kept | |
| 122 | ||
| 123 | On the Worker itself. Every deploy runs `wrangler deploy --message | |
| 124 | "g1t-deploy <40-char sha> <subject>" --tag g1t-<12-char sha>`, which | |
| 125 | Cloudflare keeps as the version's `workers/message` and `workers/tag` | |
| 126 | annotations. `plan` reads them back with `wrangler deployments status` | |
| 127 | (the live version) and `wrangler versions list` (its annotations): two | |
| 128 | read-only calls per unit, in parallel; a plan of all 22 units takes about | |
| 129 | 10 seconds. No KV namespace or other infrastructure is needed. | |
| 130 | ||
| 131 | - A version made by `wrangler secret put` keeps the code of the one before | |
| 132 | it, so the tool looks through those to the deploy before. | |
| 133 | - A version deployed any other way (by hand, from the dashboard) has no | |
| 134 | commit, and the unit is deployed again. | |
| 135 | - During a gradual rollout the version with the most traffic counts. | |
| 136 | - After `wrangler rollback`, the plan sees the older commit and deploys | |
| 137 | what changed since. | |
| 138 | ||
| 139 | ### What a deploy does | |
| 140 | ||
| 141 | 1. Plans: for each unit, the live commit; `git diff` from it to `HEAD`; | |
| 142 | whether the changed files touch the unit (its folder, the crates and | |
| 143 | packages it is built from, its inputs, lockfile changes that reach it). | |
| 144 | 2. Refuses if uncommitted changes touch what would deploy (`--allow-dirty`). | |
| 145 | 3. Applies every pending migration (`wrangler d1 migrations apply --remote`), | |
| 146 | in parallel. Any failure stops the deploy before code. | |
| 147 | 4. Installs worker-build once if any Rust unit is deploying. | |
| 148 | 5. Each stage in turn: units in parallel (`--concurrency`), each `npm run | |
| 149 | build` first for React Router and Astro, then `wrangler deploy` in the | |
| 150 | unit's folder with the annotation. If any unit fails, later stages are | |
| 151 | not started. | |
| 152 | 6. Prints a table: unit, stage, result, version id, time. Each unit's full | |
| 153 | output is kept in `$TMPDIR/g1t-deploy/<unit>.log`. | |
| 154 | ||
| Deploying: a migration keeps the live code right, because it runs a minute before the new code | 155 | #### Migrations run while the old code is still live |
| 156 | ||
| 157 | Migrations apply before any code, and a stage takes a minute or more, so | |
| 158 | for that long the code in production is the old code reading the new | |
| 159 | schema. A migration must keep the old code right: | |
| 160 | ||
| 161 | - **Add, never change meaning.** New tables and columns (with defaults) are | |
| 162 | safe. Rewriting what existing rows mean is not: on 2026-10-08, | |
| 163 | `billing/0039` turned comped accounts into `custom` terms at 100%, and | |
| 164 | for about 40 seconds the old billing code, which knew only `comped`, saw | |
| 165 | flagon-io as a free workspace and refused its workflows. | |
| 166 | - **Change meaning in two deploys.** First ship code that reads both the old | |
| 167 | and the new form (and keeps writing the old one); then, in a later | |
| 168 | deploy, the migration that rewrites the rows; then, if you like, code | |
| 169 | that drops the old form. | |
| 170 | - **Never drop or rename** a column or table the live code reads in the same | |
| 171 | deploy that stops reading it. | |
| 172 | ||
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 173 | #### Telling the status page about a deploy |
| 174 | ||
| 175 | Restarts during a deploy can make a part slow for a minute, which the | |
| 176 | status page's checks would otherwise draft as an incident. Before the | |
| 177 | first stage and after the last, a deploy can say so with | |
| 178 | `scripts/deploy/status-window.mjs` (`announceDeploy("started" | "finished", | |
| 179 | { id })`, or `node scripts/deploy/status-window.mjs started|finished [id]`). | |
| 180 | It posts to `POST https://status.g1t.sh/deploys` with | |
| 181 | `Authorization: Bearer $STATUS_DEPLOY_TOKEN`, the same value as the status | |
| 182 | Worker's `STATUS_DEPLOY_TOKEN` secret. During the deploy and for 3 minutes | |
| 183 | after it, detection keeps counting failed and slow checks but makes no new | |
| 184 | draft; trouble that outlasts that is drafted with its true start. A | |
| 185 | start with no finish stops counting after 30 minutes. Without the token | |
| 186 | the helper does nothing, and it never fails a deploy. | |
| 187 | ||
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 188 | On a laptop the tool uses your `wrangler login` (or `CLOUDFLARE_DEPLOY_TOKEN` |
| 189 | if set), as `scripts/deploy.sh` always did: a `CLOUDFLARE_API_TOKEN` or | |
| 190 | global API key in your shell, or in the repository's `.env`, is ignored. | |
| 191 | With `CI=true` it uses `CLOUDFLARE_API_TOKEN`. | |
| 192 | ||
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 193 | ### The runner's images |
| 194 | ||
| 195 | `services/runner` runs every sandbox (agents, checks, the merge queue, | |
| 196 | workflow jobs, g1t.page builds) from one Containers image, made in two | |
| 197 | parts: | |
| 198 | ||
| 199 | | Image | Built from | Holds | Rebuilt | | |
| 200 | | --- | --- | --- | --- | | |
| Merge branch 'main' into actions-toolkit-oidc-artifacts | 201 | | **Base**, `g1t-runner:base-<date>-<inputs>` | `services/runner/base/Dockerfile` | Debian bookworm, Node 24, Python 3.11, Docker (Engine, Buildx, Compose, from Docker's apt repository), Go (from go.dev), Rust stable for the `node` user with rustfmt, clippy and the `wasm32-unknown-unknown` target, build-essential, musl-tools, git, ripgrep, jq, zstd, sudo, and the pinned Claude Code CLI on top | When its folder changes, weekly, or by hand (`build-base`) | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 202 | | **Runner**, `g1t-runner:<content hash>` | `services/runner/Dockerfile`: `FROM` the base, plus one file | The g1t runner, a static binary | When the binary or the base changes | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 203 | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 204 | Both are pushed to one repository of Cloudflare's registry, |
| 205 | `registry.cloudflare.com/<account>/g1t-runner`, so pushing the runner's | |
| 206 | image uploads only its own layer (about 5 MB): the base's layers are | |
| 207 | already there. | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 208 | |
| The runner's base image, pushed as Wrangler builds images, with pushes that fail loudly | 209 | Both are built as Wrangler builds images (`--platform linux/amd64 |
| 210 | --provenance=false --sbom=false`): one manifest, not an OCI index with a | |
| 211 | BuildKit attestation beside it. A push is tried up to three times and counts | |
| 212 | only when Docker reports the digest; the first push of the base once failed | |
| 213 | with `blob unknown to registry` and went through when run again (see | |
| 214 | `docs/CLOUDFLARE_FEEDBACK.md`, C3). `build-base` and `image` exit non-zero | |
| 215 | when a push fails, and `base.json` is written only after the push. | |
| 216 | ||
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 217 | **The base** is recorded in `services/runner/base.json`: its reference, its |
| 218 | digest, a hash of its folder (`inputs`), when it was built, its size and | |
| 219 | each toolchain's version. `node scripts/deploy.mjs build-base` builds it, | |
| 220 | pushes it and rewrites the file; commit the file, and the next deploy | |
| 221 | builds the runner's image on it. `npm run test:deploy` fails while the | |
| 222 | folder and the file disagree, so a change to the base's Dockerfile cannot | |
| 223 | merge without the base it describes. Layers go from what changes least to | |
| 224 | most (system packages, Go, Rust, the Claude Code CLI), and the apt and npm | |
| 225 | caches stay in BuildKit's cache, out of the image. | |
| 226 | ||
| 227 | The base's build cache is the base itself: it is built with | |
| 228 | `BUILDKIT_INLINE_CACHE`, which records in the image how each layer was | |
| 229 | made, and `build-base` builds `--cache-from` the base it replaces. A | |
| 230 | machine with an empty cache, or one just pruned, pulls the unchanged | |
| 231 | layers from the registry instead of building them. (BuildKit's other | |
| 232 | registry cache, `--cache-to type=registry`, pushes a separate cache | |
| 233 | manifest that not every registry takes, and for a one-stage image adds | |
| 234 | nothing the inline cache lacks.) | |
| 235 | ||
| 236 | **The runner binary** (`crates/runner`) is built outside Docker by | |
| 237 | `scripts/build-runner.mjs` as one static binary for | |
| 238 | `x86_64-unknown-linux-musl`, so it runs on the base whatever its libc, and | |
| 239 | on a self-hosted runner's machine too. Where it is built: | |
| 240 | ||
| 241 | - on x86-64 Linux with the musl target and `musl-gcc` | |
| 242 | (`rustup target add x86_64-unknown-linux-musl`, `apt-get install musl-tools`), | |
| 243 | with the machine's own Cargo; | |
| 244 | - anywhere else (Windows, macOS) in a small builder container, Rust on | |
| 245 | Alpine (whose own target is musl), with Docker volumes keeping Cargo's | |
| 246 | registry and target directory between builds. Windows has no musl | |
| 247 | cross-linker, and `ring` (under ureq's TLS) needs a C compiler for the | |
| 248 | target, so a container is the dependable route. | |
| 249 | ||
| 250 | **The runner's image tag** is a hash of everything it is built from: its | |
| 251 | Dockerfile, `base.json`, the crates the binary is built from, the | |
| 252 | workspace's Cargo files and the build script. The same source always names | |
| 253 | the same image, so: | |
| 254 | ||
| 255 | 1. A deploy computes the tag and asks the registry whether it is there | |
| 256 | (a `HEAD` of its manifest, with credentials from Wrangler; no Docker). | |
| 257 | 2. If it is, nothing is built: the deploy uses it. | |
| Merge branch 'main' into actions-toolkit-oidc-artifacts | 258 | 3. If not, it builds the binary and the image (seconds on a warm machine) |
| 259 | and pushes it. In `deploy.yml` that is the `runner-image` job, on | |
| 260 | `g1t-4core`, with the job's own Docker Engine (see | |
| 261 | [Docker in workflow jobs](#docker-in-workflow-jobs)): it adds the musl | |
| 262 | target, builds the binary natively (the base has `musl-gcc`), pulls the | |
| 263 | base from Cloudflare's registry, builds, and pushes one layer. | |
| 264 | 4. If Docker does not answer (a machine without it, or jobs with Docker | |
| 265 | turned off), the unit fails saying to run `node scripts/deploy.mjs | |
| 266 | image` on a machine with Docker; then re-run the workflow. | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 267 | |
| 268 | Then `wrangler deploy` is given the image by reference, from a generated | |
| 269 | config (`services/runner/wrangler.deploy.json`, deleted after, ignored by | |
| 270 | git), so Wrangler builds nothing. When nothing the image is built from | |
| 271 | changed since the runner's live commit, the deploy also passes | |
| 272 | `--containers-rollout none`, which leaves running sandboxes alone. | |
| 273 | ||
| 274 | To get a new base out: | |
| 275 | ||
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 276 | ```sh |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 277 | node scripts/deploy.mjs build-base # build, push, write base.json (needs Docker) |
| 278 | git commit services/runner/base.json -m "A new base image for g1t's sandboxes" | |
| 279 | node scripts/deploy.mjs image # optional: push the runner's image now, so CI finds it | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 280 | ``` |
| 281 | ||
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 282 | `.g1t/workflows/runner-base.yml` does the same weekly (and when the |
| 283 | base's folder changes on `main`), and opens a pull request with | |
| Merge branch 'main' into actions-toolkit-oidc-artifacts | 284 | `base.json`. It runs on a self-hosted runner with the `docker` label |
| 285 | (`runs-on: [self-hosted, docker]`): g1t's own machines have Docker now, | |
| 286 | but the base's build downloads from Docker's apt repository over HTTPS, | |
| 287 | which does not trust a guarded job's egress certificate, so it stays on an | |
| Runner base workflow: skipped until a self-hosted runner exists | 288 | open network. Its job is skipped until the repository variable |
| 289 | `RUNNER_BASE_SELF_HOSTED` is `true` (set it once a runner is registered; | |
| 290 | without one, runs waited in the queue for a day and a half and then | |
| 291 | failed); until then run `build-base` by hand. | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 292 | |
| 293 | **Sandboxes start from the image.** Cloudflare pulls an image to a | |
| 294 | machine the first time a sandbox lands there, and keeps it. A smaller base | |
| 295 | pulls sooner, and a change to the runner alone sends machines one 5 MB | |
| 296 | layer instead of the whole image. | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 297 | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 298 | #### Larger machines |
| 299 | ||
| 300 | The same image runs on three instance types, each a Durable Object class | |
| 301 | of its own in `services/runner/wrangler.jsonc`: `AttemptSandbox` | |
| 302 | (`standard-1`), `Sandbox2Core` (`standard-3`) and `Sandbox4Core` | |
| 303 | (`standard-4`). Workflow jobs choose with `runs-on: g1t-2core` or | |
| 304 | `g1t-4core` (`g1t_contracts::actions::INSTANCE_TYPES`); the actions | |
| 305 | service passes the label to the runner, which starts the job in that | |
| 306 | class. Billing prices the larger ones from their memory and disk, and | |
| 307 | their CPU (see the public billing guide). The account's Containers limits | |
| 308 | must allow `standard-4`; Wrangler refuses the deploy otherwise. | |
| 309 | ||
| Merge branch 'main' into actions-toolkit-oidc-artifacts | 310 | #### Docker in workflow jobs |
| 311 | ||
| 312 | Workflow jobs on g1t's machines have a Docker Engine of their own | |
| 313 | (`crates/runner/src/docker/`; the public guide is | |
| 314 | `apps/docs/src/content/docs/guides/actions.md`, "Docker"). What Cloudflare | |
| 315 | Containers allow decides how it runs (findings in `docs/PLAN.md`, | |
| 316 | "Docker in workflow jobs"): | |
| 317 | ||
| 318 | | Piece | What it does | | |
| 319 | | --- | --- | | |
| 320 | | `dockerd` | Started as root with `sudo`, only when the job first uses Docker or has `services:` or `container:`. Flags: `--iptables=false --ip6tables=false --ip-forward=false` (Containers allow neither), the containerd image store, Docker Hub through `mirror.gcr.io`. Its config, socket and log are in `/run/g1t-docker` (`dockerd.log` is the place to look); its data in `/var/lib/docker`, on overlays when the disk takes them, plain copies (`native`) when not. | | |
| 321 | | `/var/run/docker.sock` | The runner's API proxy (`docker/api.rs`), which starts the Engine on the first connection. Containers that ask for a bridge network get the job's own (`host`), the names they would have had resolve to 127.0.0.1, and ports published under another number are forwarded. | | |
| 322 | | `runc` | The Engine finds the runner binary first on its `PATH` as `runc` (`docker/oci.rs`): BuildKit's `RUN` steps join the job's network, and in a guarded job every container gets the egress certificate at `/dev/g1t-egress`. Then the real `/usr/bin/runc` runs. | | |
| 323 | | cgroups | Before the Engine starts, the sandbox's processes move to a cgroup of their own and every controller is handed down, as Docker's own Docker-in-Docker image does, so `--cpus` and `--memory` work. | | |
| 324 | ||
| 325 | - **Turning it off:** set the runner Worker's `DOCKER` var to `off` | |
| 326 | (`services/runner/wrangler.jsonc`) and deploy the runner: new jobs get | |
| 327 | no Engine (`G1T_DOCKER=off`), and jobs that need one fail saying Docker | |
| 328 | does not answer. Anything else is on. | |
| 329 | - **Network:** containers share the job's network, so the guardrails, the | |
| 330 | workflow-only domains and the egress Worker apply to them unchanged. The | |
| 331 | public registries are in `BUILD_HOSTS` (`services/runner/src/egress.ts`). | |
| 332 | - **Isolation:** one Engine per job, inside the job's sandbox (its own | |
| 333 | VM), gone with it. The job already had root through `sudo`; Docker adds | |
| 334 | no reach beyond the sandbox, and no host socket is ever mounted into one. | |
| 335 | - **Checked locally** (2026-10-08, Docker Desktop, the base and runner | |
| 336 | image built from this tree, a privileged container standing in for a | |
| 337 | sandbox, a pretend API): services with health checks, `localhost` and | |
| 338 | names, port forwarding, `docker build` with a networked `RUN`, Compose | |
| 339 | with a healthy dependency, `docker://` steps, a Dockerfile action, a | |
| 340 | `container:` job with a JavaScript action, an Alpine job container, the | |
| 341 | egress certificate in `run`, `exec` and build steps and in no layer, | |
| 342 | plain-copy storage, and the deploy's own build and push to a registry. | |
| 343 | Not yet seen on Cloudflare itself: watch the first runs' logs for the | |
| 344 | `Docker: started` line, and `dockerd.log` if it does not come. | |
| 345 | ||
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 346 | ## Build speed |
| 347 | ||
| 348 | Measured on the development machine (Windows, 32 cores, warm Cargo cache), | |
| 349 | building `events`, `search` and `repos` after a change to `crates/kit`, as a | |
| 350 | deploy does but without uploading (`wrangler deploy --dry-run`): | |
| 351 | ||
| 352 | | | Time | | |
| 353 | | --- | --- | | |
| 354 | | Before: one after another, `cargo install worker-build` each time, wasm-opt `-O` | 81 s, 84 s | | |
| 355 | | Concurrent builds, wasm-opt `-O` | 50 s, 68 s | | |
| 356 | | Concurrent builds, wasm-opt `-O1` (now) | 28 s, 33 s | | |
| 357 | ||
| 358 | Where the time went, and what changed: | |
| 359 | ||
| 360 | - **wasm-opt** was most of it: `-O` took 38 s on `repos` and 19 s on `api`; | |
| 361 | `-O1` takes 1 to 3 s. With worker-build's flags (it keeps the names | |
| 362 | section) the `.wasm` is 24 to 28% larger raw but only 1 to 7% larger | |
| 363 | gzipped, and Workers' size limit is on the compressed upload. `-Os` and | |
| 364 | `-Oz` were no faster than `-O`. Set per crate in | |
| 365 | `[package.metadata.wasm-pack.profile.release]`; a test keeps every Rust | |
| 366 | unit on the same level. | |
| 367 | - **worker-build** is installed only when missing or another version | |
| 368 | (`scripts/build-rust-worker.mjs`, which pins it). Locally `cargo install` | |
| 369 | on an installed version cost under a second; on a fresh CI sandbox it is | |
| 370 | a full compile, which the workflow caches instead. | |
| 371 | - **One Cargo target**: every Rust unit is a member of the workspace, so | |
| 372 | they already share `target/`. Concurrent builds take turns on Cargo's | |
| 373 | lock for the compile, and their wasm-bindgen, wasm-opt and uploads | |
| 374 | overlap. | |
| 375 | - **No joint `cargo build -p a -p b`**: tried, and it is slower. Cargo | |
| 376 | unifies features across the packages of one build (`serde_json`'s | |
| 377 | `preserve_order` from `api` and `actions`, `digest` features from | |
| 378 | `secrets`), so each worker-build afterwards compiled its own variant | |
| 379 | again. | |
| The runner's base image, pushed as Wrangler builds images, with pushes that fail loudly | 380 | - **The runner's images**, measured on the same machine on 2026-10-06 |
| 381 | (Docker Desktop, 8 vCPUs; its disk was busy with other containers, so | |
| 382 | the cold figures are slow and noisy): | |
| 383 | ||
| 384 | | | Before (one image) | Now | | |
| 385 | | --- | --- | --- | | |
| 386 | | Size, unpacked / compressed (what a machine pulls) | 3.08 GB / 819 MB | 2.68 GB / 686 MB | | |
| 387 | | A change to the runner | Docker rebuilds the image's Rust stage and pushes the image (1198 s in the first deploy after a prune) | binary 46–53 s (6 s unchanged), image 7 s, push one 5 MB layer (1.4 s to a local registry) | | |
| 388 | | The base from nothing | 431 s (whole image, cold) | 758 s cold, rarely: weekly or when its folder changes | | |
| 389 | | The base after `docker builder prune` | as from nothing | 68 s, its layers pulled from the registry it was pushed to | | |
| 390 | | The runner binary, cold (builder container) | | 99 s | | |
| 391 | | A Rust CI job's build (events, search, repos; 4 vCPUs) | | 51 s cold, 13 s with the Cargo target restored (107 MB zstd entry) | | |
| 392 | ||
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 393 | - **Only what changed** is the largest saving: a change to one service |
| 394 | deploys one service. | |
| 395 | ||
| 396 | ## The workflow | |
| 397 | ||
| 398 | `.g1t/workflows/deploy.yml` runs on every push to `main`, and by hand | |
| 399 | (**Actions → Deploy → Run workflow**) with `units` (deploy these, changed or | |
| 400 | not), `all` and `dry_run` (plan only). | |
| 401 | ||
| 402 | | Job | Does | Needs | | |
| 403 | | --- | --- | --- | | |
| 404 | | `check` | `manifest --check` and `npm run test:deploy` | — | | |
| 405 | | `plan` | `plan --github-output`: outputs per stage, the plan in the run's summary | `check` | | |
| 406 | | `migrate` | `migrate --only <units with pending migrations>` | `plan`; skipped when none are pending | | |
| 407 | | `core`, `edge`, `front` | `deploy --only <units> --force --no-migrations`, one job per build group | the stages before; skipped when empty | | |
| 408 | ||
| 409 | - **One at a time:** `concurrency: deploy-production`, never cancelled in | |
| 410 | progress; a second push waits. | |
| 411 | - **Build groups:** a stage's units are split so each job shares a build: | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 412 | Rust workers at most four to a job (each a 4-vCPU `g1t-4core` machine), the |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 413 | TypeScript Workers together, each site alone, and a unit whose image must |
| Merge branch 'main' into actions-toolkit-oidc-artifacts | 414 | be rebuilt alone (`image: true` in the matrix, also on `g1t-4core`, where |
| 415 | it builds and pushes the image with the job's own Docker Engine). `fail-fast: false`, so one failed job does not cut | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 416 | another off mid-upload; the next stage then does not start. |
| 417 | - **Tests:** there is no CI workflow on g1t yet; `main` is kept passing by | |
| 418 | the merge queue's checks. `check` runs the deploy tool's own tests. When a | |
| 419 | CI workflow is added, make `plan` wait for it (`workflow_run`, or a job in | |
| 420 | this file). | |
| Merge branch 'main' into actions-toolkit-oidc-artifacts | 421 | - **Machines:** Rust jobs and the runner's image run on `g1t-4core` (4 vCPUs, |
| 422 | 12 GiB, 20 GB), the others on the standard machine | |
| 423 | (`runs-on: ${{ (matrix.rust || matrix.image) && 'g1t-4core' || 'ubuntu-latest' }}`). | |
| 424 | The image job needs the room: the base it builds on is about 3.2 GB | |
| 425 | unpacked. | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 426 | - **Caching** (`actions/cache`: up to 2 GB an entry, 10 GB a repository, |
| 427 | kept until unused for 7 days): the worker-build binary, worker-build's | |
| 428 | downloaded tools, `~/.cargo/registry/cache`, and the Cargo target's | |
| 429 | release dependencies (`target/release` and | |
| 430 | `target/wasm32-unknown-unknown/release`, without `incremental` or | |
| 431 | `.wasm`), keyed by the build group, `Cargo.lock` and `base.json`. The | |
| 432 | workspace's own crates are compiled again on every run (a checkout's | |
| 433 | sources are newer than any cache); the crates.io dependencies are not. | |
| 434 | npm's cache is not kept: every job runs `npm ci` of only what its units | |
| 435 | need (`deploy.mjs install`: Wrangler alone for Rust jobs). | |
| 436 | - **Conditions:** each stage runs with `!failure() && !cancelled()`, which | |
| 437 | on g1t (as on GitHub) is true when no job before it failed, however far | |
| 438 | back: a `migrate` job skipped for having nothing to apply does not stop | |
| 439 | the stages after it, and a failed `check` stops all of them. | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 440 | - `crates/actions/tests/repository_workflows.rs` reads the workflow with |
| 441 | g1t's own parser and expressions, and checks the jobs start, wait and | |
| 442 | stop as above (`cargo test -p g1t-actions --test repository_workflows`). | |
| 443 | ||
| 444 | ### What the sandbox has | |
| 445 | ||
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 446 | The base image (`services/runner/base/Dockerfile`) has Node 24, npm, git, |
| Merge branch 'main' into actions-toolkit-oidc-artifacts | 447 | Go, zstd, Docker, musl-tools, and Rust stable for the `node` user with |
| 448 | rustfmt, clippy and the `wasm32-unknown-unknown` target, but not | |
| 449 | worker-build. The workflow's `rustup target add wasm32-unknown-unknown` is | |
| 450 | then a no-op, and worker-build is restored from the cache, installed on a | |
| 451 | miss. The image job adds `x86_64-unknown-linux-musl` (about 30 MB from | |
| 452 | `static.rust-lang.org`) and keeps its Cargo target in the cache. worker-build | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 453 | fetches wasm-bindgen and wasm-opt from GitHub releases and esbuild from |
| 454 | npm. All of those hosts are on the list every workflow job may reach. | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 455 | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 456 | ### Network |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 457 | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 458 | A workflow job reaches its project's allowed domains, g1t, what builds |
| 459 | need (`services/runner/src/egress.ts`, `BUILD_HOSTS`), and the project's | |
| 460 | **workflow-only domains** that name its workflow and environment. Those are | |
| 461 | never reached by agents, checks, the merge queue, deploy builds or runs of | |
| 462 | pull requests from forks (`Guardrails::workflow_hosts`, the runner's | |
| 463 | `jobHosts`). Under flagon-io/g1t's **Settings → Guardrails** | |
| 464 | (Maintain role or higher), **Workflow-only domains**: | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 465 | |
| The runner's base image, pushed as Wrangler builds images, with pushes that fail loudly | 466 | ```text |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 467 | api.cloudflare.com | deploy.yml | production |
| 468 | registry.cloudflare.com | deploy.yml, runner-base.yml | production | |
| 469 | ``` | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 470 | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 471 | `api.cloudflare.com` is Wrangler's API; `registry.cloudflare.com` is where |
| Merge branch 'main' into actions-toolkit-oidc-artifacts | 472 | the deploy asks whether the runner's image is already built, and where the |
| 473 | `runner-image` job pulls the base from and pushes the runner's image to | |
| 474 | (as `runner-base.yml` pushes the base). The job's Docker Engine shares the | |
| 475 | job's network, so these lines are what let it reach the registry. If a pull | |
| 476 | is refused with `g1t guardrails: <host> is not on this project's allowed | |
| 477 | domains`, the registry sent the layers from another host: add that host on | |
| 478 | the same line. Only `deploy.yml`'s and `runner-base.yml`'s | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 479 | jobs with `environment: production` reach them, which are also the only |
| 480 | jobs that can read `CLOUDFLARE_API_TOKEN`. Each change to the list is in | |
| 481 | the workspace's audit log as `update_guardrails`. | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 482 | |
| 483 | ### The API token | |
| 484 | ||
| 485 | Create it at **dash.cloudflare.com → My Profile → API Tokens → Create | |
| 486 | Token → Custom token**, named `g1t deploys (CI)`: | |
| 487 | ||
| 488 | | Scope | Permission | Why | | |
| 489 | | --- | --- | --- | | |
| 490 | | Account | Workers Scripts: Edit | Upload, versions, deployments, crons, bindings, `secret list` (doctor) | | |
| 491 | | Account | D1: Edit | `d1 migrations list` and `apply` | | |
| 492 | | Account | Queues: Edit | Attaching each unit's queue consumers on deploy | | |
| 493 | | Account | Workers R2 Storage: Read | Wrangler checks `og`'s bucket binding | | |
| 494 | | Account | Account Settings: Read | Wrangler reads the account | | |
| Merge branch 'main' into actions-toolkit-oidc-artifacts | 495 | | Account | Containers: Edit | The runner's deploy updates its applications (the image reference, the three classes), and gets registry credentials (`wrangler containers registries credentials --push`, one hour) to look for, pull and push its image. `runner-base.yml` pushes images with it. | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 496 | | Zone (`g1t.sh`, `g1t.page`) | Workers Routes: Edit | `pages`' zone routes, and custom domains | |
| 497 | | Zone (`g1t.sh`, `g1t.page`) | DNS: Edit | Custom domains (`api`, `mcp`, `og`, `models`, `status`, `sudo`, `docs`, `g1t.sh`, `g1t.page`) keep their DNS records | | |
| 498 | | Zone (`g1t.sh`, `g1t.page`) | Zone: Read | Finding the zone a route names | | |
| 499 | ||
| 500 | Restrict it to account `syntaqx` (`1e6f2cffa3f445920836e8ebe446bb58`) and | |
| 501 | the two zones. Not needed for deploys: KV (bindings are by id; creating a | |
| 502 | namespace is a one-time setup), Vectorize, Workers for Platforms beyond | |
| 503 | Workers Scripts, Cloudflare for SaaS custom hostnames (the deployments | |
| 504 | service does that at runtime with its own token), SSL and Certificates. | |
| 505 | Workers KV Storage: Edit and Vectorize: Edit are only for first-time setup, | |
| 506 | which stays a person's job. | |
| 507 | ||
| 508 | The list follows what our configs use; Cloudflare does not publish exactly | |
| 509 | what `wrangler deploy` checks for each binding. Bindings with no listed | |
| 510 | permission (Browser Rendering, Workers AI, Vectorize, Email Sending, | |
| 511 | Artifacts, dispatch namespaces) are assumed to need none beyond Workers | |
| 512 | Scripts. Verify on the first run with `workflow_dispatch` and `units: | |
| 513 | pages` (small, no secrets), then `units: og` (R2) and `units: runner` | |
| 514 | (Containers); a missing permission fails with `Authentication error [code: | |
| 515 | 10000]` and the route it was refused. | |
| 516 | ||
| 517 | Add it to the repository: | |
| 518 | ||
| 519 | 1. On g1t.sh, open **flagon-io/g1t → Settings → Secrets and variables**. | |
| 520 | 2. **Add**: key `CLOUDFLARE_API_TOKEN`, type **Secret**, available to | |
| 521 | **Workflows**, environment **Production**. Only jobs with | |
| 522 | `environment: production` (the deploy jobs) can read it. | |
| 523 | 3. **Add**: key `CLOUDFLARE_ACCOUNT_ID`, type **Variable**, value | |
| 524 | `1e6f2cffa3f445920836e8ebe446bb58`, available to **Workflows**, all | |
| 525 | environments. | |
| 526 | ||
| 527 | Or through the API: | |
| 528 | ||
| 529 | ```sh | |
| 530 | curl -X PUT https://api.g1t.sh/repos/flagon-io/g1t/actions/secrets/CLOUDFLARE_API_TOKEN \ | |
| 531 | -H "Authorization: Bearer $G1T_TOKEN" -H "Content-Type: application/json" \ | |
| 532 | -d '{"value":"<the token>","environments":["production"],"available_to":["workflows"]}' | |
| 533 | ||
| 534 | curl -X POST https://api.g1t.sh/repos/flagon-io/g1t/actions/variables \ | |
| 535 | -H "Authorization: Bearer $G1T_TOKEN" -H "Content-Type: application/json" \ | |
| 536 | -d '{"name":"CLOUDFLARE_ACCOUNT_ID","value":"1e6f2cffa3f445920836e8ebe446bb58","available_to":["workflows"]}' | |
| 537 | ``` | |
| 538 | ||
| 539 | ## Turning it on | |
| 540 | ||
| 541 | 1. Create the token and add the secret and variable (above). | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 542 | 2. Add `api.cloudflare.com` and `registry.cloudflare.com` to flagon-io/g1t's |
| 543 | workflow-only domains, for `deploy.yml` in `production` (above). | |
| 544 | 3. Build and push the base once, and commit `services/runner/base.json`: | |
| 545 | `node scripts/deploy.mjs build-base`. Create the cache bucket: | |
| Actions: OIDC tokens, the toolkit's cache and artifact services, and artifacts in R2 | 546 | `npx wrangler r2 bucket create g1t-actions-cache`, with lifecycle rules |
| 547 | deleting cache entries (`c/`) 30 days after upload and artifacts (`a/`) | |
| 548 | after 91, a day past the longest they are kept, and unfinished uploads | |
| 549 | after a day (the actions service deletes both sooner; the rules catch | |
| 550 | what it misses): | |
| 551 | `npx wrangler r2 bucket lifecycle add g1t-actions-cache expire-cache c/ --expire-days 30 --abort-multipart-days 1` | |
| 552 | and `npx wrangler r2 bucket lifecycle add g1t-actions-cache expire-artifacts a/ --expire-days 91 --abort-multipart-days 1`. | |
| 553 | A bucket made before artifacts moved there has one rule for everything, | |
| 554 | `expire`, which would delete artifacts kept longer than 30 days: remove | |
| 555 | it (`npx wrangler r2 bucket lifecycle remove g1t-actions-cache --id expire`) | |
| 556 | and add the two above. | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 557 | 4. Adopt the live Workers once, from a laptop: deploy everything with the |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 558 | tool so each version records its commit (`scripts/deploy.sh`, or |
| 559 | `node scripts/deploy.mjs deploy --all`). Until then every plan says "no | |
| 560 | known commit" and deploys every unit. To see what has changed since a | |
| 561 | commit you know production runs, without deploying: | |
| 562 | `node scripts/deploy.mjs plan --since <sha>`. | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 563 | 5. Run the workflow by hand with `dry_run`, then with `units: pages`. |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 564 | |
| 565 | ## First deploy of a new account | |
| 566 | ||
| 567 | What the configs refer to must exist first. `node scripts/deploy.mjs | |
| 568 | manifest --json` lists, per unit, its queues, KV, R2, Vectorize and | |
| 569 | dispatch namespaces; each unit's `setup` and `secrets` say the rest. | |
| 570 | ||
| 571 | - D1: `npx wrangler d1 create <database>`, then put its id in the unit's | |
| 572 | `wrangler.jsonc`. | |
| 573 | - KV: `npx wrangler kv namespace create <name>` for each name under | |
| 574 | `resources.kv`, then the ids in the configs. | |
| 575 | - Queues: `npx wrangler queues create <queue>` for each queue in the | |
| 576 | manifest: `g1t-events`, `g1t-events-<service>` for every subscriber, | |
| 577 | `g1t-search-jobs`, `g1t-context-jobs`. | |
| Merge branch 'worktree-agent-a1b995daa94e4e1b7' | 578 | - R2: `npx wrangler r2 bucket create g1t-screenshots`, |
| Actions: OIDC tokens, the toolkit's cache and artifact services, and artifacts in R2 | 579 | `npx wrangler r2 bucket create g1t-actions-cache` (with its two |
| 580 | lifecycle rules, above), and `npx wrangler r2 bucket create g1t-git-packs`, | |
| Merge branch 'worktree-agent-a1b995daa94e4e1b7' | 581 | the clone pack cache (`services/repos/src/pack_cache.rs`), with a rule |
| 582 | deleting packs 7 days after they were written and unfinished uploads | |
| 583 | after a day: | |
| 584 | `npx wrangler r2 bucket lifecycle add g1t-git-packs expire-packs packs/ --expire-days 7 --abort-multipart-days 1`. | |
| 585 | A Worker bound to a bucket that does not exist fails to deploy, so make | |
| 586 | it before the first deploy of `g1t-repos` that binds it. | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 587 | - The runner's base image: `node scripts/deploy.mjs build-base`. |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 588 | - Vectorize, dispatch namespace, DNS, Access, Email Sending, Artifacts: each |
| 589 | unit's `setup`. | |
| 590 | - Secrets: `npx wrangler secret put <NAME>` in the unit's folder; | |
| 591 | `node scripts/deploy.mjs doctor` lists what is missing. | |
| 592 | ||
| 593 | Then `node scripts/deploy.mjs deploy --all`. A Worker bound to a service | |
| 594 | that does not exist yet may be refused; deploy that service first with | |
| 595 | `--only`. | |
| 596 | ||
| Actions: OIDC tokens, the toolkit's cache and artifact services, and artifacts in R2 | 597 | ## OIDC tokens for workflow jobs |
| 598 | ||
| 599 | The API is the issuer of workflow jobs' OIDC tokens, | |
| 600 | `https://api.g1t.sh/actions/oidc` (`apps/api/src/oidc.rs`): no host or DNS | |
| 601 | of its own. It signs with an RSA key kept as the API's secret | |
| 602 | `ACTIONS_OIDC_KEY`. Without it, the issuer's addresses answer 404 and jobs | |
| 603 | are not told where to ask for a token. | |
| 604 | ||
| 605 | To turn it on, make a key on a trusted machine and store it, then delete | |
| 606 | the file: | |
| 607 | ||
| 608 | ```sh | |
| 609 | openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out oidc.pem | |
| 610 | cd apps/api && npx wrangler secret put ACTIONS_OIDC_KEY < ../../oidc.pem | |
| 611 | rm ../../oidc.pem | |
| 612 | ``` | |
| 613 | ||
| 614 | Check `https://api.g1t.sh/actions/oidc/.well-known/jwks` lists one key. | |
| 615 | Its `kid` is the key's RFC 7638 thumbprint. | |
| 616 | ||
| 617 | To rotate it, keep the old key published while tokens it signed can still | |
| 618 | be presented (they last 5 minutes; relying parties cache keys for longer): | |
| 619 | ||
| 620 | 1. Store the current key as `ACTIONS_OIDC_KEY_PREVIOUS` (the same PEM). | |
| 621 | 2. Make a new key and store it as `ACTIONS_OIDC_KEY`. The JWKS now lists | |
| 622 | both; new tokens are signed with the new one. | |
| 623 | 3. A day later, `npx wrangler secret delete ACTIONS_OIDC_KEY_PREVIOUS`. | |
| 624 | ||
| 625 | A key that may have leaked is rotated the same way, skipping the first | |
| 626 | step, so that tokens it signed stop verifying at once. | |
| 627 | ||
| Deploying: a deploy by hand shows on the repository's Deployments page | 628 | ## Deployments on g1t |
| 629 | ||
| 630 | Every deploy shows on the repository's Deployments page, so its production | |
| 631 | card says which commit runs. | |
| 632 | ||
| 633 | - **From the workflow**, the deploy jobs name `environment: {name: | |
| 634 | production, url: https://g1t.sh}`, and g1t Actions records one production | |
| 635 | deployment per run. | |
| 636 | - **By hand**, `deploy` reports one itself: in progress once migrations are | |
| 637 | in, then success or failure. It needs a g1t token with `deployments:write` | |
| 638 | in `G1T_DEPLOY_TOKEN` or the file `.credentials/g1t-deploy-token`; without | |
| 639 | one, or from a dirty tree or a dry run, it sends nothing. It finds the | |
| 640 | repository from the git remote on g1t.sh. A report that fails is one line | |
| 641 | in the log and never fails the deploy. | |
| 642 | ||
| Merge branch 'main' into actions-toolkit-oidc-artifacts | 643 | When CI cannot finish a deploy, for example a runner image that will not |
| 644 | build there, deploy from a machine with Docker. | |
| Deploying: a deploy by hand shows on the repository's Deployments page | 645 | The report records it, and production shows the commit that really runs. |
| 646 | ||
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 647 | ## Rolling back |
| 648 | ||
| 649 | - **One unit, at once:** `npx wrangler rollback` in its folder (or | |
| 650 | `npx wrangler rollback <version-id>`; `npx wrangler versions list` shows | |
| 651 | each version's commit in its message). Code only: D1 migrations are not | |
| 652 | undone. The next plan sees the older commit and deploys what changed since, | |
| 653 | so revert the commit on `main` too, or the next push brings it back. | |
| 654 | - **To a commit:** check it out and `node scripts/deploy.mjs deploy --only | |
| Deploying: never roll production back by accident | 655 | <units> --rollback`. Without `--rollback` the tool refuses any unit whose |
| 656 | live commit is newer than the one checked out, even with `--force`, so a | |
| 657 | re-run of an old workflow run (or an old checkout) never rolls production | |
| 658 | back by accident. Migrations never run backwards: a migration that needs | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 659 | undoing is a new migration. |
| 660 | - **The runner's image:** a rollback of the Worker does not roll back the | |
| Deploying: never roll production back by accident | 661 | container image. Redeploy the older commit (`--only runner --rollback`): its |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 662 | image's tag is the hash of that commit's source, which is still in the |
| 663 | registry, so nothing is built. A bad base is undone by reverting the | |
| 664 | commit that changed `services/runner/base.json`. | |
| Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow | 665 | |
| 666 | ## Adding a unit | |
| 667 | ||
| 668 | 1. Make its folder with a `wrangler.jsonc` (and its D1 migrations, if any). | |
| 669 | A Rust Worker is a workspace member in the root `Cargo.toml` with | |
| 670 | `"build": { "command": "node ../../scripts/build-rust-worker.mjs" }` and | |
| 671 | the `wasm-opt = ["-O1"]` metadata; a TypeScript one is an npm workspace. | |
| 672 | 2. Add it to `deploy/stack.jsonc`: path, kind, worker, `d1`, stage (the | |
| 673 | earliest stage after everything it binds to), secrets, setup, self_host. | |
| 674 | Name any new KV id under `resources.kv`. | |
| 675 | 3. If its sources import a file outside its folder that is not a workspace | |
| 676 | crate or package, list it under `inputs`. | |
| 677 | 4. Add a row to the service table in `docs/SELF_HOSTING.md`. | |
| 678 | 5. `npm run test:deploy` and `node scripts/deploy.mjs manifest --check` | |
| 679 | say what is missing. Then create its resources and secrets, and | |
| 680 | `node scripts/deploy.mjs deploy --only <unit>`. | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 681 | |
| 682 | ## The self-hosted runner | |
| 683 | ||
| 684 | `g1t-runner` (crates/runner) is also what customers run on their own | |
| 685 | machines (guide: `apps/docs/src/content/docs/guides/self-hosted-runners.md`). | |
| 686 | It is not deployed with the stack: it is released, and runners already out | |
| 687 | there update themselves to each release. | |
| 688 | ||
| 689 | | Piece | Where | | |
| 690 | | --- | --- | | |
| 691 | | The tool | `scripts/runner-release.mjs` (`keygen`, `build`, `sign`, `verify`, `publish`) | | |
| 692 | | The workflow | `.g1t/workflows/runner-release.yml`, on a tag `runner-v<version>` or by hand | | |
| 693 | | Where it is published | The R2 bucket `g1t-downloads`, served by the site at `g1t.sh/downloads/runner/<version>/<file>` and `/latest/<file>` (`apps/web/app/routes/downloads-runner.ts`) | | |
| The runner's image is g1t.sh/flagon-io/g1t-runner, on g1t's own registry, public | 694 | | Its image | `deploy/runner/Dockerfile`, pushed to `g1t.sh/flagon-io/g1t-runner` (public) for amd64 and arm64 | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 695 | |
| 696 | A release is five binaries (Linux x64 and arm64, both static musl; macOS | |
| 697 | x64 and arm64; Windows x64), `SHA256SUMS`, and `manifest.json`; | |
| 698 | `latest.json` and its Ed25519 signature `latest.json.sig` name the newest. | |
| 699 | A runner updates only to a release whose signature checks out against the | |
| 700 | public key built into it and whose download matches the manifest's SHA-256. | |
| 701 | ||
| 702 | The first time: | |
| 703 | ||
| 704 | 1. `node scripts/runner-release.mjs keygen`. Put `RUNNER_RELEASE_KEY` in the | |
| 705 | repository's secrets (production environment) and keep a copy offline; | |
| g1t-runner 0.1.0 is released: signed binaries for five platforms at g1t.sh/downloads/runner; the Artifacts checks' results | 706 | put the public key in its variables as `RUNNER_RELEASE_PUBLIC_KEY` (names |
| 707 | starting `G1T_` are reserved; the workflow hands it to the build as | |
| 708 | `G1T_RUNNER_RELEASE_KEY`). A build made without the public key never | |
| 709 | updates itself. | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 710 | 2. `npx wrangler r2 bucket create g1t-downloads`, and deploy the site so it |
| 711 | has the `DOWNLOADS` binding. | |
| The runner's image is g1t.sh/flagon-io/g1t-runner, on g1t's own registry, public | 712 | 3. The image job pushes to g1t's own registry with the run's `G1T_TOKEN`; |
| 713 | the `g1t-runner` package in flagon-io is public. Set the variable | |
| 714 | `RUNNER_AGENT_IMAGE` (a public copy of `g1t-runner-base`, the image agent | |
| 715 | work runs in on customers' runners) once there is one. | |
| Fast pages, required checks on the branch, self-hosted runners, honest incidents | 716 | 4. Register a self-hosted runner with the `docker` label for the image job. |
| 717 | ||
| 718 | Each release: | |
| 719 | ||
| 720 | 1. Bump `version` in `crates/runner/Cargo.toml` and merge it. | |
| 721 | 2. Tag the commit `runner-v<version>` and push the tag. The workflow builds | |
| 722 | every platform with cargo-zigbuild, signs, verifies, publishes the files | |
| 723 | (the version's first, `latest.json` last), and pushes the image. | |
| 724 | 3. By hand, the same is `node scripts/runner-release.mjs build`, then `sign`, | |
| 725 | `verify` and `publish`, with the keys in the environment. | |
| 726 | ||
| 727 | Rolling back a release: copy the older version's `manifest.json` over | |
| 728 | `runner/latest.json` and sign it again (`sign` after checking the older | |
| 729 | version out). Runners never move to an older version on their own; a | |
| 730 | runner on a bad release is fixed by the next good one, or by downloading | |
| 731 | the older binary over it. |
This file's history is long; its oldest lines are credited to the oldest commit read.