| 1 | # Deploying g1t |
| 2 | |
| 3 | How g1t.sh gets to Cloudflare: one manifest that lists every deployable |
| 4 | unit, one tool that deploys only what changed, and a g1t Actions workflow |
| 5 | that runs that tool on every push to `main`. Internal: the public |
| 6 | self-hosting guide is `apps/docs/src/content/docs/guides/self-hosting.md`. |
| 7 | |
| 8 | | Piece | Where | |
| 9 | | --- | --- | |
| 10 | | The manifest | `deploy/stack.jsonc` | |
| 11 | | The tool | `scripts/deploy.mjs` (library and tests in `scripts/deploy/`) | |
| 12 | | Rust Worker builds | `scripts/build-rust-worker.mjs`, every Rust unit's build command | |
| 13 | | The runner's images | `services/runner/base/Dockerfile`, `services/runner/Dockerfile`, `services/runner/base.json`, `scripts/build-runner.mjs`, `scripts/deploy/image.mjs` | |
| 14 | | The workflows | `.g1t/workflows/deploy.yml`, `.g1t/workflows/runner-base.yml` | |
| 15 | | The old entry point | `scripts/deploy.sh`, now a wrapper | |
| 16 | | Spend guardrails (platform pause, hourly usage watch) | [SPEND-GUARDRAILS.md](SPEND-GUARDRAILS.md) | |
| 17 | |
| 18 | ## The manifest |
| 19 | |
| 20 | `deploy/stack.jsonc` names every unit: each folder with a `wrangler.jsonc`. |
| 21 | It is JSONC rather than TOML so that Node reads it with no dependency, |
| 22 | with the same parser as the Wrangler configs, and comments stay possible. |
| 23 | |
| 24 | | Field | | |
| 25 | | --- | --- | |
| 26 | | `path` | The unit's folder. | |
| 27 | | `kind` | `rust-worker` (built by worker-build), `ts-worker` (Wrangler bundles it), `react-router` (`vite build` first), `astro` (`astro build` first). | |
| 28 | | `worker` | The Worker's name. Must match its `wrangler.jsonc`. | |
| 29 | | `d1` | `{ database, migrations }`, when it has a database. Must match its `wrangler.jsonc`. | |
| 30 | | `stage` | `core`, `edge` or `front` (below). | |
| 31 | | `secrets` | The Wrangler secrets it needs, by name. `node scripts/deploy.mjs doctor` checks they are set. | |
| 32 | | `setup` | One-time steps no config can say, for a first deploy. | |
| 33 | | `inputs` | Files outside its folder it is built from that no workspace metadata names. A test finds such imports. | |
| 34 | | `image` | A Containers image: `dockerfile` (the image deployed), `crate` (the binary it adds), `base` (`{ context, lock }`: the base image's folder and the file recording the base that was pushed) and `repository` (where both are pushed). See [the runner's images](#the-runners-images). | |
| 35 | | `self_host` | `run`, `off`, `separate` or `none`: what `deploy/self-host/configs.mjs` does with it. | |
| 36 | |
| 37 | What a unit is **built from** is never listed by hand. The tool reads it: |
| 38 | Rust path dependencies from `cargo metadata`, workspace packages from each |
| 39 | `package.json`, closed transitively. `node scripts/deploy.mjs manifest` |
| 40 | prints the result: |
| 41 | |
| 42 | ``` |
| 43 | unit stage kind worker d1 built from (besides its folder) |
| 44 | events core rust-worker g1t-events g1t-events crates/contracts crates/kit |
| 45 | repos core rust-worker g1t-repos g1t-repos crates/contracts crates/kit crates/scan crates/secrets |
| 46 | runner core ts-worker g1t-runner crates/actions crates/runner packages/contracts |
| 47 | og core ts-worker g1t-og packages/contracts |
| 48 | web front react-router g1t packages/contracts packages/theme |
| 49 | ... |
| 50 | ``` |
| 51 | |
| 52 | Root files count too: `Cargo.toml`, `Cargo.lock` and |
| 53 | `scripts/build-rust-worker.mjs` for Rust units; `package-lock.json` and |
| 54 | `tsconfig.base.json` for the others; `package.json` for all. A lockfile |
| 55 | change counts for a unit only if a package in that unit's graph changed, |
| 56 | read from the lockfile itself (`scripts/deploy/lockfiles.mjs`), so bumping |
| 57 | `sharp` for the docs does not redeploy the Rust services. |
| 58 | |
| 59 | ### Stages |
| 60 | |
| 61 | | Stage | What | Units | |
| 62 | | --- | --- | --- | |
| 63 | | `migrations` | Every pending D1 migration, in parallel, before any code | each unit's `d1` | |
| 64 | | `core` | Services reached through bindings | the services, `og` | |
| 65 | | `edge` | Public endpoints other than the site | `api`, `models`, `pages`, `status` | |
| 66 | | `front` | The site, sudo, the docs | `web`, `sudo`, `docs` | |
| 67 | |
| 68 | A stage starts only when the one before it succeeded. Inside a stage units |
| 69 | deploy in parallel. The rule the tests enforce: **a unit binds only to units |
| 70 | in its own stage or an earlier one**, so new code never calls a service |
| 71 | that has not shipped. (Services in `core` bind to each other in cycles, |
| 72 | which is why they share a stage.) |
| 73 | |
| 74 | ### What reads the manifest |
| 75 | |
| 76 | - `scripts/deploy.mjs`: everything below. |
| 77 | - `deploy/self-host/configs.mjs`: which Workers a self-hosted installation |
| 78 | runs (`self_host: "run"`, the site first) and which are bound to the off |
| 79 | Worker (`"off"`). Its output is identical to before, apart from the order |
| 80 | of `workers.txt` after the site. |
| 81 | - Tests (`npm run test:deploy`) check that: every `wrangler.jsonc` in the |
| 82 | repository has a unit; `worker`, `d1` and `image` match the configs; |
| 83 | every KV id is named under `resources.kv`; stages follow bindings; the |
| 84 | derived dependencies agree with Cargo's own resolved graph; every import |
| 85 | that leaves a unit's folder is covered; `deploy/self-host/Dockerfile` |
| 86 | builds exactly the Rust units self-hosting runs; and the service table in |
| 87 | `docs/SELF_HOSTING.md` names every unit. |
| 88 | |
| 89 | ## The tool |
| 90 | |
| 91 | ```sh |
| 92 | node scripts/deploy.mjs plan # what would deploy, and why (read-only) |
| 93 | node scripts/deploy.mjs deploy # migrations, then every changed unit |
| 94 | node scripts/deploy.mjs deploy --only web,api # just these, if they changed |
| 95 | node scripts/deploy.mjs deploy --only web --force # just this, changed or not |
| 96 | node scripts/deploy.mjs deploy --all # everything |
| 97 | node scripts/deploy.mjs build --only events # build as a deploy would; upload nothing |
| 98 | node scripts/deploy.mjs migrate # pending migrations only |
| 99 | node scripts/deploy.mjs manifest [--check|--json] |
| 100 | node scripts/deploy.mjs doctor # secrets each unit lacks |
| 101 | node scripts/deploy.mjs build-base # build and push the runner's base image (Docker) |
| 102 | node scripts/deploy.mjs image # build and push the runner's image for this checkout (Docker) |
| 103 | scripts/deploy.sh [units...] # the old entry point: all, or those named, always |
| 104 | ``` |
| 105 | |
| 106 | | Flag | | |
| 107 | | --- | --- | |
| 108 | | `--only a,b` / `--skip a,b` | Units by short name, folder or Worker name. | |
| 109 | | `--all` | Every unit, changed or not. | |
| 110 | | `--force` | Deploy the selected units even if unchanged. | |
| 111 | | `--concurrency N` | Units at once inside a stage, and migrations at once (default 4). | |
| 112 | | `--stage core` | One stage only. | |
| 113 | | `--no-migrations` | Skip the migrations step (the workflow runs it as its own job). | |
| 114 | | `--allow-dirty` | Deploy with uncommitted changes in what deploys. The version records no commit, so the next plan deploys it again. | |
| 115 | | `--rebuild-image` | Build the runner's image even if one for this source is already in the registry. | |
| 116 | | `--rebuild-base` | Build and push a new base first (needs Docker), then the runner's image on it. Writes `services/runner/base.json`: commit it. | |
| 117 | | `--no-push` | `build-base` and `image`: build locally, push nothing. | |
| 118 | | `--no-cache` | `build-base`: build every layer again. | |
| 119 | | `--since REV` | Treat Workers with no recorded commit as running `REV`. Used once to adopt Workers deployed before this tool. | |
| 120 | | `--json`, `--out FILE`, `--github-output` | The plan as data, for the workflow. | |
| 121 | |
| 122 | ### Where the deployed commit is kept |
| 123 | |
| 124 | On the Worker itself. Every deploy runs `wrangler deploy --message |
| 125 | "g1t-deploy <40-char sha> <subject>" --tag g1t-<12-char sha>`, which |
| 126 | Cloudflare keeps as the version's `workers/message` and `workers/tag` |
| 127 | annotations. `plan` reads them back with two read-only requests per unit |
| 128 | to Cloudflare's API, the ones `wrangler deployments status` and `wrangler |
| 129 | versions list` make: `GET /accounts/{account_id}/workers/scripts/{worker}/deployments` |
| 130 | (the live version) and `GET .../versions?deployable=true` (its |
| 131 | annotations). Pending migrations are one more per database: `POST |
| 132 | /accounts/{account_id}/d1/database/{database_id}/query` with `SELECT name |
| 133 | FROM "d1_migrations"` (the table Wrangler keeps; a unit's |
| 134 | `migrations_table` if it names one), compared with the `.sql` files in the |
| 135 | unit's migrations folder. Every request starts at once, at most 16 in |
| 136 | flight, each tried once more after a 403, 429, 5xx or network error, so a |
| 137 | plan of every unit takes a few seconds and needs no Wrangler. No KV |
| 138 | namespace or other infrastructure is needed. |
| 139 | |
| 140 | The API is used when the tool has the token Wrangler would be given (in CI, |
| 141 | `CLOUDFLARE_API_TOKEN`; on a laptop, `CLOUDFLARE_DEPLOY_TOKEN`) and |
| 142 | `CLOUDFLARE_ACCOUNT_ID` (or g1t's account by default). Without a token |
| 143 | (your `wrangler login`) the plan asks Wrangler instead, with the same |
| 144 | answers, a few units at a time; that takes minutes. Either way a Worker |
| 145 | that does not exist (404, or Cloudflare's code 10007) is "never deployed", |
| 146 | and any other failure is a reason in the plan, never a crash. |
| 147 | |
| 148 | - A version made by `wrangler secret put` keeps the code of the one before |
| 149 | it, so the tool looks through those to the deploy before. |
| 150 | - A version deployed any other way (by hand, from the dashboard) has no |
| 151 | commit, and the unit is deployed again. |
| 152 | - During a gradual rollout the version with the most traffic counts. |
| 153 | - After `wrangler rollback`, the plan sees the older commit and deploys |
| 154 | what changed since. |
| 155 | |
| 156 | ### What a deploy does |
| 157 | |
| 158 | 1. Plans: for each unit, the live commit; `git diff` from it to `HEAD`; |
| 159 | whether the changed files touch the unit (its folder, the crates and |
| 160 | packages it is built from, its inputs, lockfile changes that reach it). |
| 161 | A Rust crate's `tests/`, `benches/` and `examples/` are not what it is |
| 162 | built from, and neither is a source file compiled only for tests: one |
| 163 | whose every `mod` declaration is under `#[cfg(test)]` (with or without |
| 164 | `#[path]`), or inside a module that is. A change to a crate's tests |
| 165 | alone deploys nothing (`scripts/deploy/stack.mjs`, `testOnlySource`). |
| 166 | 2. Refuses if uncommitted changes touch what would deploy (`--allow-dirty`). |
| 167 | 3. Applies every pending migration (`wrangler d1 migrations apply --remote`), |
| 168 | in parallel. Any failure stops the deploy before code. |
| 169 | 4. Installs worker-build once if any Rust unit is deploying. |
| 170 | 5. Each stage in turn: units in parallel (`--concurrency`), each `npm run |
| 171 | build` first for React Router and Astro, then `wrangler deploy` in the |
| 172 | unit's folder with the annotation. If any unit fails, later stages are |
| 173 | not started. |
| 174 | 6. Prints a table: unit, stage, result, version id, time. Each unit's full |
| 175 | output is kept in `$TMPDIR/g1t-deploy/<unit>.log`. |
| 176 | |
| 177 | #### Migrations run while the old code is still live |
| 178 | |
| 179 | Migrations apply before any code, and a stage takes a minute or more, so |
| 180 | for that long the code in production is the old code reading the new |
| 181 | schema. A migration must keep the old code right: |
| 182 | |
| 183 | - **Add, never change meaning.** New tables and columns (with defaults) are |
| 184 | safe. Rewriting what existing rows mean is not: on 2026-10-08, |
| 185 | `billing/0039` turned comped accounts into `custom` terms at 100%, and |
| 186 | for about 40 seconds the old billing code, which knew only `comped`, saw |
| 187 | flagon-io as a free workspace and refused its workflows. |
| 188 | - **Change meaning in two deploys.** First ship code that reads both the old |
| 189 | and the new form (and keeps writing the old one); then, in a later |
| 190 | deploy, the migration that rewrites the rows; then, if you like, code |
| 191 | that drops the old form. |
| 192 | - **Never drop or rename** a column or table the live code reads in the same |
| 193 | deploy that stops reading it. |
| 194 | |
| 195 | #### Telling the status page about a deploy |
| 196 | |
| 197 | Restarts during a deploy can make a part slow for a minute, which the |
| 198 | status page's checks would otherwise draft as an incident. So |
| 199 | `scripts/deploy.mjs deploy` says when it starts its stages and when they |
| 200 | end (`withDeployWindow` in `scripts/deploy/status-window.mjs`; never in a |
| 201 | dry run, and only once there is something to ship), by hand and in g1t |
| 202 | Actions alike. It posts to `POST https://status.g1t.sh/deploys` with |
| 203 | `Authorization: Bearer $STATUS_DEPLOY_TOKEN`, the same value as the status |
| 204 | Worker's `STATUS_DEPLOY_TOKEN` secret, and `{"phase": "started" | |
| 205 | "finished", "id": "<commit>"}`. During the deploy and for 3 minutes after |
| 206 | it, detection keeps counting failed and slow checks but makes no new |
| 207 | draft; trouble that outlasts that is drafted with its true start. Deploys |
| 208 | that overlap (the jobs of one stage run at once, each announcing itself) |
| 209 | are one window: the status Worker counts the starts, and the window closes |
| 210 | when the last one finishes. A start with no finish stops counting 30 |
| 211 | minutes after the latest start. Without the token nothing is sent, and an |
| 212 | announcement never fails a deploy: a refusal or network error is one |
| 213 | warning line. By hand: `node scripts/deploy/status-window.mjs |
| 214 | started|finished [id]`. |
| 215 | |
| 216 | To turn it on (once; until then deploys are not announced): |
| 217 | |
| 218 | 1. Make a token and set it as the status Worker's secret: |
| 219 | `npx wrangler secret put STATUS_DEPLOY_TOKEN` in `apps/status` (as of |
| 220 | 2026-10-08 the Worker has only `STATUS_SECRET`). Without it, |
| 221 | `POST /deploys` answers 404. |
| 222 | 2. Set the same value as the **`STATUS_DEPLOY_TOKEN`** Actions secret on |
| 223 | flagon-io/g1t (Settings, Secrets and variables, or |
| 224 | `PUT /repos/flagon-io/g1t/actions/secrets/STATUS_DEPLOY_TOKEN`); |
| 225 | `.g1t/workflows/deploy.yml` passes it to every deploy job. |
| 226 | 3. Add `status.g1t.sh | deploy.yml | production` to the project's |
| 227 | **Workflow-only domains** (see Network below), or the job's request is |
| 228 | refused by the guardrails (the deploy still goes on, with a warning). |
| 229 | 4. For deploys by hand, set `STATUS_DEPLOY_TOKEN` in your shell's |
| 230 | environment (the tool does not read `.env`). |
| 231 | |
| 232 | On a laptop the tool uses your `wrangler login` (or `CLOUDFLARE_DEPLOY_TOKEN` |
| 233 | if set), as `scripts/deploy.sh` always did: a `CLOUDFLARE_API_TOKEN` or |
| 234 | global API key in your shell, or in the repository's `.env`, is ignored. |
| 235 | With `CI=true` it uses `CLOUDFLARE_API_TOKEN`. With a token, `plan` reads |
| 236 | Cloudflare's API itself; with only `wrangler login`, it asks Wrangler (see |
| 237 | "Where the deployed commit is kept"). |
| 238 | |
| 239 | ### The runner's images |
| 240 | |
| 241 | `services/runner` runs every sandbox (agents, checks, the merge queue, |
| 242 | workflow jobs, g1t.page builds) from one Containers image, made in two |
| 243 | parts: |
| 244 | |
| 245 | | Image | Built from | Holds | Rebuilt | |
| 246 | | --- | --- | --- | --- | |
| 247 | | **Base**, `g1t-runner:base-<date>-<inputs>` | `services/runner/base/Dockerfile` | Debian bookworm, Node 24, Python 3.11, Docker (Engine, Buildx, Compose, from Docker's apt repository), Go (from go.dev), Rust stable for the `node` user with rustfmt, clippy and the `wasm32-unknown-unknown` target, build-essential, musl-tools, git, ripgrep, jq, zstd, sudo, and the pinned Claude Code CLI on top | When its folder changes, weekly, or by hand (`build-base`) | |
| 248 | | **Runner**, `g1t-runner:<content hash>` | `services/runner/Dockerfile`: `FROM` the base, plus one file | The g1t runner, a static binary | When the binary or the base changes | |
| 249 | |
| 250 | Both are pushed to one repository of Cloudflare's registry, |
| 251 | `registry.cloudflare.com/<account>/g1t-runner`, so pushing the runner's |
| 252 | image uploads only its own layer (about 5 MB): the base's layers are |
| 253 | already there. |
| 254 | |
| 255 | Both are built as Wrangler builds images (`--platform linux/amd64 |
| 256 | --provenance=false --sbom=false`): one manifest, not an OCI index with a |
| 257 | BuildKit attestation beside it. A push is tried up to three times and counts |
| 258 | only when Docker reports the digest; the first push of the base once failed |
| 259 | with `blob unknown to registry` and went through when run again (see |
| 260 | `docs/CLOUDFLARE_FEEDBACK.md`, C3). `build-base` and `image` exit non-zero |
| 261 | when a push fails, and `base.json` is written only after the push. |
| 262 | |
| 263 | **The base** is recorded in `services/runner/base.json`: its reference, its |
| 264 | digest, a hash of its folder (`inputs`), when it was built, its size and |
| 265 | each toolchain's version. `node scripts/deploy.mjs build-base` builds it, |
| 266 | pushes it and rewrites the file; commit the file, and the next deploy |
| 267 | builds the runner's image on it. `npm run test:deploy` fails while the |
| 268 | folder and the file disagree, so a change to the base's Dockerfile cannot |
| 269 | merge without the base it describes. Layers go from what changes least to |
| 270 | most (system packages, Go, Rust, Java, .NET and Ruby, the Claude Code |
| 271 | CLI), and the apt and npm caches stay in BuildKit's cache, out of the |
| 272 | image. Ruby is compiled from source in a single step (a few minutes on a |
| 273 | cold cache) that removes its source tree before the layer is written. |
| 274 | |
| 275 | The base's build cache is the base itself: it is built with |
| 276 | `BUILDKIT_INLINE_CACHE`, which records in the image how each layer was |
| 277 | made, and `build-base` builds `--cache-from` the base it replaces. A |
| 278 | machine with an empty cache, or one just pruned, pulls the unchanged |
| 279 | layers from the registry instead of building them. (BuildKit's other |
| 280 | registry cache, `--cache-to type=registry`, pushes a separate cache |
| 281 | manifest that not every registry takes, and for a one-stage image adds |
| 282 | nothing the inline cache lacks.) |
| 283 | |
| 284 | **The runner binary** (`crates/runner`) is built outside Docker by |
| 285 | `scripts/build-runner.mjs` as one static binary for |
| 286 | `x86_64-unknown-linux-musl`, so it runs on the base whatever its libc, and |
| 287 | on a self-hosted runner's machine too. Where it is built: |
| 288 | |
| 289 | - on x86-64 Linux with the musl target and `musl-gcc` |
| 290 | (`rustup target add x86_64-unknown-linux-musl`, `apt-get install musl-tools`), |
| 291 | with the machine's own Cargo; |
| 292 | - anywhere else (Windows, macOS) in a small builder container, Rust on |
| 293 | Alpine (whose own target is musl), with Docker volumes keeping Cargo's |
| 294 | registry and target directory between builds. Windows has no musl |
| 295 | cross-linker, and `ring` (under ureq's TLS) needs a C compiler for the |
| 296 | target, so a container is the dependable route. |
| 297 | |
| 298 | **The runner's image tag** is a hash of everything it is built from: its |
| 299 | Dockerfile, `base.json`, the crates the binary is built from, the |
| 300 | workspace's Cargo files and the build script. The same source always names |
| 301 | the same image, so: |
| 302 | |
| 303 | 1. A deploy computes the tag and asks the registry whether it is there |
| 304 | (a `HEAD` of its manifest, with credentials from Wrangler; no Docker). |
| 305 | 2. If it is, nothing is built: the deploy uses it. |
| 306 | 3. If not, it builds the binary and the image (seconds on a warm machine) |
| 307 | and pushes it. In `deploy.yml` that is the `runner-image` job, on |
| 308 | `g1t-4core`, with the job's own Docker Engine (see |
| 309 | [Docker in workflow jobs](#docker-in-workflow-jobs)): it adds the musl |
| 310 | target, builds the binary natively (the base has `musl-gcc`), pulls the |
| 311 | base from Cloudflare's registry, builds, and pushes one layer. |
| 312 | 4. If Docker does not answer (a machine without it, or jobs with Docker |
| 313 | turned off), the unit fails saying to run `node scripts/deploy.mjs |
| 314 | image` on a machine with Docker; then re-run the workflow. |
| 315 | |
| 316 | Then `wrangler deploy` is given the image by reference, from a generated |
| 317 | config (`services/runner/wrangler.deploy.json`, deleted after, ignored by |
| 318 | git), so Wrangler builds nothing. When nothing the image is built from |
| 319 | changed since the runner's live commit, the deploy also passes |
| 320 | `--containers-rollout none`, which leaves running sandboxes alone. |
| 321 | |
| 322 | To get a new base out: |
| 323 | |
| 324 | ```sh |
| 325 | node scripts/deploy.mjs build-base # build, push, write base.json (needs Docker) |
| 326 | git commit services/runner/base.json -m "A new base image for g1t's sandboxes" |
| 327 | node scripts/deploy.mjs image # optional: push the runner's image now, so CI finds it |
| 328 | ``` |
| 329 | |
| 330 | `.g1t/workflows/runner-base.yml` does the same weekly (and when the |
| 331 | base's folder changes on `main`), and opens a pull request with |
| 332 | `base.json`. It runs on a self-hosted runner with the `docker` label |
| 333 | (`runs-on: [self-hosted, docker]`): g1t's own machines have Docker now, |
| 334 | but the base's build downloads from Docker's apt repository over HTTPS, |
| 335 | which does not trust a guarded job's egress certificate, so it stays on an |
| 336 | open network. Its job is skipped until the repository variable |
| 337 | `RUNNER_BASE_SELF_HOSTED` is `true` (set it once a runner is registered; |
| 338 | without one, runs waited in the queue for a day and a half and then |
| 339 | failed); until then run `build-base` by hand. |
| 340 | |
| 341 | **Sandboxes start from the image.** Cloudflare pulls an image to a |
| 342 | machine the first time a sandbox lands there, and keeps it. A smaller base |
| 343 | pulls sooner, and a change to the runner alone sends machines one 5 MB |
| 344 | layer instead of the whole image. |
| 345 | |
| 346 | #### Larger machines |
| 347 | |
| 348 | The same image runs on three instance types, each a Durable Object class |
| 349 | of its own in `services/runner/wrangler.jsonc`: `AttemptSandbox` |
| 350 | (`standard-1`), `Sandbox2Core` (`standard-3`) and `Sandbox4Core` |
| 351 | (`standard-4`). Workflow jobs choose with `runs-on: g1t-2core` or |
| 352 | `g1t-4core` (`g1t_contracts::actions::INSTANCE_TYPES`); the actions |
| 353 | service passes the label to the runner, which starts the job in that |
| 354 | class. Billing prices the larger ones from their memory and disk, and |
| 355 | their CPU (see the public billing guide). The account's Containers limits |
| 356 | must allow `standard-4`; Wrangler refuses the deploy otherwise. |
| 357 | |
| 358 | #### Docker in workflow jobs |
| 359 | |
| 360 | Workflow jobs on g1t's machines have a Docker Engine of their own |
| 361 | (`crates/runner/src/docker/`; the public guide is |
| 362 | `apps/docs/src/content/docs/guides/actions.md`, "Docker"). What Cloudflare |
| 363 | Containers allow decides how it runs (findings in `docs/PLAN.md`, |
| 364 | "Docker in workflow jobs"): |
| 365 | |
| 366 | | Piece | What it does | |
| 367 | | --- | --- | |
| 368 | | `dockerd` | Started as root with `sudo`, only when the job first uses Docker or has `services:` or `container:`. Flags: `--iptables=false --ip6tables=false --ip-forward=false` (Containers allow neither), the containerd image store, Docker Hub through `mirror.gcr.io`. Its config, socket and log are in `/run/g1t-docker` (`dockerd.log` is the place to look); its data in `/var/lib/docker`, on overlays when the disk takes them, plain copies (`native`) when not. | |
| 369 | | `/var/run/docker.sock` | The runner's API proxy (`docker/api.rs`), which starts the Engine on the first connection. Containers that ask for a bridge network get the job's own (`host`), the names they would have had resolve to 127.0.0.1, and ports published under another number are forwarded. | |
| 370 | | `runc` | The Engine finds the runner binary first on its `PATH` as `runc` (`docker/oci.rs`): BuildKit's `RUN` steps join the job's network, and in a guarded job every container gets the egress certificate at `/dev/g1t-egress`. Then the real `/usr/bin/runc` runs. | |
| 371 | | cgroups | Before the Engine starts, the sandbox's processes move to a cgroup of their own and every controller is handed down, as Docker's own Docker-in-Docker image does, so `--cpus` and `--memory` work. | |
| 372 | |
| 373 | - **Turning it off:** set the runner Worker's `DOCKER` var to `off` |
| 374 | (`services/runner/wrangler.jsonc`) and deploy the runner: new jobs get |
| 375 | no Engine (`G1T_DOCKER=off`), and jobs that need one fail saying Docker |
| 376 | does not answer. Anything else is on. |
| 377 | - **Network:** containers share the job's network, so the guardrails, the |
| 378 | workflow-only domains and the egress Worker apply to them unchanged. The |
| 379 | public registries are in `BUILD_HOSTS` (`services/runner/src/egress.ts`). |
| 380 | - **Isolation:** one Engine per job, inside the job's sandbox (its own |
| 381 | VM), gone with it. The job already had root through `sudo`; Docker adds |
| 382 | no reach beyond the sandbox, and no host socket is ever mounted into one. |
| 383 | - **Checked locally** (2026-10-08, Docker Desktop, the base and runner |
| 384 | image built from this tree, a privileged container standing in for a |
| 385 | sandbox, a pretend API): services with health checks, `localhost` and |
| 386 | names, port forwarding, `docker build` with a networked `RUN`, Compose |
| 387 | with a healthy dependency, `docker://` steps, a Dockerfile action, a |
| 388 | `container:` job with a JavaScript action, an Alpine job container, the |
| 389 | egress certificate in `run`, `exec` and build steps and in no layer, |
| 390 | plain-copy storage, and the deploy's own build and push to a registry. |
| 391 | Not yet seen on Cloudflare itself: watch the first runs' logs for the |
| 392 | `Docker: started` line, and `dockerd.log` if it does not come. |
| 393 | |
| 394 | ## Build speed |
| 395 | |
| 396 | Measured on the development machine (Windows, 32 cores, warm Cargo cache), |
| 397 | building `events`, `search` and `repos` after a change to `crates/kit`, as a |
| 398 | deploy does but without uploading (`wrangler deploy --dry-run`): |
| 399 | |
| 400 | | | Time | |
| 401 | | --- | --- | |
| 402 | | Before: one after another, `cargo install worker-build` each time, wasm-opt `-O` | 81 s, 84 s | |
| 403 | | Concurrent builds, wasm-opt `-O` | 50 s, 68 s | |
| 404 | | Concurrent builds, wasm-opt `-O1` (now) | 28 s, 33 s | |
| 405 | |
| 406 | Where the time went, and what changed: |
| 407 | |
| 408 | - **wasm-opt** was most of it: `-O` took 38 s on `repos` and 19 s on `api`; |
| 409 | `-O1` takes 1 to 3 s. With worker-build's flags (it keeps the names |
| 410 | section) the `.wasm` is 24 to 28% larger raw but only 1 to 7% larger |
| 411 | gzipped, and Workers' size limit is on the compressed upload. `-Os` and |
| 412 | `-Oz` were no faster than `-O`. Set per crate in |
| 413 | `[package.metadata.wasm-pack.profile.release]`; a test keeps every Rust |
| 414 | unit on the same level. |
| 415 | - **worker-build** is installed only when missing or another version |
| 416 | (`scripts/build-rust-worker.mjs`, which pins it). Locally `cargo install` |
| 417 | on an installed version cost under a second; on a fresh CI sandbox it is |
| 418 | a full compile, which the workflow caches instead. |
| 419 | - **One Cargo target**: every Rust unit is a member of the workspace, so |
| 420 | they already share `target/`. Concurrent builds take turns on Cargo's |
| 421 | lock for the compile, and their wasm-bindgen, wasm-opt and uploads |
| 422 | overlap. |
| 423 | - **No joint `cargo build -p a -p b`**: tried, and it is slower. Cargo |
| 424 | unifies features across the packages of one build (`serde_json`'s |
| 425 | `preserve_order` from `api` and `actions`, `digest` features from |
| 426 | `secrets`), so each worker-build afterwards compiled its own variant |
| 427 | again. |
| 428 | - **The runner's images**, measured on the same machine on 2026-10-06 |
| 429 | (Docker Desktop, 8 vCPUs; its disk was busy with other containers, so |
| 430 | the cold figures are slow and noisy): |
| 431 | |
| 432 | | | Before (one image) | Now | |
| 433 | | --- | --- | --- | |
| 434 | | Size, unpacked / compressed (what a machine pulls) | 3.08 GB / 819 MB | 2.68 GB / 686 MB | |
| 435 | | A change to the runner | Docker rebuilds the image's Rust stage and pushes the image (1198 s in the first deploy after a prune) | binary 46–53 s (6 s unchanged), image 7 s, push one 5 MB layer (1.4 s to a local registry) | |
| 436 | | The base from nothing | 431 s (whole image, cold) | 758 s cold, rarely: weekly or when its folder changes | |
| 437 | | The base after `docker builder prune` | as from nothing | 68 s, its layers pulled from the registry it was pushed to | |
| 438 | | The runner binary, cold (builder container) | | 99 s | |
| 439 | | A Rust CI job's build (events, search, repos; 4 vCPUs) | | 51 s cold, 13 s with the Cargo target restored (107 MB zstd entry) | |
| 440 | |
| 441 | - **sccache** (`scripts/sccache.sh`, pinned by version and sha256): a |
| 442 | checkout gives every source a new mtime, so Cargo calls rustc again for |
| 443 | every workspace crate a unit uses, however much of `target/` was |
| 444 | restored. With `RUSTC_WRAPPER=sccache` and its GitHub Actions backend, |
| 445 | each call that makes a library is looked up by its inputs in the |
| 446 | repository's Actions cache (the same cache as `actions/cache`, scoped to |
| 447 | `main`), and an unchanged crate comes back from it. Measured on the |
| 448 | development machine on 2026-10-08, `repos` and `events` for wasm32, |
| 449 | release, `-j 4`, the dependencies already built, every workspace source |
| 450 | touched as a checkout does (sccache's local disk cache; the Actions |
| 451 | cache adds a download per hit): |
| 452 | |
| 453 | | | Time | |
| 454 | | --- | --- | |
| 455 | | Without sccache (before) | 44.3 s, 44.6 s | |
| 456 | | sccache, empty cache (its first run, filling it) | 48.8 s | |
| 457 | | sccache, filled (6 hits: `contracts`, `kit`, `rules`, `scan`, `secrets`, `blobstore`) | 24.2 s | |
| 458 | |
| 459 | CI's build the same way (`cargo test --workspace --no-run`, debug, |
| 460 | `-j 4`, `CARGO_INCREMENTAL=0`): 70.2 s and 74.6 s without sccache, 66.5 s |
| 461 | filling it, 40.6 s filled (7 library hits; 23 test harnesses and |
| 462 | cdylibs compiled). |
| 463 | |
| 464 | What is left is each Worker's own crate: a `cdylib` is linked, and |
| 465 | sccache does not cache what rustc links (binaries, cdylibs, proc |
| 466 | macros, build scripts, test harnesses). That compile is real work |
| 467 | anyway: a unit deploys because it, or a crate it uses, changed. The |
| 468 | same holds for CI's `cargo test`: the workspace's libraries come back |
| 469 | from the cache, the test harnesses are compiled. Every Rust job of |
| 470 | Deploy and CI's Rust job end with sccache's hits and misses in the |
| 471 | run's summary. If the cache cannot be reached, or the download fails |
| 472 | its checksum, the job warns and builds as before. On g1t, sccache |
| 473 | speaks the toolkit cache's older protocol (`GITHUB_SERVER_URL` is not |
| 474 | github.com): a lookup, a reservation, one `PATCH` of the whole entry |
| 475 | (`Content-Range: bytes 0-N/*`) and a commit per miss; a lookup and a |
| 476 | blob `GET` per hit. Its storage check saves `sccache/.sccache_check` |
| 477 | once, and takes the `409` on later runs as "already there". |
| 478 | - **Not restoring mtimes:** setting each file's mtime from git (so Cargo |
| 479 | would trust the restored `target/`) was considered and left out. A |
| 480 | restored `target/` may come from another commit (the cache's |
| 481 | `restore-keys` take the nearest earlier entry, of any group), and a |
| 482 | file changed by an older commit than that build would look unchanged: |
| 483 | a stale crate deployed. sccache looks at what is compiled, not when. |
| 484 | - **Only what changed** is the largest saving: a change to one service |
| 485 | deploys one service, and a change only to a crate's tests deploys |
| 486 | nothing. |
| 487 | |
| 488 | ## The workflow |
| 489 | |
| 490 | `.g1t/workflows/deploy.yml` runs on every push to `main`, and by hand |
| 491 | (**Actions → Deploy → Run workflow**) with `units` (deploy these, changed or |
| 492 | not), `all` and `dry_run` (plan only). |
| 493 | |
| 494 | | Job | Does | Needs | |
| 495 | | --- | --- | --- | |
| 496 | | `check` | `manifest --check` and `npm run test:deploy` | — | |
| 497 | | `plan` | `plan --github-output`: outputs per stage, the plan in the run's summary. Reads Cloudflare's API itself, so it installs nothing. | — (runs beside `check`) | |
| 498 | | `migrate` | `migrate --only <units with pending migrations>` | `check` and `plan`; skipped when none are pending | |
| 499 | | `core`, `edge`, `front` | `deploy --only <units> --force --no-migrations`, one job per build group | `check`, `plan` and the stages before; skipped when empty | |
| 500 | | `smoke` | `node scripts/ops/smoke.mjs`: the landing page, sign-in, sign-up and pricing load, and the waitlist form reaches identity (sent an address identity refuses before keeping or counting anything, so the real waitlist is never touched) | `check`, `plan` and every stage; skipped when nothing deployed | |
| 501 | |
| 502 | - **One at a time:** `concurrency: deploy-production`, never cancelled in |
| 503 | progress; a second push waits. |
| 504 | - **Check and plan side by side:** neither waits for the other, so the |
| 505 | plan's few seconds overlap the check's tests; nothing migrates or deploys |
| 506 | until both have succeeded. The plan job keeps `fetch-depth: 0`: it diffs |
| 507 | from each Worker's live commit, which may be any commit, and tells a |
| 508 | rollback by ancestry, which a shallow clone cannot answer. |
| 509 | - **Build groups:** a stage's units are split so each job shares a build: |
| 510 | Rust workers at most four to a job (each a 4-vCPU `g1t-4core` machine), the |
| 511 | TypeScript Workers together, each site alone, and a unit whose image must |
| 512 | be rebuilt alone (`image: true` in the matrix, also on `g1t-4core`, where |
| 513 | it builds and pushes the image with the job's own Docker Engine). `fail-fast: false`, so one failed job does not cut |
| 514 | another off mid-upload; the next stage then does not start. |
| 515 | - **Tests:** `.g1t/workflows/ci.yml` tests every pull request into `main` |
| 516 | (and, on pushes to `main`, runs only its Rust job, to keep the caches |
| 517 | pull requests restore from current). `check` runs the deploy tool's own |
| 518 | tests. Deploy does not wait for CI. |
| 519 | - **Machines:** Rust jobs and the runner's image run on `g1t-4core` (4 vCPUs, |
| 520 | 12 GiB, 20 GB), the others on the standard machine |
| 521 | (`runs-on: ${{ (matrix.rust || matrix.image) && 'g1t-4core' || 'ubuntu-latest' }}`). |
| 522 | The image job needs the room: the base it builds on is about 3.2 GB |
| 523 | unpacked. |
| 524 | - **Caching** (`actions/cache`: up to 2 GB an entry, 10 GB a repository, |
| 525 | kept until unused for 7 days): the worker-build binary, worker-build's |
| 526 | downloaded tools, `~/.cargo/registry/cache`, and the Cargo target's |
| 527 | release dependencies (`target/release` and |
| 528 | `target/wasm32-unknown-unknown/release`, without `incremental` or |
| 529 | `.wasm`), keyed by the build group, `Cargo.lock` and `base.json`. Cargo |
| 530 | calls rustc again for the workspace's own crates on every run (a |
| 531 | checkout's sources are newer than any cache); **sccache** answers those |
| 532 | calls from the repository's Actions cache when a crate's inputs did not |
| 533 | change (see [Build speed](#build-speed)). npm's cache is not kept: every |
| 534 | job runs `npm ci` of only what its units need (`deploy.mjs install`: |
| 535 | Wrangler alone for Rust jobs). |
| 536 | - **Conditions:** each stage runs with `!failure() && !cancelled()`, which |
| 537 | on g1t (as on GitHub) is true when no job before it failed, however far |
| 538 | back: a `migrate` job skipped for having nothing to apply does not stop |
| 539 | the stages after it, and a failed `check` or `plan` stops all of them |
| 540 | (`migrate`'s own condition needs both to have succeeded). |
| 541 | - `crates/actions/tests/repository_workflows.rs` reads the workflow with |
| 542 | g1t's own parser and expressions, and checks the jobs start, wait and |
| 543 | stop as above (`cargo test -p g1t-actions --test repository_workflows`). |
| 544 | |
| 545 | ### What the sandbox has |
| 546 | |
| 547 | The base image (`services/runner/base/Dockerfile`) has Node 24, npm, git, |
| 548 | Go, zstd, Docker, musl-tools, Rust stable for the `node` user with |
| 549 | rustfmt, clippy and the `wasm32-unknown-unknown` target, Java (Temurin 21), |
| 550 | the .NET 8 SDK and Ruby 3.3, but not worker-build or `gh`. Java and Ruby |
| 551 | sit in the tool cache (`/home/runner/_tool/Java_Temurin-Hotspot_jdk/<semver |
| 552 | with + as ->/x64` and `/home/runner/_tool/Ruby/<version>/x64`, each with |
| 553 | an `x64.complete` marker), which is where `actions/setup-java` and |
| 554 | `ruby/setup-ruby` look; .NET is in `/usr/share/dotnet`, owned by `node`, |
| 555 | which is where `actions/setup-dotnet` installs. Bumping one means changing |
| 556 | its version and checksum `ARG`s together (Java's tool cache name is |
| 557 | Adoptium's `version_data.semver`; .NET's SHA-512 is in its release |
| 558 | metadata; Ruby's SHA-256 is in `cache.ruby-lang.org/pub/ruby/index.txt`). |
| 559 | The user docs list what jobs get in |
| 560 | `apps/docs/src/content/docs/guides/actions.md` (The runner). The workflow's `rustup target add wasm32-unknown-unknown` is |
| 561 | then a no-op, and worker-build is restored from the cache, installed on a |
| 562 | miss. The image job adds `x86_64-unknown-linux-musl` (about 30 MB from |
| 563 | `static.rust-lang.org`) and keeps its Cargo target in the cache. worker-build |
| 564 | fetches wasm-bindgen and wasm-opt from GitHub releases and esbuild from |
| 565 | npm. Each Rust job downloads sccache (about 10 MB) from its GitHub release |
| 566 | too (`scripts/sccache.sh`, which checks its sha256); it is not in the base |
| 567 | image, and putting it there means a pinned download in the base's |
| 568 | Dockerfile and a base rebuild (`build-base`). All of those hosts are on |
| 569 | the list every workflow job may reach. |
| 570 | |
| 571 | ### Network |
| 572 | |
| 573 | A workflow job reaches its project's allowed domains, g1t, what builds |
| 574 | need (`services/runner/src/egress.ts`, `BUILD_HOSTS`), and the project's |
| 575 | **workflow-only domains** that name its workflow and environment. Those are |
| 576 | never reached by agents, checks, the merge queue, deploy builds or runs of |
| 577 | pull requests from forks (`Guardrails::workflow_hosts`, the runner's |
| 578 | `jobHosts`). Under flagon-io/g1t's **Settings → Guardrails** |
| 579 | (Maintain role or higher), **Workflow-only domains**: |
| 580 | |
| 581 | ```text |
| 582 | api.cloudflare.com | deploy.yml | production |
| 583 | registry.cloudflare.com | deploy.yml, runner-base.yml | production |
| 584 | status.g1t.sh | deploy.yml | production |
| 585 | ``` |
| 586 | |
| 587 | `api.cloudflare.com` is Cloudflare's API, which Wrangler and the plan call; `registry.cloudflare.com` is where |
| 588 | the deploy asks whether the runner's image is already built, and where the |
| 589 | `runner-image` job pulls the base from and pushes the runner's image to |
| 590 | (as `runner-base.yml` pushes the base); `status.g1t.sh` hears the deploy |
| 591 | start and finish (see "Telling the status page about a deploy"). The job's Docker Engine shares the |
| 592 | job's network, so these lines are what let it reach the registry. If a pull |
| 593 | is refused with `g1t guardrails: <host> is not on this project's allowed |
| 594 | domains`, the registry sent the layers from another host: add that host on |
| 595 | the same line. Only `deploy.yml`'s and `runner-base.yml`'s |
| 596 | jobs with `environment: production` reach them, which are also the only |
| 597 | jobs that can read `CLOUDFLARE_API_TOKEN`. Each change to the list is in |
| 598 | the workspace's audit log as `update_guardrails`. |
| 599 | |
| 600 | ### The API token |
| 601 | |
| 602 | Create it at **dash.cloudflare.com → My Profile → API Tokens → Create |
| 603 | Token → Custom token**, named `g1t deploys (CI)`: |
| 604 | |
| 605 | | Scope | Permission | Why | |
| 606 | | --- | --- | --- | |
| 607 | | Account | Workers Scripts: Edit | Upload, versions, deployments (the plan reads both), crons, bindings, `secret list` (doctor) | |
| 608 | | Account | D1: Edit | The plan's query of each database's `d1_migrations`, and `d1 migrations apply` | |
| 609 | | Account | Queues: Edit | Attaching each unit's queue consumers on deploy | |
| 610 | | Account | Workers R2 Storage: Read | Wrangler checks `og`'s bucket binding | |
| 611 | | Account | Account Settings: Read | Wrangler reads the account | |
| 612 | | Account | Containers: Edit | The runner's deploy updates its applications (the image reference, the three classes), and gets registry credentials (`wrangler containers registries credentials --push`, one hour) to look for, pull and push its image. `runner-base.yml` pushes images with it. | |
| 613 | | Zone (`g1t.sh`, `g1t.page`) | Workers Routes: Edit | `pages`' zone routes, and custom domains | |
| 614 | | Zone (`g1t.sh`, `g1t.page`) | DNS: Edit | Custom domains (`api`, `mcp`, `og`, `models`, `status`, `sudo`, `docs`, `g1t.sh`, `g1t.page`) keep their DNS records | |
| 615 | | Zone (`g1t.sh`, `g1t.page`) | Zone: Read | Finding the zone a route names | |
| 616 | |
| 617 | Restrict it to account `syntaqx` (`1e6f2cffa3f445920836e8ebe446bb58`) and |
| 618 | the two zones. Not needed for deploys: KV (bindings are by id; creating a |
| 619 | namespace is a one-time setup), Vectorize, Workers for Platforms beyond |
| 620 | Workers Scripts, Cloudflare for SaaS custom hostnames (the deployments |
| 621 | service does that at runtime with its own token), SSL and Certificates. |
| 622 | Workers KV Storage: Edit and Vectorize: Edit are only for first-time setup, |
| 623 | which stays a person's job. |
| 624 | |
| 625 | The list follows what our configs use; Cloudflare does not publish exactly |
| 626 | what `wrangler deploy` checks for each binding. Bindings with no listed |
| 627 | permission (Browser Rendering, Workers AI, Vectorize, Email Sending, |
| 628 | Artifacts, dispatch namespaces) are assumed to need none beyond Workers |
| 629 | Scripts. Verify on the first run with `workflow_dispatch` and `units: |
| 630 | pages` (small, no secrets), then `units: og` (R2) and `units: runner` |
| 631 | (Containers); a missing permission fails with `Authentication error [code: |
| 632 | 10000]` and the route it was refused. |
| 633 | |
| 634 | Add it to the repository: |
| 635 | |
| 636 | 1. On g1t.sh, open **flagon-io/g1t → Settings → Secrets and variables**. |
| 637 | 2. **Add**: key `CLOUDFLARE_API_TOKEN`, type **Secret**, available to |
| 638 | **Workflows**, environment **Production**. Only jobs with |
| 639 | `environment: production` (the deploy jobs) can read it. |
| 640 | 3. **Add**: key `CLOUDFLARE_ACCOUNT_ID`, type **Variable**, value |
| 641 | `1e6f2cffa3f445920836e8ebe446bb58`, available to **Workflows**, all |
| 642 | environments. |
| 643 | |
| 644 | Or through the API: |
| 645 | |
| 646 | ```sh |
| 647 | curl -X PUT https://api.g1t.sh/repos/flagon-io/g1t/actions/secrets/CLOUDFLARE_API_TOKEN \ |
| 648 | -H "Authorization: Bearer $G1T_TOKEN" -H "Content-Type: application/json" \ |
| 649 | -d '{"value":"<the token>","environments":["production"],"available_to":["workflows"]}' |
| 650 | |
| 651 | curl -X POST https://api.g1t.sh/repos/flagon-io/g1t/actions/variables \ |
| 652 | -H "Authorization: Bearer $G1T_TOKEN" -H "Content-Type: application/json" \ |
| 653 | -d '{"name":"CLOUDFLARE_ACCOUNT_ID","value":"1e6f2cffa3f445920836e8ebe446bb58","available_to":["workflows"]}' |
| 654 | ``` |
| 655 | |
| 656 | ## Turning it on |
| 657 | |
| 658 | 1. Create the token and add the secret and variable (above). |
| 659 | 2. Add `api.cloudflare.com` and `registry.cloudflare.com` to flagon-io/g1t's |
| 660 | workflow-only domains, for `deploy.yml` in `production` (above). |
| 661 | 3. Build and push the base once, and commit `services/runner/base.json`: |
| 662 | `node scripts/deploy.mjs build-base`. Create the cache bucket: |
| 663 | `npx wrangler r2 bucket create g1t-actions-cache`, with lifecycle rules |
| 664 | deleting cache entries (`c/`) 30 days after upload and artifacts (`a/`) |
| 665 | after 91, a day past the longest they are kept, and unfinished uploads |
| 666 | after a day (the actions service deletes both sooner; the rules catch |
| 667 | what it misses): |
| 668 | `npx wrangler r2 bucket lifecycle add g1t-actions-cache expire-cache c/ --expire-days 30 --abort-multipart-days 1` |
| 669 | and `npx wrangler r2 bucket lifecycle add g1t-actions-cache expire-artifacts a/ --expire-days 91 --abort-multipart-days 1`. |
| 670 | A bucket made before artifacts moved there has one rule for everything, |
| 671 | `expire`, which would delete artifacts kept longer than 30 days: remove |
| 672 | it (`npx wrangler r2 bucket lifecycle remove g1t-actions-cache --id expire`) |
| 673 | and add the two above. |
| 674 | 4. Adopt the live Workers once, from a laptop: deploy everything with the |
| 675 | tool so each version records its commit (`scripts/deploy.sh`, or |
| 676 | `node scripts/deploy.mjs deploy --all`). Until then every plan says "no |
| 677 | known commit" and deploys every unit. To see what has changed since a |
| 678 | commit you know production runs, without deploying: |
| 679 | `node scripts/deploy.mjs plan --since <sha>`. |
| 680 | 5. Run the workflow by hand with `dry_run`, then with `units: pages`. |
| 681 | |
| 682 | ## First deploy of a new account |
| 683 | |
| 684 | What the configs refer to must exist first. `node scripts/deploy.mjs |
| 685 | manifest --json` lists, per unit, its queues, KV, R2, Vectorize and |
| 686 | dispatch namespaces; each unit's `setup` and `secrets` say the rest. |
| 687 | |
| 688 | - D1: `npx wrangler d1 create <database>`, then put its id in the unit's |
| 689 | `wrangler.jsonc`. |
| 690 | - KV: `npx wrangler kv namespace create <name>` for each name under |
| 691 | `resources.kv`, then the ids in the configs. |
| 692 | - Queues: `npx wrangler queues create <queue>` for each queue in the |
| 693 | manifest: `g1t-events`, `g1t-events-<service>` for every subscriber, |
| 694 | `g1t-search-jobs`, `g1t-context-jobs`, and the dead-letter queue |
| 695 | `g1t-events-dlq` (below). |
| 696 | - R2: `npx wrangler r2 bucket create g1t-screenshots`, |
| 697 | `npx wrangler r2 bucket create g1t-actions-cache` (with its two |
| 698 | lifecycle rules, above), and `npx wrangler r2 bucket create g1t-git-packs`, |
| 699 | the clone pack cache (`services/repos/src/pack_cache.rs`), with a rule |
| 700 | deleting packs 7 days after they were written and unfinished uploads |
| 701 | after a day: |
| 702 | `npx wrangler r2 bucket lifecycle add g1t-git-packs expire-packs packs/ --expire-days 7 --abort-multipart-days 1`. |
| 703 | A Worker bound to a bucket that does not exist fails to deploy, so make |
| 704 | it before the first deploy of `g1t-repos` that binds it. |
| 705 | - The runner's base image: `node scripts/deploy.mjs build-base`. |
| 706 | - Vectorize, dispatch namespace, DNS, Access, Email Sending, Artifacts: each |
| 707 | unit's `setup`. |
| 708 | - Secrets: `npx wrangler secret put <NAME>` in the unit's folder; |
| 709 | `node scripts/deploy.mjs doctor` lists what is missing. |
| 710 | |
| 711 | Then `node scripts/deploy.mjs deploy --all`. A Worker bound to a service |
| 712 | that does not exist yet may be refused; deploy that service first with |
| 713 | `--only`. |
| 714 | |
| 715 | ## The dead-letter queue |
| 716 | |
| 717 | Every queue consumer (`g1t-events`, each `g1t-events-<service>`, |
| 718 | `g1t-search-jobs`, `g1t-context-jobs`) names `max_retries` and the |
| 719 | dead-letter queue `g1t-events-dlq`, so a message that keeps failing stops |
| 720 | after its retries instead of being retried for ever. A consumer bound to a |
| 721 | queue that does not exist fails to deploy, so create it once per account, |
| 722 | before the first deploy that names it: |
| 723 | |
| 724 | ```sh |
| 725 | npx wrangler queues create g1t-events-dlq |
| 726 | ``` |
| 727 | |
| 728 | Nothing consumes it: read what landed there with `npx wrangler queues |
| 729 | info g1t-events-dlq`, fix the cause, and replay by hand if needed. The |
| 730 | manifest lists each unit's dead-letter queues under `queues.dead_letter`. |
| 731 | |
| 732 | ## OIDC tokens for workflow jobs |
| 733 | |
| 734 | The API is the issuer of workflow jobs' OIDC tokens, |
| 735 | `https://api.g1t.sh/actions/oidc` (`apps/api/src/oidc.rs`): no host or DNS |
| 736 | of its own. It signs with an RSA key kept as the API's secret |
| 737 | `ACTIONS_OIDC_KEY`. Without it, the issuer's addresses answer 404 and jobs |
| 738 | are not told where to ask for a token. |
| 739 | |
| 740 | To turn it on, make a key on a trusted machine and store it, then delete |
| 741 | the file: |
| 742 | |
| 743 | ```sh |
| 744 | openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out oidc.pem |
| 745 | cd apps/api && npx wrangler secret put ACTIONS_OIDC_KEY < ../../oidc.pem |
| 746 | rm ../../oidc.pem |
| 747 | ``` |
| 748 | |
| 749 | Check `https://api.g1t.sh/actions/oidc/.well-known/jwks` lists one key. |
| 750 | Its `kid` is the key's RFC 7638 thumbprint. |
| 751 | |
| 752 | To rotate it, keep the old key published while tokens it signed can still |
| 753 | be presented (they last 5 minutes; relying parties cache keys for longer): |
| 754 | |
| 755 | 1. Store the current key as `ACTIONS_OIDC_KEY_PREVIOUS` (the same PEM). |
| 756 | 2. Make a new key and store it as `ACTIONS_OIDC_KEY`. The JWKS now lists |
| 757 | both; new tokens are signed with the new one. |
| 758 | 3. A day later, `npx wrangler secret delete ACTIONS_OIDC_KEY_PREVIOUS`. |
| 759 | |
| 760 | A key that may have leaked is rotated the same way, skipping the first |
| 761 | step, so that tokens it signed stop verifying at once. |
| 762 | |
| 763 | ## Deployments on g1t |
| 764 | |
| 765 | Every deploy shows on the repository's Deployments page, so its production |
| 766 | card says which commit runs. |
| 767 | |
| 768 | - **From the workflow**, the deploy jobs name `environment: {name: |
| 769 | production, url: https://g1t.sh}`, and g1t Actions records one production |
| 770 | deployment per run. |
| 771 | - **By hand**, `deploy` reports one itself: in progress once migrations are |
| 772 | in, then success or failure. It needs a g1t token with `deployments:write` |
| 773 | in `G1T_DEPLOY_TOKEN` or the file `.credentials/g1t-deploy-token`; without |
| 774 | one, or from a dirty tree or a dry run, it sends nothing. It finds the |
| 775 | repository from the git remote on g1t.sh. A report that fails is one line |
| 776 | in the log and never fails the deploy. |
| 777 | |
| 778 | When CI cannot finish a deploy, for example a runner image that will not |
| 779 | build there, deploy from a machine with Docker. |
| 780 | The report records it, and production shows the commit that really runs. |
| 781 | |
| 782 | ## Rolling back |
| 783 | |
| 784 | - **One unit, at once:** `npx wrangler rollback` in its folder (or |
| 785 | `npx wrangler rollback <version-id>`; `npx wrangler versions list` shows |
| 786 | each version's commit in its message). Code only: D1 migrations are not |
| 787 | undone. The next plan sees the older commit and deploys what changed since, |
| 788 | so revert the commit on `main` too, or the next push brings it back. |
| 789 | - **To a commit:** check it out and `node scripts/deploy.mjs deploy --only |
| 790 | <units> --rollback`. Without `--rollback` the tool refuses any unit whose |
| 791 | live commit is newer than the one checked out, even with `--force`, so a |
| 792 | re-run of an old workflow run (or an old checkout) never rolls production |
| 793 | back by accident. Migrations never run backwards: a migration that needs |
| 794 | undoing is a new migration. |
| 795 | - **The runner's image:** a rollback of the Worker does not roll back the |
| 796 | container image. Redeploy the older commit (`--only runner --rollback`): its |
| 797 | image's tag is the hash of that commit's source, which is still in the |
| 798 | registry, so nothing is built. A bad base is undone by reverting the |
| 799 | commit that changed `services/runner/base.json`. |
| 800 | |
| 801 | ## Adding a unit |
| 802 | |
| 803 | 1. Make its folder with a `wrangler.jsonc` (and its D1 migrations, if any). |
| 804 | A Rust Worker is a workspace member in the root `Cargo.toml` with |
| 805 | `"build": { "command": "node ../../scripts/build-rust-worker.mjs" }` and |
| 806 | the `wasm-opt = ["-O1"]` metadata; a TypeScript one is an npm workspace. |
| 807 | 2. Add it to `deploy/stack.jsonc`: path, kind, worker, `d1`, stage (the |
| 808 | earliest stage after everything it binds to), secrets, setup, self_host. |
| 809 | Name any new KV id under `resources.kv`. |
| 810 | 3. If its sources import a file outside its folder that is not a workspace |
| 811 | crate or package, list it under `inputs`. |
| 812 | 4. Add a row to the service table in `docs/SELF_HOSTING.md`. |
| 813 | 5. `npm run test:deploy` and `node scripts/deploy.mjs manifest --check` |
| 814 | say what is missing. Then create its resources and secrets, and |
| 815 | `node scripts/deploy.mjs deploy --only <unit>`. |
| 816 | |
| 817 | ## The self-hosted runner |
| 818 | |
| 819 | `g1t-runner` (crates/runner) is also what customers run on their own |
| 820 | machines (guide: `apps/docs/src/content/docs/guides/self-hosted-runners.md`). |
| 821 | It is not deployed with the stack: it is released, and runners already out |
| 822 | there update themselves to each release. |
| 823 | |
| 824 | | Piece | Where | |
| 825 | | --- | --- | |
| 826 | | The tool | `scripts/runner-release.mjs` (`keygen`, `build`, `sign`, `verify`, `publish`) | |
| 827 | | The workflow | `.g1t/workflows/runner-release.yml`, on a tag `runner-v<version>` or by hand | |
| 828 | | Where it is published | The R2 bucket `g1t-downloads`, served by the site at `g1t.sh/downloads/runner/<version>/<file>` and `/latest/<file>` (`apps/web/app/routes/downloads-runner.ts`) | |
| 829 | | Its image | `deploy/runner/Dockerfile`, pushed to `g1t.sh/flagon-io/g1t-runner` (public) for amd64 and arm64 | |
| 830 | |
| 831 | A release is five binaries (Linux x64 and arm64, both static musl; macOS |
| 832 | x64 and arm64; Windows x64), `SHA256SUMS`, and `manifest.json`; |
| 833 | `latest.json` and its Ed25519 signature `latest.json.sig` name the newest. |
| 834 | A runner updates only to a release whose signature checks out against the |
| 835 | public key built into it and whose download matches the manifest's SHA-256. |
| 836 | |
| 837 | The first time: |
| 838 | |
| 839 | 1. `node scripts/runner-release.mjs keygen`. Put `RUNNER_RELEASE_KEY` in the |
| 840 | repository's secrets (production environment) and keep a copy offline; |
| 841 | put the public key in its variables as `RUNNER_RELEASE_PUBLIC_KEY` (names |
| 842 | starting `G1T_` are reserved; the workflow hands it to the build as |
| 843 | `G1T_RUNNER_RELEASE_KEY`). A build made without the public key never |
| 844 | updates itself. |
| 845 | 2. `npx wrangler r2 bucket create g1t-downloads`, and deploy the site so it |
| 846 | has the `DOWNLOADS` binding. |
| 847 | 3. The image job pushes to g1t's own registry with the run's `G1T_TOKEN`; |
| 848 | the `g1t-runner` package in flagon-io is public. Set the variable |
| 849 | `RUNNER_AGENT_IMAGE` (a public copy of `g1t-runner-base`, the image agent |
| 850 | work runs in on customers' runners) once there is one. |
| 851 | 4. Register a self-hosted runner with the `docker` label for the image job. |
| 852 | |
| 853 | Each release: |
| 854 | |
| 855 | 1. Bump `version` in `crates/runner/Cargo.toml` and merge it. |
| 856 | 2. Tag the commit `runner-v<version>` and push the tag. The workflow builds |
| 857 | every platform with cargo-zigbuild, signs, verifies, publishes the files |
| 858 | (the version's first, `latest.json` last), and pushes the image. |
| 859 | 3. By hand, the same is `node scripts/runner-release.mjs build`, then `sign`, |
| 860 | `verify` and `publish`, with the keys in the environment. |
| 861 | |
| 862 | Rolling back a release: copy the older version's `manifest.json` over |
| 863 | `runner/latest.json` and sign it again (`sign` after checking the older |
| 864 | version out). Runners never move to an older version on their own; a |
| 865 | runner on a bad release is fixed by the next good one, or by downloading |
| 866 | the older binary over it. |