flagon-io/g1t

public

g1t/docs/DEPLOYING.md

636 lines35,869 bytesCode Blame

Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.

Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	1	# Deploying g1t
	2
	3	How g1t.sh gets to Cloudflare: one manifest that lists every deployable
	4	unit, one tool that deploys only what changed, and a g1t Actions workflow
	5	that runs that tool on every push to `main`. Internal: the public
	6	self-hosting guide is `apps/docs/src/content/docs/guides/self-hosting.md`.
	7
	8	\| Piece \| Where \|
	9	\| --- \| --- \|
	10	\| The manifest \| `deploy/stack.jsonc` \|
	11	\| The tool \| `scripts/deploy.mjs` (library and tests in `scripts/deploy/`) \|
	12	\| Rust Worker builds \| `scripts/build-rust-worker.mjs`, every Rust unit's build command \|
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	13	\| The runner's images \| `services/runner/base/Dockerfile`, `services/runner/Dockerfile`, `services/runner/base.json`, `scripts/build-runner.mjs`, `scripts/deploy/image.mjs` \|
	14	\| The workflows \| `.g1t/workflows/deploy.yml`, `.g1t/workflows/runner-base.yml` \|
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	15	\| The old entry point \| `scripts/deploy.sh`, now a wrapper \|
	16
	17	## The manifest
	18
	19	`deploy/stack.jsonc` names every unit: each folder with a `wrangler.jsonc`.
	20	It is JSONC rather than TOML so that Node reads it with no dependency,
	21	with the same parser as the Wrangler configs, and comments stay possible.
	22
	23	\| Field \| \|
	24	\| --- \| --- \|
	25	\| `path` \| The unit's folder. \|
	26	\| `kind` \| `rust-worker` (built by worker-build), `ts-worker` (Wrangler bundles it), `react-router` (`vite build` first), `astro` (`astro build` first). \|
	27	\| `worker` \| The Worker's name. Must match its `wrangler.jsonc`. \|
	28	\| `d1` \| `{ database, migrations }`, when it has a database. Must match its `wrangler.jsonc`. \|
	29	\| `stage` \| `core`, `edge` or `front` (below). \|
	30	\| `secrets` \| The Wrangler secrets it needs, by name. `node scripts/deploy.mjs doctor` checks they are set. \|
	31	\| `setup` \| One-time steps no config can say, for a first deploy. \|
	32	\| `inputs` \| Files outside its folder it is built from that no workspace metadata names. A test finds such imports. \|
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	33	\| `image` \| A Containers image: `dockerfile` (the image deployed), `crate` (the binary it adds), `base` (`{ context, lock }`: the base image's folder and the file recording the base that was pushed) and `repository` (where both are pushed). See [the runner's images](#the-runners-images). \|
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	34	\| `self_host` \| `run`, `off`, `separate` or `none`: what `deploy/self-host/configs.mjs` does with it. \|
	35
	36	What a unit is built from is never listed by hand. The tool reads it:
	37	Rust path dependencies from `cargo metadata`, workspace packages from each
	38	`package.json`, closed transitively. `node scripts/deploy.mjs manifest`
	39	prints the result:
	40
	41	```
	42	unit stage kind worker d1 built from (besides its folder)
	43	events core rust-worker g1t-events g1t-events crates/contracts crates/kit
	44	repos core rust-worker g1t-repos g1t-repos crates/contracts crates/kit crates/scan crates/secrets
	45	runner core ts-worker g1t-runner crates/actions crates/runner packages/contracts
	46	og core ts-worker g1t-og packages/contracts
	47	web front react-router g1t packages/contracts packages/theme
	48	...
	49	```
	50
	51	Root files count too: `Cargo.toml`, `Cargo.lock` and
	52	`scripts/build-rust-worker.mjs` for Rust units; `package-lock.json` and
	53	`tsconfig.base.json` for the others; `package.json` for all. A lockfile
	54	change counts for a unit only if a package in that unit's graph changed,
	55	read from the lockfile itself (`scripts/deploy/lockfiles.mjs`), so bumping
	56	`sharp` for the docs does not redeploy the Rust services.
	57
	58	### Stages
	59
	60	\| Stage \| What \| Units \|
	61	\| --- \| --- \| --- \|
	62	\| `migrations` \| Every pending D1 migration, in parallel, before any code \| each unit's `d1` \|
	63	\| `core` \| Services reached through bindings \| the services, `og` \|
	64	\| `edge` \| Public endpoints other than the site \| `api`, `models`, `pages`, `status` \|
	65	\| `front` \| The site, sudo, the docs \| `web`, `sudo`, `docs` \|
	66
	67	A stage starts only when the one before it succeeded. Inside a stage units
	68	deploy in parallel. The rule the tests enforce: **a unit binds only to units
	69	in its own stage or an earlier one**, so new code never calls a service
	70	that has not shipped. (Services in `core` bind to each other in cycles,
	71	which is why they share a stage.)
	72
	73	### What reads the manifest
	74
	75	- `scripts/deploy.mjs`: everything below.
	76	- `deploy/self-host/configs.mjs`: which Workers a self-hosted installation
	77	runs (`self_host: "run"`, the site first) and which are bound to the off
	78	Worker (`"off"`). Its output is identical to before, apart from the order
	79	of `workers.txt` after the site.
	80	- Tests (`npm run test:deploy`) check that: every `wrangler.jsonc` in the
	81	repository has a unit; `worker`, `d1` and `image` match the configs;
	82	every KV id is named under `resources.kv`; stages follow bindings; the
	83	derived dependencies agree with Cargo's own resolved graph; every import
	84	that leaves a unit's folder is covered; `deploy/self-host/Dockerfile`
	85	builds exactly the Rust units self-hosting runs; and the service table in
	86	`docs/SELF_HOSTING.md` names every unit.
	87
	88	## The tool
	89
	90	```sh
	91	node scripts/deploy.mjs plan # what would deploy, and why (read-only)
	92	node scripts/deploy.mjs deploy # migrations, then every changed unit
	93	node scripts/deploy.mjs deploy --only web,api # just these, if they changed
	94	node scripts/deploy.mjs deploy --only web --force # just this, changed or not
	95	node scripts/deploy.mjs deploy --all # everything
	96	node scripts/deploy.mjs build --only events # build as a deploy would; upload nothing
	97	node scripts/deploy.mjs migrate # pending migrations only
	98	node scripts/deploy.mjs manifest [--check\|--json]
	99	node scripts/deploy.mjs doctor # secrets each unit lacks
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	100	node scripts/deploy.mjs build-base # build and push the runner's base image (Docker)
	101	node scripts/deploy.mjs image # build and push the runner's image for this checkout (Docker)
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	102	scripts/deploy.sh [units...] # the old entry point: all, or those named, always
	103	```
	104
	105	\| Flag \| \|
	106	\| --- \| --- \|
	107	\| `--only a,b` / `--skip a,b` \| Units by short name, folder or Worker name. \|
	108	\| `--all` \| Every unit, changed or not. \|
	109	\| `--force` \| Deploy the selected units even if unchanged. \|
	110	\| `--concurrency N` \| Units at once inside a stage, and migrations at once (default 4). \|
	111	\| `--stage core` \| One stage only. \|
	112	\| `--no-migrations` \| Skip the migrations step (the workflow runs it as its own job). \|
	113	\| `--allow-dirty` \| Deploy with uncommitted changes in what deploys. The version records no commit, so the next plan deploys it again. \|
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	114	\| `--rebuild-image` \| Build the runner's image even if one for this source is already in the registry. \|
	115	\| `--rebuild-base` \| Build and push a new base first (needs Docker), then the runner's image on it. Writes `services/runner/base.json`: commit it. \|
	116	\| `--no-push` \| `build-base` and `image`: build locally, push nothing. \|
	117	\| `--no-cache` \| `build-base`: build every layer again. \|
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	118	\| `--since REV` \| Treat Workers with no recorded commit as running `REV`. Used once to adopt Workers deployed before this tool. \|
	119	\| `--json`, `--out FILE`, `--github-output` \| The plan as data, for the workflow. \|
	120
	121	### Where the deployed commit is kept
	122
	123	On the Worker itself. Every deploy runs `wrangler deploy --message
	124	"g1t-deploy <40-char sha> <subject>" --tag g1t-<12-char sha>`, which
	125	Cloudflare keeps as the version's `workers/message` and `workers/tag`
	126	annotations. `plan` reads them back with `wrangler deployments status`
	127	(the live version) and `wrangler versions list` (its annotations): two
	128	read-only calls per unit, in parallel; a plan of all 22 units takes about
	129	10 seconds. No KV namespace or other infrastructure is needed.
	130
	131	- A version made by `wrangler secret put` keeps the code of the one before
	132	it, so the tool looks through those to the deploy before.
	133	- A version deployed any other way (by hand, from the dashboard) has no
	134	commit, and the unit is deployed again.
	135	- During a gradual rollout the version with the most traffic counts.
	136	- After `wrangler rollback`, the plan sees the older commit and deploys
	137	what changed since.
	138
	139	### What a deploy does
	140
	141	1. Plans: for each unit, the live commit; `git diff` from it to `HEAD`;
	142	whether the changed files touch the unit (its folder, the crates and
	143	packages it is built from, its inputs, lockfile changes that reach it).
	144	2. Refuses if uncommitted changes touch what would deploy (`--allow-dirty`).
	145	3. Applies every pending migration (`wrangler d1 migrations apply --remote`),
	146	in parallel. Any failure stops the deploy before code.
	147	4. Installs worker-build once if any Rust unit is deploying.
	148	5. Each stage in turn: units in parallel (`--concurrency`), each `npm run
	149	build` first for React Router and Astro, then `wrangler deploy` in the
	150	unit's folder with the annotation. If any unit fails, later stages are
	151	not started.
	152	6. Prints a table: unit, stage, result, version id, time. Each unit's full
	153	output is kept in `$TMPDIR/g1t-deploy/<unit>.log`.
	154
Deploying: a migration keeps the live code right, because it runs a minute before the new code18h ago	155	#### Migrations run while the old code is still live
	156
	157	Migrations apply before any code, and a stage takes a minute or more, so
	158	for that long the code in production is the old code reading the new
	159	schema. A migration must keep the old code right:
	160
	161	- Add, never change meaning. New tables and columns (with defaults) are
	162	safe. Rewriting what existing rows mean is not: on 2026-10-08,
	163	`billing/0039` turned comped accounts into `custom` terms at 100%, and
	164	for about 40 seconds the old billing code, which knew only `comped`, saw
	165	flagon-io as a free workspace and refused its workflows.
	166	- Change meaning in two deploys. First ship code that reads both the old
	167	and the new form (and keeps writing the old one); then, in a later
	168	deploy, the migration that rewrites the rows; then, if you like, code
	169	that drops the old form.
	170	- Never drop or rename a column or table the live code reads in the same
	171	deploy that stops reading it.
	172
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	173	#### Telling the status page about a deploy
	174
	175	Restarts during a deploy can make a part slow for a minute, which the
	176	status page's checks would otherwise draft as an incident. Before the
	177	first stage and after the last, a deploy can say so with
	178	`scripts/deploy/status-window.mjs` (`announceDeploy("started" \| "finished",
	179	{ id })`, or `node scripts/deploy/status-window.mjs started\|finished [id]`).
	180	It posts to `POST https://status.g1t.sh/deploys` with
	181	`Authorization: Bearer $STATUS_DEPLOY_TOKEN`, the same value as the status
	182	Worker's `STATUS_DEPLOY_TOKEN` secret. During the deploy and for 3 minutes
	183	after it, detection keeps counting failed and slow checks but makes no new
	184	draft; trouble that outlasts that is drafted with its true start. A
	185	start with no finish stops counting after 30 minutes. Without the token
	186	the helper does nothing, and it never fails a deploy.
	187
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	188	On a laptop the tool uses your `wrangler login` (or `CLOUDFLARE_DEPLOY_TOKEN`
	189	if set), as `scripts/deploy.sh` always did: a `CLOUDFLARE_API_TOKEN` or
	190	global API key in your shell, or in the repository's `.env`, is ignored.
	191	With `CI=true` it uses `CLOUDFLARE_API_TOKEN`.
	192
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	193	### The runner's images
	194
	195	`services/runner` runs every sandbox (agents, checks, the merge queue,
	196	workflow jobs, g1t.page builds) from one Containers image, made in two
	197	parts:
	198
	199	\| Image \| Built from \| Holds \| Rebuilt \|
	200	\| --- \| --- \| --- \| --- \|
	201	\| Base, `g1t-runner:base-<date>-<inputs>` \| `services/runner/base/Dockerfile` \| Debian bookworm, Node 24, Python 3.11, Go (from go.dev), Rust stable for the `node` user with rustfmt, clippy and the `wasm32-unknown-unknown` target, build-essential, git, ripgrep, jq, zstd, sudo, and the pinned Claude Code CLI on top \| When its folder changes, weekly, or by hand (`build-base`) \|
	202	\| Runner, `g1t-runner:<content hash>` \| `services/runner/Dockerfile`: `FROM` the base, plus one file \| The g1t runner, a static binary \| When the binary or the base changes \|
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	203
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	204	Both are pushed to one repository of Cloudflare's registry,
	205	`registry.cloudflare.com/<account>/g1t-runner`, so pushing the runner's
	206	image uploads only its own layer (about 5 MB): the base's layers are
	207	already there.
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	208
The runner's base image, pushed as Wrangler builds images, with pushes that fail loudly2d ago	209	Both are built as Wrangler builds images (`--platform linux/amd64
	210	--provenance=false --sbom=false`): one manifest, not an OCI index with a
	211	BuildKit attestation beside it. A push is tried up to three times and counts
	212	only when Docker reports the digest; the first push of the base once failed
	213	with `blob unknown to registry` and went through when run again (see
	214	`docs/CLOUDFLARE_FEEDBACK.md`, C3). `build-base` and `image` exit non-zero
	215	when a push fails, and `base.json` is written only after the push.
	216
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	217	The base is recorded in `services/runner/base.json`: its reference, its
	218	digest, a hash of its folder (`inputs`), when it was built, its size and
	219	each toolchain's version. `node scripts/deploy.mjs build-base` builds it,
	220	pushes it and rewrites the file; commit the file, and the next deploy
	221	builds the runner's image on it. `npm run test:deploy` fails while the
	222	folder and the file disagree, so a change to the base's Dockerfile cannot
	223	merge without the base it describes. Layers go from what changes least to
	224	most (system packages, Go, Rust, the Claude Code CLI), and the apt and npm
	225	caches stay in BuildKit's cache, out of the image.
	226
	227	The base's build cache is the base itself: it is built with
	228	`BUILDKIT_INLINE_CACHE`, which records in the image how each layer was
	229	made, and `build-base` builds `--cache-from` the base it replaces. A
	230	machine with an empty cache, or one just pruned, pulls the unchanged
	231	layers from the registry instead of building them. (BuildKit's other
	232	registry cache, `--cache-to type=registry`, pushes a separate cache
	233	manifest that not every registry takes, and for a one-stage image adds
	234	nothing the inline cache lacks.)
	235
	236	The runner binary (`crates/runner`) is built outside Docker by
	237	`scripts/build-runner.mjs` as one static binary for
	238	`x86_64-unknown-linux-musl`, so it runs on the base whatever its libc, and
	239	on a self-hosted runner's machine too. Where it is built:
	240
	241	- on x86-64 Linux with the musl target and `musl-gcc`
	242	(`rustup target add x86_64-unknown-linux-musl`, `apt-get install musl-tools`),
	243	with the machine's own Cargo;
	244	- anywhere else (Windows, macOS) in a small builder container, Rust on
	245	Alpine (whose own target is musl), with Docker volumes keeping Cargo's
	246	registry and target directory between builds. Windows has no musl
	247	cross-linker, and `ring` (under ureq's TLS) needs a C compiler for the
	248	target, so a container is the dependable route.
	249
	250	The runner's image tag is a hash of everything it is built from: its
	251	Dockerfile, `base.json`, the crates the binary is built from, the
	252	workspace's Cargo files and the build script. The same source always names
	253	the same image, so:
	254
	255	1. A deploy computes the tag and asks the registry whether it is there
	256	(a `HEAD` of its manifest, with credentials from Wrangler; no Docker).
	257	2. If it is, nothing is built: the deploy uses it.
	258	3. If not, and Docker is here, it builds the binary and the image (seconds
	259	on a warm machine) and pushes it.
	260	4. If not, and Docker is not here (a g1t Actions sandbox), the unit fails
	261	saying to run `node scripts/deploy.mjs image` on a machine with Docker;
	262	then re-run the workflow.
	263
	264	Then `wrangler deploy` is given the image by reference, from a generated
	265	config (`services/runner/wrangler.deploy.json`, deleted after, ignored by
	266	git), so Wrangler builds nothing. When nothing the image is built from
	267	changed since the runner's live commit, the deploy also passes
	268	`--containers-rollout none`, which leaves running sandboxes alone.
	269
	270	To get a new base out:
	271
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	272	```sh
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	273	node scripts/deploy.mjs build-base # build, push, write base.json (needs Docker)
	274	git commit services/runner/base.json -m "A new base image for g1t's sandboxes"
	275	node scripts/deploy.mjs image # optional: push the runner's image now, so CI finds it
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	276	```
	277
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	278	`.g1t/workflows/runner-base.yml` does the same weekly (and when the
	279	base's folder changes on `main`), and opens a pull request with
	280	`base.json`. It needs Docker, so it runs on a self-hosted runner with the
	281	`docker` label (`runs-on: [self-hosted, docker]`). Until one is
	282	registered, its runs wait for one; run `build-base` by hand instead.
	283
	284	Sandboxes start from the image. Cloudflare pulls an image to a
	285	machine the first time a sandbox lands there, and keeps it. A smaller base
	286	pulls sooner, and a change to the runner alone sends machines one 5 MB
	287	layer instead of the whole image.
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	288
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	289	#### Larger machines
	290
	291	The same image runs on three instance types, each a Durable Object class
	292	of its own in `services/runner/wrangler.jsonc`: `AttemptSandbox`
	293	(`standard-1`), `Sandbox2Core` (`standard-3`) and `Sandbox4Core`
	294	(`standard-4`). Workflow jobs choose with `runs-on: g1t-2core` or
	295	`g1t-4core` (`g1t_contracts::actions::INSTANCE_TYPES`); the actions
	296	service passes the label to the runner, which starts the job in that
	297	class. Billing prices the larger ones from their memory and disk, and
	298	their CPU (see the public billing guide). The account's Containers limits
	299	must allow `standard-4`; Wrangler refuses the deploy otherwise.
	300
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	301	## Build speed
	302
	303	Measured on the development machine (Windows, 32 cores, warm Cargo cache),
	304	building `events`, `search` and `repos` after a change to `crates/kit`, as a
	305	deploy does but without uploading (`wrangler deploy --dry-run`):
	306
	307	\| \| Time \|
	308	\| --- \| --- \|
	309	\| Before: one after another, `cargo install worker-build` each time, wasm-opt `-O` \| 81 s, 84 s \|
	310	\| Concurrent builds, wasm-opt `-O` \| 50 s, 68 s \|
	311	\| Concurrent builds, wasm-opt `-O1` (now) \| 28 s, 33 s \|
	312
	313	Where the time went, and what changed:
	314
	315	- wasm-opt was most of it: `-O` took 38 s on `repos` and 19 s on `api`;
	316	`-O1` takes 1 to 3 s. With worker-build's flags (it keeps the names
	317	section) the `.wasm` is 24 to 28% larger raw but only 1 to 7% larger
	318	gzipped, and Workers' size limit is on the compressed upload. `-Os` and
	319	`-Oz` were no faster than `-O`. Set per crate in
	320	`[package.metadata.wasm-pack.profile.release]`; a test keeps every Rust
	321	unit on the same level.
	322	- worker-build is installed only when missing or another version
	323	(`scripts/build-rust-worker.mjs`, which pins it). Locally `cargo install`
	324	on an installed version cost under a second; on a fresh CI sandbox it is
	325	a full compile, which the workflow caches instead.
	326	- One Cargo target: every Rust unit is a member of the workspace, so
	327	they already share `target/`. Concurrent builds take turns on Cargo's
	328	lock for the compile, and their wasm-bindgen, wasm-opt and uploads
	329	overlap.
	330	- No joint `cargo build -p a -p b`: tried, and it is slower. Cargo
	331	unifies features across the packages of one build (`serde_json`'s
	332	`preserve_order` from `api` and `actions`, `digest` features from
	333	`secrets`), so each worker-build afterwards compiled its own variant
	334	again.
The runner's base image, pushed as Wrangler builds images, with pushes that fail loudly2d ago	335	- The runner's images, measured on the same machine on 2026-10-06
	336	(Docker Desktop, 8 vCPUs; its disk was busy with other containers, so
	337	the cold figures are slow and noisy):
	338
	339	\| \| Before (one image) \| Now \|
	340	\| --- \| --- \| --- \|
	341	\| Size, unpacked / compressed (what a machine pulls) \| 3.08 GB / 819 MB \| 2.68 GB / 686 MB \|
	342	\| A change to the runner \| Docker rebuilds the image's Rust stage and pushes the image (1198 s in the first deploy after a prune) \| binary 46–53 s (6 s unchanged), image 7 s, push one 5 MB layer (1.4 s to a local registry) \|
	343	\| The base from nothing \| 431 s (whole image, cold) \| 758 s cold, rarely: weekly or when its folder changes \|
	344	\| The base after `docker builder prune` \| as from nothing \| 68 s, its layers pulled from the registry it was pushed to \|
	345	\| The runner binary, cold (builder container) \| \| 99 s \|
	346	\| A Rust CI job's build (events, search, repos; 4 vCPUs) \| \| 51 s cold, 13 s with the Cargo target restored (107 MB zstd entry) \|
	347
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	348	- Only what changed is the largest saving: a change to one service
	349	deploys one service.
	350
	351	## The workflow
	352
	353	`.g1t/workflows/deploy.yml` runs on every push to `main`, and by hand
	354	(Actions → Deploy → Run workflow) with `units` (deploy these, changed or
	355	not), `all` and `dry_run` (plan only).
	356
	357	\| Job \| Does \| Needs \|
	358	\| --- \| --- \| --- \|
	359	\| `check` \| `manifest --check` and `npm run test:deploy` \| — \|
	360	\| `plan` \| `plan --github-output`: outputs per stage, the plan in the run's summary \| `check` \|
	361	\| `migrate` \| `migrate --only <units with pending migrations>` \| `plan`; skipped when none are pending \|
	362	\| `core`, `edge`, `front` \| `deploy --only <units> --force --no-migrations`, one job per build group \| the stages before; skipped when empty \|
	363
	364	- One at a time: `concurrency: deploy-production`, never cancelled in
	365	progress; a second push waits.
	366	- Build groups: a stage's units are split so each job shares a build:
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	367	Rust workers at most four to a job (each a 4-vCPU `g1t-4core` machine), the
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	368	TypeScript Workers together, each site alone, and a unit whose image must
	369	be rebuilt alone. `fail-fast: false`, so one failed job does not cut
	370	another off mid-upload; the next stage then does not start.
	371	- Tests: there is no CI workflow on g1t yet; `main` is kept passing by
	372	the merge queue's checks. `check` runs the deploy tool's own tests. When a
	373	CI workflow is added, make `plan` wait for it (`workflow_run`, or a job in
	374	this file).
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	375	- Machines: Rust jobs run on `g1t-4core` (4 vCPUs, 12 GiB), the
	376	others on the standard machine (`runs-on: ${{ matrix.rust && 'g1t-4core' \|\| 'ubuntu-latest' }}`).
	377	- Caching (`actions/cache`: up to 2 GB an entry, 10 GB a repository,
	378	kept until unused for 7 days): the worker-build binary, worker-build's
	379	downloaded tools, `~/.cargo/registry/cache`, and the Cargo target's
	380	release dependencies (`target/release` and
	381	`target/wasm32-unknown-unknown/release`, without `incremental` or
	382	`.wasm`), keyed by the build group, `Cargo.lock` and `base.json`. The
	383	workspace's own crates are compiled again on every run (a checkout's
	384	sources are newer than any cache); the crates.io dependencies are not.
	385	npm's cache is not kept: every job runs `npm ci` of only what its units
	386	need (`deploy.mjs install`: Wrangler alone for Rust jobs).
	387	- Conditions: each stage runs with `!failure() && !cancelled()`, which
	388	on g1t (as on GitHub) is true when no job before it failed, however far
	389	back: a `migrate` job skipped for having nothing to apply does not stop
	390	the stages after it, and a failed `check` stops all of them.
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	391	- `crates/actions/tests/repository_workflows.rs` reads the workflow with
	392	g1t's own parser and expressions, and checks the jobs start, wait and
	393	stop as above (`cargo test -p g1t-actions --test repository_workflows`).
	394
	395	### What the sandbox has
	396
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	397	The base image (`services/runner/base/Dockerfile`) has Node 24, npm, git,
	398	Go, zstd, and Rust stable for the `node` user with rustfmt, clippy and the
	399	`wasm32-unknown-unknown` target, but not worker-build or Docker. The
	400	workflow's `rustup target add wasm32-unknown-unknown` is then a no-op, and
	401	worker-build is restored from the cache, installed on a miss. worker-build
	402	fetches wasm-bindgen and wasm-opt from GitHub releases and esbuild from
	403	npm. All of those hosts are on the list every workflow job may reach.
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	404
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	405	### Network
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	406
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	407	A workflow job reaches its project's allowed domains, g1t, what builds
	408	need (`services/runner/src/egress.ts`, `BUILD_HOSTS`), and the project's
	409	workflow-only domains that name its workflow and environment. Those are
	410	never reached by agents, checks, the merge queue, deploy builds or runs of
	411	pull requests from forks (`Guardrails::workflow_hosts`, the runner's
	412	`jobHosts`). Under flagon-io/g1t's Settings → Guardrails
	413	(Maintain role or higher), Workflow-only domains:
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	414
The runner's base image, pushed as Wrangler builds images, with pushes that fail loudly2d ago	415	```text
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	416	api.cloudflare.com \| deploy.yml \| production
	417	registry.cloudflare.com \| deploy.yml, runner-base.yml \| production
	418	```
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	419
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	420	`api.cloudflare.com` is Wrangler's API; `registry.cloudflare.com` is where
	421	the deploy asks whether the runner's image is already built (and where
	422	`runner-base.yml` pushes). Only `deploy.yml`'s and `runner-base.yml`'s
	423	jobs with `environment: production` reach them, which are also the only
	424	jobs that can read `CLOUDFLARE_API_TOKEN`. Each change to the list is in
	425	the workspace's audit log as `update_guardrails`.
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	426
	427	### The API token
	428
	429	Create it at **dash.cloudflare.com → My Profile → API Tokens → Create
	430	Token → Custom token**, named `g1t deploys (CI)`:
	431
	432	\| Scope \| Permission \| Why \|
	433	\| --- \| --- \| --- \|
	434	\| Account \| Workers Scripts: Edit \| Upload, versions, deployments, crons, bindings, `secret list` (doctor) \|
	435	\| Account \| D1: Edit \| `d1 migrations list` and `apply` \|
	436	\| Account \| Queues: Edit \| Attaching each unit's queue consumers on deploy \|
	437	\| Account \| Workers R2 Storage: Read \| Wrangler checks `og`'s bucket binding \|
	438	\| Account \| Account Settings: Read \| Wrangler reads the account \|
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	439	\| Account \| Containers: Edit \| The runner's deploy updates its applications (the image reference, the three classes), and gets registry credentials to look for its image. `runner-base.yml` pushes images with it. \|
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	440	\| Zone (`g1t.sh`, `g1t.page`) \| Workers Routes: Edit \| `pages`' zone routes, and custom domains \|
	441	\| Zone (`g1t.sh`, `g1t.page`) \| DNS: Edit \| Custom domains (`api`, `mcp`, `og`, `models`, `status`, `sudo`, `docs`, `g1t.sh`, `g1t.page`) keep their DNS records \|
	442	\| Zone (`g1t.sh`, `g1t.page`) \| Zone: Read \| Finding the zone a route names \|
	443
	444	Restrict it to account `syntaqx` (`1e6f2cffa3f445920836e8ebe446bb58`) and
	445	the two zones. Not needed for deploys: KV (bindings are by id; creating a
	446	namespace is a one-time setup), Vectorize, Workers for Platforms beyond
	447	Workers Scripts, Cloudflare for SaaS custom hostnames (the deployments
	448	service does that at runtime with its own token), SSL and Certificates.
	449	Workers KV Storage: Edit and Vectorize: Edit are only for first-time setup,
	450	which stays a person's job.
	451
	452	The list follows what our configs use; Cloudflare does not publish exactly
	453	what `wrangler deploy` checks for each binding. Bindings with no listed
	454	permission (Browser Rendering, Workers AI, Vectorize, Email Sending,
	455	Artifacts, dispatch namespaces) are assumed to need none beyond Workers
	456	Scripts. Verify on the first run with `workflow_dispatch` and `units:
	457	pages` (small, no secrets), then `units: og` (R2) and `units: runner`
	458	(Containers); a missing permission fails with `Authentication error [code:
	459	10000]` and the route it was refused.
	460
	461	Add it to the repository:
	462
	463	1. On g1t.sh, open flagon-io/g1t → Settings → Secrets and variables.
	464	2. Add: key `CLOUDFLARE_API_TOKEN`, type Secret, available to
	465	Workflows, environment Production. Only jobs with
	466	`environment: production` (the deploy jobs) can read it.
	467	3. Add: key `CLOUDFLARE_ACCOUNT_ID`, type Variable, value
	468	`1e6f2cffa3f445920836e8ebe446bb58`, available to Workflows, all
	469	environments.
	470
	471	Or through the API:
	472
	473	```sh
	474	curl -X PUT https://api.g1t.sh/repos/flagon-io/g1t/actions/secrets/CLOUDFLARE_API_TOKEN \
	475	-H "Authorization: Bearer $G1T_TOKEN" -H "Content-Type: application/json" \
	476	-d '{"value":"<the token>","environments":["production"],"available_to":["workflows"]}'
	477
	478	curl -X POST https://api.g1t.sh/repos/flagon-io/g1t/actions/variables \
	479	-H "Authorization: Bearer $G1T_TOKEN" -H "Content-Type: application/json" \
	480	-d '{"name":"CLOUDFLARE_ACCOUNT_ID","value":"1e6f2cffa3f445920836e8ebe446bb58","available_to":["workflows"]}'
	481	```
	482
	483	## Turning it on
	484
	485	1. Create the token and add the secret and variable (above).
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	486	2. Add `api.cloudflare.com` and `registry.cloudflare.com` to flagon-io/g1t's
	487	workflow-only domains, for `deploy.yml` in `production` (above).
	488	3. Build and push the base once, and commit `services/runner/base.json`:
	489	`node scripts/deploy.mjs build-base`. Create the cache bucket:
	490	`npx wrangler r2 bucket create g1t-actions-cache`, with a lifecycle rule
	491	deleting objects 30 days after upload
	492	(`npx wrangler r2 bucket lifecycle add g1t-actions-cache expire --expire-days 30 --abort-multipart-days 1`).
	493	4. Adopt the live Workers once, from a laptop: deploy everything with the
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	494	tool so each version records its commit (`scripts/deploy.sh`, or
	495	`node scripts/deploy.mjs deploy --all`). Until then every plan says "no
	496	known commit" and deploys every unit. To see what has changed since a
	497	commit you know production runs, without deploying:
	498	`node scripts/deploy.mjs plan --since <sha>`.
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	499	5. Run the workflow by hand with `dry_run`, then with `units: pages`.
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	500
	501	## First deploy of a new account
	502
	503	What the configs refer to must exist first. `node scripts/deploy.mjs
	504	manifest --json` lists, per unit, its queues, KV, R2, Vectorize and
	505	dispatch namespaces; each unit's `setup` and `secrets` say the rest.
	506
	507	- D1: `npx wrangler d1 create <database>`, then put its id in the unit's
	508	`wrangler.jsonc`.
	509	- KV: `npx wrangler kv namespace create <name>` for each name under
	510	`resources.kv`, then the ids in the configs.
	511	- Queues: `npx wrangler queues create <queue>` for each queue in the
	512	manifest: `g1t-events`, `g1t-events-<service>` for every subscriber,
	513	`g1t-search-jobs`, `g1t-context-jobs`.
Merge branch 'worktree-agent-a1b995daa94e4e1b7'1d ago	514	- R2: `npx wrangler r2 bucket create g1t-screenshots`,
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	515	`npx wrangler r2 bucket create g1t-actions-cache` (with its 30-day
Merge branch 'worktree-agent-a1b995daa94e4e1b7'1d ago	516	lifecycle rule, above), and `npx wrangler r2 bucket create g1t-git-packs`,
	517	the clone pack cache (`services/repos/src/pack_cache.rs`), with a rule
	518	deleting packs 7 days after they were written and unfinished uploads
	519	after a day:
	520	`npx wrangler r2 bucket lifecycle add g1t-git-packs expire-packs packs/ --expire-days 7 --abort-multipart-days 1`.
	521	A Worker bound to a bucket that does not exist fails to deploy, so make
	522	it before the first deploy of `g1t-repos` that binds it.
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	523	- The runner's base image: `node scripts/deploy.mjs build-base`.
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	524	- Vectorize, dispatch namespace, DNS, Access, Email Sending, Artifacts: each
	525	unit's `setup`.
	526	- Secrets: `npx wrangler secret put <NAME>` in the unit's folder;
	527	`node scripts/deploy.mjs doctor` lists what is missing.
	528
	529	Then `node scripts/deploy.mjs deploy --all`. A Worker bound to a service
	530	that does not exist yet may be refused; deploy that service first with
	531	`--only`.
	532
Deploying: a deploy by hand shows on the repository's Deployments page15h ago	533	## Deployments on g1t
	534
	535	Every deploy shows on the repository's Deployments page, so its production
	536	card says which commit runs.
	537
	538	- From the workflow, the deploy jobs name `environment: {name:
	539	production, url: https://g1t.sh}`, and g1t Actions records one production
	540	deployment per run.
	541	- By hand, `deploy` reports one itself: in progress once migrations are
	542	in, then success or failure. It needs a g1t token with `deployments:write`
	543	in `G1T_DEPLOY_TOKEN` or the file `.credentials/g1t-deploy-token`; without
	544	one, or from a dirty tree or a dry run, it sends nothing. It finds the
	545	repository from the git remote on g1t.sh. A report that fails is one line
	546	in the log and never fails the deploy.
	547
	548	When CI cannot finish a deploy, for example a runner image that must be
	549	built (hosted runners have no Docker), deploy from a machine with Docker.
	550	The report records it, and production shows the commit that really runs.
	551
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	552	## Rolling back
	553
	554	- One unit, at once: `npx wrangler rollback` in its folder (or
	555	`npx wrangler rollback <version-id>`; `npx wrangler versions list` shows
	556	each version's commit in its message). Code only: D1 migrations are not
	557	undone. The next plan sees the older commit and deploys what changed since,
	558	so revert the commit on `main` too, or the next push brings it back.
	559	- To a commit: check it out and `node scripts/deploy.mjs deploy --only
Deploying: never roll production back by accident17h ago	560	<units> --rollback`. Without `--rollback` the tool refuses any unit whose
	561	live commit is newer than the one checked out, even with `--force`, so a
	562	re-run of an old workflow run (or an old checkout) never rolls production
	563	back by accident. Migrations never run backwards: a migration that needs
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	564	undoing is a new migration.
	565	- The runner's image: a rollback of the Worker does not roll back the
Deploying: never roll production back by accident17h ago	566	container image. Redeploy the older commit (`--only runner --rollback`): its
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	567	image's tag is the hash of that commit's source, which is still in the
	568	registry, so nothing is built. A bad base is undone by reverting the
	569	commit that changed `services/runner/base.json`.
Deploys as code: a manifest of every Worker, a deploy tool that ships only what changed in parallel stages, and a g1t Actions workflow2d ago	570
	571	## Adding a unit
	572
	573	1. Make its folder with a `wrangler.jsonc` (and its D1 migrations, if any).
	574	A Rust Worker is a workspace member in the root `Cargo.toml` with
	575	`"build": { "command": "node ../../scripts/build-rust-worker.mjs" }` and
	576	the `wasm-opt = ["-O1"]` metadata; a TypeScript one is an npm workspace.
	577	2. Add it to `deploy/stack.jsonc`: path, kind, worker, `d1`, stage (the
	578	earliest stage after everything it binds to), secrets, setup, self_host.
	579	Name any new KV id under `resources.kv`.
	580	3. If its sources import a file outside its folder that is not a workspace
	581	crate or package, list it under `inputs`.
	582	4. Add a row to the service table in `docs/SELF_HOSTING.md`.
	583	5. `npm run test:deploy` and `node scripts/deploy.mjs manifest --check`
	584	say what is missing. Then create its resources and secrets, and
	585	`node scripts/deploy.mjs deploy --only <unit>`.
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	586
	587	## The self-hosted runner
	588
	589	`g1t-runner` (crates/runner) is also what customers run on their own
	590	machines (guide: `apps/docs/src/content/docs/guides/self-hosted-runners.md`).
	591	It is not deployed with the stack: it is released, and runners already out
	592	there update themselves to each release.
	593
	594	\| Piece \| Where \|
	595	\| --- \| --- \|
	596	\| The tool \| `scripts/runner-release.mjs` (`keygen`, `build`, `sign`, `verify`, `publish`) \|
	597	\| The workflow \| `.g1t/workflows/runner-release.yml`, on a tag `runner-v<version>` or by hand \|
	598	\| Where it is published \| The R2 bucket `g1t-downloads`, served by the site at `g1t.sh/downloads/runner/<version>/<file>` and `/latest/<file>` (`apps/web/app/routes/downloads-runner.ts`) \|
The runner's image is g1t.sh/flagon-io/g1t-runner, on g1t's own registry, public1d ago	599	\| Its image \| `deploy/runner/Dockerfile`, pushed to `g1t.sh/flagon-io/g1t-runner` (public) for amd64 and arm64 \|
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	600
	601	A release is five binaries (Linux x64 and arm64, both static musl; macOS
	602	x64 and arm64; Windows x64), `SHA256SUMS`, and `manifest.json`;
	603	`latest.json` and its Ed25519 signature `latest.json.sig` name the newest.
	604	A runner updates only to a release whose signature checks out against the
	605	public key built into it and whose download matches the manifest's SHA-256.
	606
	607	The first time:
	608
	609	1. `node scripts/runner-release.mjs keygen`. Put `RUNNER_RELEASE_KEY` in the
	610	repository's secrets (production environment) and keep a copy offline;
g1t-runner 0.1.0 is released: signed binaries for five platforms at g1t.sh/downloads/runner; the Artifacts checks' results2d ago	611	put the public key in its variables as `RUNNER_RELEASE_PUBLIC_KEY` (names
	612	starting `G1T_` are reserved; the workflow hands it to the build as
	613	`G1T_RUNNER_RELEASE_KEY`). A build made without the public key never
	614	updates itself.
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	615	2. `npx wrangler r2 bucket create g1t-downloads`, and deploy the site so it
	616	has the `DOWNLOADS` binding.
The runner's image is g1t.sh/flagon-io/g1t-runner, on g1t's own registry, public1d ago	617	3. The image job pushes to g1t's own registry with the run's `G1T_TOKEN`;
	618	the `g1t-runner` package in flagon-io is public. Set the variable
	619	`RUNNER_AGENT_IMAGE` (a public copy of `g1t-runner-base`, the image agent
	620	work runs in on customers' runners) once there is one.
Fast pages, required checks on the branch, self-hosted runners, honest incidents2d ago	621	4. Register a self-hosted runner with the `docker` label for the image job.
	622
	623	Each release:
	624
	625	1. Bump `version` in `crates/runner/Cargo.toml` and merge it.
	626	2. Tag the commit `runner-v<version>` and push the tag. The workflow builds
	627	every platform with cargo-zigbuild, signs, verifies, publishes the files
	628	(the version's first, `latest.json` last), and pushes the image.
	629	3. By hand, the same is `node scripts/runner-release.mjs build`, then `sign`,
	630	`verify` and `publish`, with the keys in the environment.
	631
	632	Rolling back a release: copy the older version's `manifest.json` over
	633	`runner/latest.json` and sign it again (`sign` after checking the older
	634	version out). Runners never move to an older version on their own; a
	635	runner on a bad release is fixed by the next good one, or by downloading
	636	the older binary over it.

This file's history is long; its oldest lines are credited to the oldest commit read.