flagon-io/g1t

public

Merge branch 'worktree-agent-af58ac8933b0dd125'

syntaqxcommitted 3h agoParentsb7ac05c 8b28352Browse files

16 files+211−1240/16 viewed

180	180		Then open http://localhost:8787 and sign up. The confirmation mail is in
181	181		Mailpit at http://localhost:8025. Repositories, push and clone, issues,
182	182		pull requests and code browsing work, and the API and MCP server answer at
183		−	http://localhost:8789; agents, deployments and context search are off in
184		−	this version.
	183	+	http://localhost:8789; packages and container images are kept in the
	184	+	bundled S3-compatible store, RustFS. Agents, deployments and context search
	185	+	are off in this version.
185	186		[docs/SELF_HOSTING.md](docs/SELF_HOSTING.md) says what works and what is next.
186	187
187	188		### On Cloudflare

+37−9View BlameViewed

17	17		\| --- \| --- \|
18	18		\| Sign up, sign in, email confirmation \| Works. Mail goes to the bundled Mailpit inbox. \|
19	19		\| Workspaces, members, access tokens \| Works \|
20		−	\| Repositories: create, push and clone over HTTP, browse code, commits \| Works. A fresh clone's pack is kept in the bundled MinIO, so the next clone of the same commit is served from there. \|
	20	+	\| Repositories: create, push and clone over HTTP, browse code, commits \| Works. A fresh clone's pack is kept in the bundled object store, so the next clone of the same commit is served from there. \|
21	21		\| Issues, comments, labels \| Works \|
22	22		\| Pull requests from a branch or from a fork, merged onto the default branch \| Works, when the pull request is up to date with the default branch. Bringing one up to date first needs g1t's agent, which is off. \|
23	23		\| The merge queue \| Takes pull requests and shows them waiting. Testing and landing them needs g1t's agent, which is off: take a pull request out of the queue, or turn the queue off, to merge it. \|
24	24		\| [The REST API](/reference/api/), OAuth and [MCP](/reference/mcp/) \| Work, on a port of their own: `http://localhost:8789`, with the MCP server at `http://localhost:8789/mcp` \|
25		−	\| [Container images](/guides/containers/): `docker login`, push and pull at your `PUBLIC_URL` \| Works, kept in the bundled MinIO, with no limit on a layer's size or on pulls \|
	25	+	\| [Container images](/guides/containers/): `docker login`, push and pull at your `PUBLIC_URL` \| Works, kept in the bundled object store, with no limit on a layer's size or on pulls \|
26	26		\| Site search \| Works \|
27	27		\| A status page of your own \| Works, at `http://localhost:8788` ([below](#the-status-page)) \|
28	28		\| Webhooks, integrations \| Work, retries included \|
⋯
116	116		the site.
117	117		2. It makes an access token and calls the API, the OAuth metadata and the
118	118		MCP server with it.
119		−	3. It opens a pull request from a branch and one from a fork, through the
	119	+	3. It publishes an npm package to your installation's registry and
	120	+	installs it back.
	121	+	4. It opens a pull request from a branch and one from a fork, through the
120	122		API, and merges both onto `main`.
121		−	4. It turns the merge queue on, merges a pull request into it, takes it
	123	+	5. It turns the merge queue on, merges a pull request into it, takes it
122	124		out again, and merges it with the queue off.
123	125
124	126		```sh
⋯
135	137		```
136	138
137	139		It prints `All checks passed` when every step worked. It needs `curl`,
138		−	`git` and `node`.
	140	+	`git`, `node` and `npm`; `PACKAGES=off` skips the npm package.
139	141
140	142		## Settings
141	143
⋯
155	157		\| `REGISTRATION_MODE` \| `open` \| `open`: anyone can make an account. `invite`: every new account needs an [invite](/guides/authentication/#invites), as on g1t.sh. \|
156	158		\| `INVITES_PER_USER` \| `5` \| How many invites each person can have out, while `REGISTRATION_MODE` is `invite` \|
157	159		\| `WAITLIST_NOTIFY_EMAIL` \| (none) \| Where a summary of new access requests goes, at most every 15 minutes. Empty sends none; requests still wait for you in the database. \|
158		−	\| `S3_ENDPOINT`, `S3_BUCKET`, `S3_REGION`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY` \| the bundled MinIO, bucket `g1t-packages` \| Where packages' files are kept: any S3-compatible store. Change the two keys before first start; MinIO is made with them. \|
	160	+	\| `S3_ENDPOINT`, `S3_BUCKET`, `S3_REGION`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY` \| the bundled RustFS, bucket `g1t-packages` \| Where packages' files are kept: any S3-compatible store. Change the two keys before first start; RustFS is made with them. \|
159	161		\| `S3_PUBLIC_ENDPOINT` \| (none) \| The store's address as clients reach it. When set, large layers are downloaded from it directly with a signed URL. \|
160		−	\| `PACK_S3_BUCKET` \| `g1t-git-packs` \| The bucket on the same store that packs for fresh clones are kept in, so the next clone of the same commit is not built again. The bundled MinIO deletes packs after 7 days; on another store, give the bucket a rule that expires objects under `packs/` and unfinished multipart uploads. \|
161		−	\| `MINIO_IMAGE` \| `pgsty/minio:latest` \| The MinIO server image the bundled store runs. MinIO no longer publishes its own images; this is a community build of the same server. \|
	162	+	\| `PACK_S3_BUCKET` \| `g1t-git-packs` \| The bucket on the same store that packs for fresh clones are kept in, so the next clone of the same commit is not built again. The bundled store deletes packs after 7 days, and uploads left unfinished after a day; on another store, give the bucket a lifecycle rule that does the same. \|
	163	+	\| `RUSTFS_IMAGE` \| `rustfs/rustfs:1.0.1` \| The image the bundled object store runs: [RustFS](https://rustfs.com), an S3-compatible server. \|
	164	+	\| `AWS_CLI_IMAGE` \| `amazon/aws-cli:2.37.10` \| The image `storage-setup` makes the buckets and the packs' lifecycle rule with. \|
162	165		\| `BACKUP_S3_BUCKET` \| `g1t-backups` \| The bucket on the same store that nightly repository backups (a `git bundle` of each repository whose branches or tags changed) are kept in. The bundles are cut by g1t's runner, which this installation does not run yet, so the bucket stays empty for now: copy the volumes, as below. \|
163	166		\| `STATUS_PORT` \| `8788` \| The port the status page is published on \|
164	167		\| `STATUS_PROBE_REPO` \| (none) \| A public repository, `workspace/repo`, whose branches the status page lists every minute as a clone would. Empty: git is not checked. \|
⋯
238	241		\| --- \| --- \|
239	242		\| `g1t_g1t-data` \| Accounts, workspaces, issues and every other record, as SQLite files; the keys that seal stored secrets (`keys.env`) \|
240	243		\| `g1t_g1t-git` \| Your repositories, one bare git repository each \|
241		−	\| `g1t_g1t-packages` \| Container images' layers and other package files, the `g1t-backups` bucket and the clone packs in `g1t-git-packs` (MinIO) \|
	244	+	\| `g1t_g1t-objects` \| The bundled object store (RustFS): container images' layers and other package files in `g1t-packages`, the `g1t-backups` bucket and the clone packs in `g1t-git-packs` \|
242	245		\| `g1t_g1t-secrets` \| The key the site and the git store share \|
243	246
244	247		To back up, stop g1t and copy the volumes:
⋯
263	266		docker compose -f deploy/self-host/docker-compose.yml up --build -d
264	267		```
265	268
	269	+	### Installations started before 7 October 2026
	270	+
	271	+	These kept packages' files and backups in MinIO, in the `g1t_g1t-packages`
	272	+	volume. The bundled store is now RustFS, in `g1t_g1t-objects`, and starts
	273	+	empty. After the upgrade above, copy the old objects across (use your own
	274	+	`S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY` if you changed them):
	275	+
	276	+	```sh
	277	+	docker run -d --name g1t-old-store --network g1t_default \
	278	+	-v g1t_g1t-packages:/data -e MINIO_ROOT_USER=g1t \
	279	+	-e MINIO_ROOT_PASSWORD=g1t-packages-secret pgsty/minio server /data
	280	+	docker run --rm --network g1t_default -e AWS_ACCESS_KEY_ID=g1t \
	281	+	-e AWS_SECRET_ACCESS_KEY=g1t-packages-secret -e AWS_DEFAULT_REGION=us-east-1 \
	282	+	--entrypoint sh amazon/aws-cli:2.37.10 -c '
	283	+	for b in g1t-packages g1t-backups; do
	284	+	aws --endpoint-url http://g1t-old-store:9000 s3 sync "s3://$b" "/tmp/$b" &&
	285	+	aws --endpoint-url http://rustfs:9000 s3 sync "/tmp/$b" "s3://$b"
	286	+	done'
	287	+	docker rm -f g1t-old-store
	288	+	```
	289	+
	290	+	The clone packs are not copied: they are a cache, and are made again on
	291	+	the next clone. Once your images and packages pull, remove the old volume
	292	+	with `docker volume rm g1t_g1t-packages`.
	293	+
266	294		## Stop and remove
267	295
268	296		```sh

+1−1View BlameViewed

3	3		version = "0.1.0"
4	4		edition.workspace = true
5	5		license.workspace = true
6		−	description = "Object storage behind one port: R2 on Cloudflare, any S3-compatible store (MinIO) when self-hosted."
	6	+	description = "Object storage behind one port: R2 on Cloudflare, any S3-compatible store (RustFS) when self-hosted."
7	7
8	8		[dependencies]
9	9		g1t-contracts.workspace = true

+1−1View BlameViewed

1	1		//! Object storage behind one port, `BlobStore`: R2 on Cloudflare, and any
2		−	//! S3-compatible store (MinIO in the self-host compose file) elsewhere.
	2	+	//! S3-compatible store (RustFS in the self-host compose file) elsewhere.
3	3		//!
4	4		//! Every service that keeps objects names its own [`Config`]: the variable
5	5		//! that chooses the store (`r2`, the default, or `s3`), the R2 bucket

+3−3View BlameViewed

1	1		//! The S3 adapter, for self-hosted installations: any S3-compatible store
2		−	//! (MinIO, Ceph, Garage, AWS) over fetch, signed with SigV4, path-style.
	2	+	//! (RustFS, Ceph, Garage, AWS) over fetch, signed with SigV4, path-style.
3	3		//! S3_ENDPOINT, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY and S3_REGION say
4	4		//! where, and the service's own variable (`Config::s3_bucket`) which
5	5		//! bucket. Its public endpoint variable, when it names one and that is
⋯
13	13		use crate::{BlobStore, Config, Got, Part, Wanted, var};
14	14
15	15		pub struct S3Store {
16		−	/// `http://minio:9000`, without a trailing slash.
	16	+	/// `http://rustfs:9000`, without a trailing slash.
17	17		endpoint: String,
18	18		/// Where clients reach the same store, for signed URLs.
19	19		public_endpoint: Option<String>,
⋯
249	249		let xml = "<InitiateMultipartUploadResult><Bucket>b</Bucket><UploadId>abc-123</UploadId></InitiateMultipartUploadResult>";
250	250		assert_eq!(xml_value(xml, "UploadId"), Some("abc-123"));
251	251		assert_eq!(xml_value(xml, "Key"), None);
252		−	assert_eq!(host_of("http://minio:9000"), "minio:9000");
	252	+	assert_eq!(host_of("http://rustfs:9000"), "rustfs:9000");
253	253		assert_eq!(host_of("https://s3.example.com/base"), "s3.example.com");
254	254		}
255	255		}

+4−4View BlameViewed

9	9		// - ARTIFACTS (git storage) becomes a service binding to workers/artifacts,
10	10		// which keeps repositories in the git store (gitstore/server.mjs);
11	11		// - EMAIL (Email Sending) becomes a service binding to workers/mail;
12		−	// - the packages service keeps files in S3-compatible storage (MinIO)
	12	+	// - the packages service keeps files in S3-compatible storage (RustFS)
13	13		// instead of R2, and the repos service its nightly backups (a bucket of
14	14		// their own, BACKUP_S3_BUCKET);
15	15		// - services that are off in this phase (agents, the context hub, the
⋯
216	216		// Access requests are summarised to your own address, not g1t.sh's.
217	217		config.vars.WAITLIST_NOTIFY_EMAIL = process.env.WAITLIST_NOTIFY_EMAIL ?? "";
218	218		}
219		−	// Packages' files go to the compose file's MinIO (or any S3-compatible
	219	+	// Packages' files go to the compose file's RustFS (or any S3-compatible
220	220		// store) instead of R2, with no request size limit, and package
221	221		// addresses start with this installation's host.
222	222		if (hosted.name === "g1t-packages") {
223	223		Object.assign(config.vars, {
224	224		BLOB_STORE: "s3",
225		−	S3_ENDPOINT: process.env.S3_ENDPOINT ?? "http://minio:9000",
	225	+	S3_ENDPOINT: process.env.S3_ENDPOINT ?? "http://rustfs:9000",
226	226		S3_BUCKET: process.env.S3_BUCKET ?? "g1t-packages",
227	227		S3_REGION: process.env.S3_REGION ?? "us-east-1",
228	228		S3_ACCESS_KEY_ID: process.env.S3_ACCESS_KEY_ID ?? "",
⋯
252	252		BACKUP_S3_BUCKET: process.env.BACKUP_S3_BUCKET ?? "g1t-backups",
253	253		PACK_STORE: "s3",
254	254		PACK_S3_BUCKET: process.env.PACK_S3_BUCKET ?? "g1t-git-packs",
255		−	S3_ENDPOINT: process.env.S3_ENDPOINT ?? "http://minio:9000",
	255	+	S3_ENDPOINT: process.env.S3_ENDPOINT ?? "http://rustfs:9000",
256	256		S3_REGION: process.env.S3_REGION ?? "us-east-1",
257	257		S3_ACCESS_KEY_ID: process.env.S3_ACCESS_KEY_ID ?? "",
258	258		S3_SECRET_ACCESS_KEY: process.env.S3_SECRET_ACCESS_KEY ?? "",

+32−34View BlameViewed

9	9		# What runs: the site and every core service in one workerd (g1t), git
10	10		# repositories as bare repos on a volume (gitstore), the API in a second
11	11		# workerd beside it, packages' files, backups and the clone pack cache in
12		−	# MinIO, and Mailpit for mail.
	12	+	# RustFS (S3-compatible storage), and Mailpit for mail.
13	13		# Agents, deployments, context search and billing are off. See
14	14		# docs/SELF_HOSTING.md.
15	15		name: g1t
⋯
51	51		INVITES_PER_USER: ${INVITES_PER_USER:-}
52	52		# Where summaries of new access requests go; empty sends none.
53	53		WAITLIST_NOTIFY_EMAIL: ${WAITLIST_NOTIFY_EMAIL:-}
54		−	# Packages' files (container images and the rest), in MinIO below or
	54	+	# Packages' files (container images and the rest), in RustFS below or
55	55		# any S3-compatible store. S3_PUBLIC_ENDPOINT, when clients can reach
56	56		# the store, sends large downloads there directly.
57		−	S3_ENDPOINT: ${S3_ENDPOINT:-http://minio:9000}
	57	+	S3_ENDPOINT: ${S3_ENDPOINT:-http://rustfs:9000}
58	58		S3_BUCKET: ${S3_BUCKET:-g1t-packages}
59	59		S3_REGION: ${S3_REGION:-us-east-1}
60	60		S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:-g1t}
⋯
64	64		# the same store (docs/SELF_HOSTING.md, "Backups").
65	65		BACKUP_S3_BUCKET: ${BACKUP_S3_BUCKET:-g1t-backups}
66	66		# Packs kept for fresh clones, so the next clone of the same commit
67		−	# does not rebuild one; minio-setup expires them after 7 days.
	67	+	# does not rebuild one; storage-setup expires them after 7 days.
68	68		PACK_S3_BUCKET: ${PACK_S3_BUCKET:-g1t-git-packs}
69	69		volumes:
70	70		- g1t-data:/data
⋯
74	74		condition: service_healthy
75	75		mailpit:
76	76		condition: service_started
77		−	minio-setup:
	77	+	storage-setup:
78	78		condition: service_completed_successfully
79	79		restart: unless-stopped
80	80
⋯
110	110		# Not published: only the g1t container reaches it.
111	111		restart: unless-stopped
112	112
113		−	# Packages' files, backups and clone packs. Not published: only the g1t
114		−	# container reaches it, unless you publish 9000 and set
115		−	# S3_PUBLIC_ENDPOINT. MinIO no longer publishes images of its own;
116		−	# MINIO_IMAGE is a community build of the same server, with `mc` in it.
117		−	# Any S3-compatible store works in its place (S3_ENDPOINT).
118		−	minio:
119		−	image: ${MINIO_IMAGE:-pgsty/minio:latest}
120		−	command: ["server", "/data"]
	113	+	# Packages' files, backups and clone packs, in RustFS (S3-compatible).
	114	+	# Not published: only the g1t container reaches it, unless you publish
	115	+	# 9000 and set S3_PUBLIC_ENDPOINT. Any S3-compatible store works in its
	116	+	# place (S3_ENDPOINT).
	117	+	rustfs:
	118	+	image: ${RUSTFS_IMAGE:-rustfs/rustfs:1.0.1}
121	119		environment:
122		−	MINIO_ROOT_USER: ${S3_ACCESS_KEY_ID:-g1t}
123		−	MINIO_ROOT_PASSWORD: ${S3_SECRET_ACCESS_KEY:-g1t-packages-secret}
124		−	# What the health check's `mc ready local` asks.
125		−	MC_HOST_local: http://localhost:9000
	120	+	RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY_ID:-g1t}
	121	+	RUSTFS_SECRET_KEY: ${S3_SECRET_ACCESS_KEY:-g1t-packages-secret}
	122	+	RUSTFS_CONSOLE_ENABLE: "false"
126	123		volumes:
127		−	- g1t-packages:/data
	124	+	- g1t-objects:/data
128	125		healthcheck:
129		−	test: ["CMD", "mc", "ready", "local"]
	126	+	test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://localhost:9000/health/ready"]
130	127		interval: 5s
131	128		retries: 20
132	129		restart: unless-stopped
133	130
134	131		# Makes the buckets once, then exits: packages' files, backups, and
135		−	# clone packs with a rule that deletes packs 7 days old. (MinIO itself
136		−	# removes uploads left unfinished after 24 hours.)
137		−	minio-setup:
138		−	image: ${MINIO_IMAGE:-pgsty/minio:latest}
	132	+	# clone packs, with a lifecycle rule on the packs' bucket that deletes
	133	+	# packs 7 days old and uploads left unfinished after a day.
	134	+	storage-setup:
	135	+	image: ${AWS_CLI_IMAGE:-amazon/aws-cli:2.37.10}
139	136		depends_on:
140		−	minio:
	137	+	rustfs:
141	138		condition: service_healthy
142	139		entrypoint:
143	140		- sh
144	141		- -c
145	142		- >-
146	143		set -e;
147		−	mc alias set local http://minio:9000 "$$MINIO_ROOT_USER" "$$MINIO_ROOT_PASSWORD";
148		−	mc mb --ignore-existing "local/$$S3_BUCKET";
149		−	mc mb --ignore-existing "local/$$BACKUP_S3_BUCKET";
150		−	mc mb --ignore-existing "local/$$PACK_S3_BUCKET";
151		−	if ! mc ilm rule ls "local/$$PACK_S3_BUCKET" >/dev/null 2>&1; then
152		−	mc ilm rule add --prefix packs/ --expire-days 7 "local/$$PACK_S3_BUCKET";
153		−	fi
	144	+	for bucket in "$$S3_BUCKET" "$$BACKUP_S3_BUCKET" "$$PACK_S3_BUCKET"; do
	145	+	aws s3api head-bucket --bucket "$$bucket" >/dev/null 2>&1 \|\| aws s3api create-bucket --bucket "$$bucket" >/dev/null;
	146	+	done;
	147	+	aws s3api put-bucket-lifecycle-configuration --bucket "$$PACK_S3_BUCKET" --lifecycle-configuration
	148	+	'{"Rules":[{"ID":"expire-packs","Status":"Enabled","Filter":{"Prefix":"packs/"},"Expiration":{"Days":7}},{"ID":"abort-unfinished-uploads","Status":"Enabled","Filter":{"Prefix":""},"AbortIncompleteMultipartUpload":{"DaysAfterInitiation":1}}]}';
	149	+	echo "buckets ready"
154	150		environment:
155		−	MINIO_ROOT_USER: ${S3_ACCESS_KEY_ID:-g1t}
156		−	MINIO_ROOT_PASSWORD: ${S3_SECRET_ACCESS_KEY:-g1t-packages-secret}
	151	+	AWS_ENDPOINT_URL: http://rustfs:9000
	152	+	AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:-g1t}
	153	+	AWS_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:-g1t-packages-secret}
	154	+	AWS_DEFAULT_REGION: ${S3_REGION:-us-east-1}
157	155		S3_BUCKET: ${S3_BUCKET:-g1t-packages}
158	156		BACKUP_S3_BUCKET: ${BACKUP_S3_BUCKET:-g1t-backups}
159	157		PACK_S3_BUCKET: ${PACK_S3_BUCKET:-g1t-git-packs}
⋯
173	171
174	172		volumes:
175	173		g1t-data:
176		−	g1t-packages:
	174	+	g1t-objects:
177	175		g1t-status:
178	176		g1t-git:
179	177		g1t-secrets:

+25−3View BlameViewed

2	2		# End-to-end check of a self-hosted g1t: sign up, confirm the email, make a
3	3		# workspace and a repository, push and clone over HTTP, open an issue, and
4	4		# read the code back through the site. Then the API on its own port (REST,
5		−	# OAuth metadata and MCP, with an access token), pull requests from a branch
	5	+	# OAuth metadata and MCP, with an access token), an npm package published to
	6	+	# the installation's registry and installed back, pull requests from a branch
6	7		# and from a fork merged onto main, the merge queue taking a pull request
7	8		# and giving it back, and every cron handler the scheduler runs.
8	9		#
⋯
21	22		# (In Git Bash on Windows, start the command with `env MSYS_NO_PATHCONV=1`
22	23		# so /data is not rewritten into a Windows path.)
23	24		# PACK_CACHE=off skips the check that a second clone is served from the
24		−	# clone pack cache.
	25	+	# clone pack cache. PACKAGES=off skips publishing an npm package to the
	26	+	# installation's registry and installing it back.
25	27		#
26		−	# Needs curl, git and node (to read JSON).
	28	+	# Needs curl, git and node (to read JSON), and npm for the package.
27	29		set -euo pipefail
28	30
29	31		G1T_URL="${G1T_URL:-http://localhost:8787}"
⋯
35	37		# off: this installation keeps no clone packs (no PACK_STORE), so a second
36	38		# clone is not checked for a kept one.
37	39		PACK_CACHE="${PACK_CACHE:-on}"
	40	+	# off: skip publishing and installing an npm package (which needs npm).
	41	+	PACKAGES="${PACKAGES:-on}"
38	42		RUN="$(date +%s)"
39	43		USER_NAME="smoke${RUN}"
40	44		EMAIL="${USER_NAME}@example.com"
⋯
209	213		[ "$code" = 200 ] && grep -q '"tools"' "$WORK/api" \|\| fail "MCP tools/list: $code $(head -c 300 "$WORK/api")"
210	214		echo "MCP lists its tools at $MCP_URL"
211	215
	216	+	if [ "$PACKAGES" != off ]; then
	217	+	step "publish an npm package and install it"
	218	+	# The tarball is kept in the packages store (S3), and read back from it.
	219	+	npmrc="@$WORKSPACE:registry=$G1T_URL/-/npm/
	220	+	//${G1T_URL#*://}/-/npm/:_authToken=$TOKEN"
	221	+	mkdir -p "$WORK/pkg" "$WORK/app"
	222	+	printf '%s\n' "$npmrc" > "$WORK/pkg/.npmrc"
	223	+	printf '%s\n' "$npmrc" > "$WORK/app/.npmrc"
	224	+	printf '{"name":"@%s/%s","version":"1.0.0","repository":"%s/%s/%s","main":"index.js"}\n' \
	225	+	"$WORKSPACE" "$REPO" "$G1T_URL" "$WORKSPACE" "$REPO" > "$WORK/pkg/package.json"
	226	+	printf 'module.exports = "published by smoke.sh %s";\n' "$RUN" > "$WORK/pkg/index.js"
	227	+	(cd "$WORK/pkg" && npm publish --silent) \|\| fail "npm publish"
	228	+	(cd "$WORK/app" && npm init -y >/dev/null && npm install --silent --no-audit --no-fund "@$WORKSPACE/$REPO@1.0.0") \|\| fail "npm install"
	229	+	got="$(cd "$WORK/app" && node -p "require('@$WORKSPACE/$REPO')")"
	230	+	[ "$got" = "published by smoke.sh $RUN" ] \|\| fail "the installed package says: $got"
	231	+	echo "installed @$WORKSPACE/$REPO@1.0.0"
	232	+	fi
	233	+
212	234		step "a pull request from a branch, merged"
213	235		git -C "$WORK/clone" config credential.helper ""
214	236		commit_file "$WORK/clone" "$remote" from-branch docs/branch.md

+3−3View BlameViewed

373	373		\| R10 \| Built in repos; work unchanged \| `divergence` works out the target's side once per target head per isolate (`coalesce.rs`: the head under the refs version, then the history by hash, kept 60 s), and what the target changed between two trees once per pair (10 minutes). `readCommit` and logs by hash come from the cache. Work's fan-out (`after_push`, up to 100 pull requests) is unchanged: its 100 `divergence` calls now cost one walk of the target instead of 100. \|
374	374		\| R7 \| Built; the namespaces are yours to make \| `shards.rs`: bindings named in `ARTIFACTS_NAMESPACES` (JSON, binding → namespace; `ARTIFACTS` → `g1t` always there), a repository's namespace kept in its `store` column as `<namespace>/<key>` (no prefix means the `ARTIFACTS` namespace, so every existing key reads the same), forks always in their repository's namespace. Placing (`Placement::choose`, loads from `namespaces.rs`): among `ARTIFACTS_NEW_REPOS`, the healthy namespaces (bound, taking writes, not failing, under `ARTIFACTS_NAMESPACE_LIMITS`' `max_repos`, busiest minute under 70% of the 12,000-a-minute limit) within 100 repositories or 5% of the emptiest, spread by an FNV hash of the id; loads are read (one D1 query each for the registry and `store_health`, kept a minute) only when there is more than one to choose from. Moving (`moves.rs`): `move_repository` or `scripts/ops/artifacts-namespaces.mjs move` queues one; the hourly sweep pauses writes, copies every ref of the repository and its working copies over git (one streamed upload-pack into one receive-pack each), switches every `store` key in one batch, and deletes the old copies after 7 days. EU residency: identity's `workspaces.data_residency`, set by an owner in the workspace's settings (shown only once `storage_options` says an EU namespace takes repositories), read by the repos service at creation only while `ARTIFACTS_EU_NAMESPACE` is set; an EU workspace's repository goes to that namespace or is not made. Health and limits: `namespaces` RPC and `scripts/ops/artifacts-namespaces.mjs`. Nothing changes until bindings and variables name new namespaces. See "R7: sharding, moves and EU residency" below. \|
375	375		\| R8 \| Built \| `crates/runner/src/clone.rs`: every sandbox clones at `--depth=1` (a full g1t clone took 5.4 s, depth 1 took 3.8 s). Work that merges (catch-up, the merge queue, merge checks, a review's diff) deepens 50, 500, then 5000 commits until the two sides share one, and fetches everything only as the last resort (`share_history`). `G1T_CLONE_DEPTH` (0 or `full` for everything) and `G1T_CLONE_FILTER=blob:none` change it per runner. \|
376		−	\| R6 \| Built; the bucket must exist before it deploys \| `pack_cache.rs`: an upload-pack POST with wants and no `have` or `shallow` lines (a fresh clone, the sandboxes' `deepen 1` ones included), uncompressed and at most 1 MiB, is keyed `packs/<repo id>/<refs_version>/<sha256>` over the request normalized: protocol v2 capabilities without `agent=`/`session-id=` and its arguments, each sorted and deduplicated; v0/v1 wants sorted, the first want's capabilities split off, sorted and without `agent=`, then `deepen`/`filter` lines, a flush and `done`. Only while `refs_cache::usable` (the version known, and no push credential out of g1t's hands), so never across a refs change. Looked up after authorization, alongside the free-workspace limits and the kept refs answer; a hit streams from the bucket (`Server-Timing` `pack;desc=hit`). A miss streams the store's 200 to git through a tee that copies it to a fill in `ctx.wait_until` (at most 5 MiB queued between them, 2 fills per isolate, one per key): under 5 MiB it is one `put` once it all arrived; larger, 5 MiB multipart parts completed only after the last part and a check that it is one whole side-band pack (well-formed pkt-lines, `PACK` on channel 1, no `ERR` or channel 3, a closing flush). Over 200 MB, a queue that falls behind, git going away or the store's stream failing lets the fill go and aborts the upload; nothing partial can be read. Meters `pack_cache.hit` (with the bytes served) and `pack_cache.miss` (counted with `record`, bytes added at the end), neither an operation by default; a hit records no `git.fetch`. Storage is behind the `PackStore` port, whose adapter puts the shared `BlobStore` (`crates/blobstore`) behind it: R2 hosted (`GIT_PACKS`, bucket `g1t-git-packs`, lifecycle: packs deleted after 7 days, unfinished uploads after 1), or with `PACK_STORE=s3` the S3 bucket `PACK_S3_BUCKET` names (self-hosted: MinIO's `g1t-git-packs`, packs deleted after 7 days, unfinished uploads by MinIO after 24 hours; `node services/repos/dev/clone-check.mjs --s3` checks it). With neither, nothing is kept. \|
	376	+	\| R6 \| Built; the bucket must exist before it deploys \| `pack_cache.rs`: an upload-pack POST with wants and no `have` or `shallow` lines (a fresh clone, the sandboxes' `deepen 1` ones included), uncompressed and at most 1 MiB, is keyed `packs/<repo id>/<refs_version>/<sha256>` over the request normalized: protocol v2 capabilities without `agent=`/`session-id=` and its arguments, each sorted and deduplicated; v0/v1 wants sorted, the first want's capabilities split off, sorted and without `agent=`, then `deepen`/`filter` lines, a flush and `done`. Only while `refs_cache::usable` (the version known, and no push credential out of g1t's hands), so never across a refs change. Looked up after authorization, alongside the free-workspace limits and the kept refs answer; a hit streams from the bucket (`Server-Timing` `pack;desc=hit`). A miss streams the store's 200 to git through a tee that copies it to a fill in `ctx.wait_until` (at most 5 MiB queued between them, 2 fills per isolate, one per key): under 5 MiB it is one `put` once it all arrived; larger, 5 MiB multipart parts completed only after the last part and a check that it is one whole side-band pack (well-formed pkt-lines, `PACK` on channel 1, no `ERR` or channel 3, a closing flush). Over 200 MB, a queue that falls behind, git going away or the store's stream failing lets the fill go and aborts the upload; nothing partial can be read. Meters `pack_cache.hit` (with the bytes served) and `pack_cache.miss` (counted with `record`, bytes added at the end), neither an operation by default; a hit records no `git.fetch`. Storage is behind the `PackStore` port, whose adapter puts the shared `BlobStore` (`crates/blobstore`) behind it: R2 hosted (`GIT_PACKS`, bucket `g1t-git-packs`, lifecycle: packs deleted after 7 days, unfinished uploads after 1), or with `PACK_STORE=s3` the S3 bucket `PACK_S3_BUCKET` names (self-hosted: RustFS's `g1t-git-packs`, with the same lifecycle rule, put by the compose file's `storage-setup`; `node services/repos/dev/clone-check.mjs --s3` checks it). With neither, nothing is kept. \|
377	377		\| R11 \| Built; not yet deployed \| Nightly `git bundle` backups to the `g1t-backups` R2 bucket, and a restore drill. Migration `0013_backups.sql` (`repo_backups`, and an `operation_mapping` row). See "R11: backups and the restore drill" below. \|
378	378		\| R12 \| Built; the host is yours to make \| `scripts/ops/restore-to-gitstore.mjs` rebuilds every repository from its bundle chain into the git store (`deploy/self-host/gitstore`, now with namespaced keys, `<root>/<namespace>/<name>.git`, and `GITSTORE_READ_ONLY=1`), on the host or over its API, and later lists and reconciles what the fallback took. `fallback.rs`: with `GIT_FALLBACK_URL`, `GIT_FALLBACK_SECRET` and `GIT_FALLBACK_NAMESPACES` set, a namespace's `GitStore` calls go to the git store's API instead of the Artifacts binding (same metering, retries and breaker, its own health as `<namespace>@fallback`), read-only unless `GIT_FALLBACK_WRITES=allow`: writes are refused before they are asked, and say so in words; kept ref listings and packs are not used, nor backups cut. status.g1t.sh shows Git storage degraded meanwhile. See "R12: the fallback store and the outage runbook" below. \|
379	379
⋯
654	654
655	655		Storage, through the `BlobStore` port in `crates/blobstore` (the adapters packages already used):
656	656		the `BACKUPS` binding (bucket `g1t-backups`) with `BACKUP_STORE=r2`; any S3-compatible store with
657		−	`BACKUP_STORE=s3` and `BACKUP_S3_BUCKET` (self-hosted: MinIO). Without either, backups are off and
	657	+	`BACKUP_STORE=s3` and `BACKUP_S3_BUCKET` (self-hosted: RustFS). Without either, backups are off and
658	658		the nightly cron does nothing.
659	659
660	660		```text
⋯
696	696		at random, the repository is one whose refs have not moved since its last backup, so any
697	697		difference is the backup's. Run it after the first night, then monthly, and after any change to
698	698		`backups.rs` or `backup.rs`. `--bundles <dir>` reads a local copy of the bucket instead
699		−	(self-hosted: `mc mirror local/g1t-backups <dir>`), with `--repo-id` and `--live <url or path>`.
	699	+	(self-hosted: `aws --endpoint-url <S3_ENDPOINT> s3 sync s3://g1t-backups <dir>`), with `--repo-id` and `--live <url or path>`.
700	700		`npm run test:ops` runs it against bundles cut with git.
701	701
702	702		A real restore into the store, as it can be done today:

+1−1View BlameViewed

36	36		`packages/contracts`. Other services talk to it over RPC and hear from it through events.
37	37		- Cloudflare in production, anything in a self-hosted install. Files go through a
38	38		`BlobStore` port. In production its adapter is R2; self-hosted it is S3-compatible storage
39		−	(MinIO in the compose file) or a local directory. Upload URLs (`presign`) are part of the port:
	39	+	(RustFS in the compose file) or a local directory. Upload URLs (`presign`) are part of the port:
40	40		R2 and S3 sign them, the disk adapter answers with a path back through the service.
41	41		Metadata is D1, which self-hosting already runs (workerd's D1 over SQLite).
42	42		- Content-addressed. Every file is stored once by its SHA-256 (`blobs/sha256/<hex>`). A

+40−25View BlameViewed

40	40		answers "agents are off" instead of failing.
41	41		- Proven on this machine with Docker: sign up, confirm the email
42	42		through Mailpit, create a workspace and a repository, push and clone over
43		−	HTTP (the second clone from the clone pack cache in MinIO), open an
	43	+	HTTP (the second clone from the clone pack cache in RustFS), open an
44	44		issue, and browse code, commits and files in the site; then the REST API,
45		−	OAuth metadata and MCP on their own port, pull requests from a branch and
46		−	from a fork merged onto `main`, the merge queue taking a pull request and
	45	+	OAuth metadata and MCP on their own port, an npm package published and
	46	+	installed and a container image pushed and pulled through RustFS, pull
	47	+	requests from a branch and from a fork merged onto `main`, the merge queue taking a pull request and
47	48		giving it back, and every cron handler the scheduler runs. All of it runs
48	49		against local storage. See [Phase 1: what works today](#3-phase-1-what-works-today).
49	50		- Long term: keep workerd as the runtime, because it is what hosted
⋯
68	69		\| Workers runtime, service bindings \| Every service. Rust through `worker` 0.8 (`#[event(fetch\\|queue\\|scheduled)]`, `Env`, `Fetcher`); TS as `export default { fetch, queue, scheduled }` \| woven (as a runtime), thin (as an API) \| workerd: the same runtime, open source. Service bindings work as they do hosted. Calls are HTTP (`POST /rpc/<method>`), so a native port could use plain HTTP clients. \|
69	70		\| Workers RPC (JS methods across a binding) \| Only `RUNNER`: `RunnerService extends WorkerEntrypoint` (`services/runner/src/index.ts:626`). `apps/web` calls `env.RUNNER.enabled/run/plan/...` directly in 11 routes. \| thin \| workerd supports it. A native port needs these on `/rpc/*` as well; the runner already has a `fetch` shim for Rust callers. \|
70	71		\| D1 \| System of record for 13 services. Rust: `env.d1("DB")`; TS: `D1Database`; `db.batch()` in `crates/kit` `rename` \| woven (SQL), thin (API) \| SQLite files. workerd/Miniflare implements D1 on SQLite, and the same `migrations/` apply with `wrangler d1 migrations apply --local`. A native port would need a `Database` port over `rusqlite`/`better-sqlite3`; the SQL is already SQLite, including FTS5. \|
71		−	\| KV \| `BLOBS`: Actions artifacts and cache (`apps/api/src/blobs.rs`, `apps/web/app/lib/artifacts.server.ts`). `AVATARS`: `services/identity/src/avatars.rs`, `apps/web/workers/app.ts`, `services/og`. `DOMAINS`: `services/deployments/src/domains.ts`, `services/pages` \| thin \| Miniflare KV on disk (SQLite plus blob files). Natively: a `BlobStore` port on the filesystem or S3/MinIO. \|
	72	+	\| KV \| `BLOBS`: Actions artifacts and cache (`apps/api/src/blobs.rs`, `apps/web/app/lib/artifacts.server.ts`). `AVATARS`: `services/identity/src/avatars.rs`, `apps/web/workers/app.ts`, `services/og`. `DOMAINS`: `services/deployments/src/domains.ts`, `services/pages` \| thin \| Miniflare KV on disk (SQLite plus blob files). Natively: a `BlobStore` port on the filesystem or S3. \|
72	73		\| Queues: the event bus \| Producer: `services/events` `BUS.sendBatch` (`lib.rs:67`). The consumer writes the log, then fans out to every binding named `SUBSCRIBER_*` (`lib.rs:161`). Twelve consumers, one queue each. Private job queues in search (`g1t-search-jobs`) and context (`g1t-context-jobs`); consumers branch on the queue name. \| woven \| Miniflare Queues: in-process and persisted, which works today. Natively: a `Bus` port with a SQLite outbox and a poller per subscriber, or NATS/Redis Streams. At-least-once delivery and idempotent consumers are already the contract. \|
73	74		\| Durable Objects \| Only `AttemptSandbox` (runner), as the containers library's base class. Uses `ctx.storage.get/put/delete`, `schedule()` (alarm), `idFromName`/`idFromString`, DO RPC (`run`, `destroy`, `noteBlocked`). Not used: WebSocket hibernation, raw `alarm()`, `ctx.storage.sql`, `ctx.exports`. \| woven, in the runner only \| workerd supports Durable Objects (on-disk SQLite). Runner state can move to the sandbox supervisor (phase 2). \|
74	75		\| Containers (`@cloudflare/containers`) \| `services/runner`: one sandbox per agent run, Actions job and deploy build. `sleepAfter`, `start({ envVars, enableInternet })`, `onStop`. Image: `services/runner/Dockerfile` (node 24, git, toolchains, Claude Code, `g1t-runner`). \| woven \| Docker or Podman through the socket, with the same image. Wrangler can already run Containers locally through Docker; whether that covers outbound interception has to be tested. \|
⋯
88	89		\| Static Assets \| `apps/web` (Vite plugin build), `apps/docs`, `apps/sudo` (`run_worker_first`) \| thin \| workerd serves them. \|
89	90		\| `placement`, `observability`, routes, custom domains \| every `wrangler.jsonc` \| config only \| Dropped by `deploy/self-host/configs.mjs`. \|
90	91		\| `cf-ray` \| Used as an audit request id, with a fallback: `services/repos/src/run_access.rs:131`, `apps/api/src/audit.rs:37` \| thin \| Falls back already. \|
91		−	\| R2 \| `services/packages` (`BLOBS`: container layers and other package files), `services/repos` (`BACKUPS`: nightly backup bundles; `GIT_PACKS`: the clone pack cache), the API's Actions cache (`ACTIONS_CACHE`), the runner's downloads \| thin \| S3-compatible storage: the `BlobStore` port in `crates/blobstore` has an R2 adapter and an S3 one (`s3.rs`, SigV4 over fetch); each service names its own bucket (`BLOB_STORE`/`S3_BUCKET` for packages, `BACKUP_STORE`/`BACKUP_S3_BUCKET` for backups, `PACK_STORE`/`PACK_S3_BUCKET` for clone packs), run against MinIO in the compose file. \|
	92	+	\| R2 \| `services/packages` (`BLOBS`: container layers and other package files), `services/repos` (`BACKUPS`: nightly backup bundles; `GIT_PACKS`: the clone pack cache), the API's Actions cache (`ACTIONS_CACHE`), the runner's downloads \| thin \| S3-compatible storage: the `BlobStore` port in `crates/blobstore` has an R2 adapter and an S3 one (`s3.rs`, SigV4 over fetch); each service names its own bucket (`BLOB_STORE`/`S3_BUCKET` for packages, `BACKUP_STORE`/`BACKUP_S3_BUCKET` for backups, `PACK_STORE`/`PACK_S3_BUCKET` for clone packs), run against RustFS in the compose file. \|
92	93		\| Not used \| Hyperdrive, Workflows, Analytics Engine, Browser Rendering, Images, Turnstile, Secrets Store, `connect()`, HTMLRewriter, `request.cf` \| — \| — \|
93	94
94	95		### By service
⋯
106	107		\| `apps/docs` \| Static \| — \| Not run (docs.g1t.sh serves them) \|
107	108		\| `apps/status` \| TS Worker \| Email Sending, cron; bound only to billing \| Runs in a process of its own (`status.sh`), so it stays up when the site does not \|
108	109		\| `services/identity` \| Rust \| Email Sending, KV `AVATARS` \| Runs unchanged; `EMAIL` goes to the mail shim \|
109		−	\| `services/repos` \| Rust \| Artifacts, R2 (`BACKUPS`), Cache API, optional KV `GIT_CACHE` with `REPOS_KEY`, optional R2 `GIT_PACKS` \| Runs unchanged; `ARTIFACTS` goes to the git store, backups to MinIO's `g1t-backups` bucket (`BACKUP_STORE=s3`), and the clone pack cache to `g1t-git-packs` (`PACK_STORE=s3`: the `PackStore` port in `src/pack_cache.rs` over the shared `BlobStore`, multipart, an object only once whole; `minio-setup` gives the bucket a rule that deletes packs after 7 days, and MinIO removes unfinished uploads after 24 hours). Without `GIT_CACHE` and `REPOS_KEY`, credentials and ref listings are kept per isolate only. Its nightly cron queues backups, but bundles are cut by the runner, which is off in phase 1: none are made yet \|
	110	+	\| `services/repos` \| Rust \| Artifacts, R2 (`BACKUPS`), Cache API, optional KV `GIT_CACHE` with `REPOS_KEY`, optional R2 `GIT_PACKS` \| Runs unchanged; `ARTIFACTS` goes to the git store, backups to the `g1t-backups` bucket in RustFS (`BACKUP_STORE=s3`), and the clone pack cache to `g1t-git-packs` (`PACK_STORE=s3`: the `PackStore` port in `src/pack_cache.rs` over the shared `BlobStore`, multipart, an object only once whole; `storage-setup` gives the bucket a lifecycle rule that deletes packs after 7 days and aborts uploads unfinished after a day). Without `GIT_CACHE` and `REPOS_KEY`, credentials and ref listings are kept per isolate only. Its nightly cron queues backups, but bundles are cut by the runner, which is off in phase 1: none are made yet \|
110	111		\| `services/work` \| Rust \| Queue consumer \| Runs unchanged \|
111	112		\| `services/events` \| Rust \| Queues (producer and fan-out) \| Runs unchanged; the off services' queues are not produced to \|
112	113		\| `services/projects` \| TS \| Queue consumer \| Runs unchanged \|
⋯
116	117		\| `services/actions` \| Rust \| Queue, cron, `ACTIONS_KEY` \| Runs; jobs need the runner, which is off \|
117	118		\| `services/webhooks` \| Rust \| Queue, cron, `WEBHOOKS_KEY` \| Runs; retries through `scheduler.mjs` \|
118	119		\| `services/integrations` \| Rust \| Queue, `INTEGRATIONS_KEY` \| Runs unchanged \|
119		−	\| `services/packages` \| Rust \| R2 (`BLOBS`), cron, queue, `PACKAGES_TOKEN_SECRET` \| Runs with `BLOB_STORE=s3` against the compose file's MinIO (`deploy/self-host/configs.mjs`); no request size limit (`MAX_REQUEST_BYTES` 0 means none) \|
	120	+	\| `services/packages` \| Rust \| R2 (`BLOBS`), cron, queue, `PACKAGES_TOKEN_SECRET` \| Runs with `BLOB_STORE=s3` against the compose file's RustFS (`deploy/self-host/configs.mjs`); no request size limit (`MAX_REQUEST_BYTES` 0 means none) \|
120	121		\| `services/deployments` \| TS \| Workers for Platforms, REST API, KV `DOMAINS`, cron \| Runs with no API token: nothing deploys \|
121	122		\| `services/runner` \| TS \| Containers, Durable Objects, outbound interception, AI Gateway, cron \| Off: bound to the off Worker \|
122	123		\| `services/context` \| TS \| Vectorize, Workers AI, Queues \| Off: bound to the off Worker \|
⋯
345	346		\| Issues, pull requests, review \| On \| On \| — \|
346	347		\| Merge queue \| On \| Takes pull requests; testing and landing them needs sandboxes \| Phase 2 \|
347	348		\| Bringing a pull request up to date before it lands (catch-up) \| On \| Off: needs a sandbox \| Phase 2 \|
348		−	\| Clone pack cache \| R2 \| MinIO (`g1t-git-packs`) \| — \|
	349	+	\| Clone pack cache \| R2 \| RustFS (`g1t-git-packs`) \| — \|
349	350		\| Site search (FTS5) \| On \| On \| — \|
350	351		\| Email \| Email Sending \| Mailpit, logged \| SMTP relay \|
351	352		\| Webhooks, integrations \| On \| On (retries through `scheduler.mjs`) \| — \|
⋯
417	418		replication. The repositories get hosted g1t's nightly bundles
418	419		(docs/ARTIFACTS.md, R11) once the runner runs: the storage is already
419	420		configured (`BACKUP_STORE=s3`, the `g1t-backups` bucket that
420		−	`minio-setup` makes, `BACKUP_S3_BUCKET` to choose another), and the
	421	+	`storage-setup` makes, `BACKUP_S3_BUCKET` to choose another), and the
421	422		restore drill reads a copy of that bucket
422		−	(`mc mirror local/g1t-backups ./copy`, then
	423	+	(`aws --endpoint-url <S3_ENDPOINT> s3 sync s3://g1t-backups ./copy`, then
423	424		`node scripts/ops/backup-restore-drill.mjs --bundles ./copy --repo-id <id> --live <bare repository>`).
424	425
425	426		## 3. Phase 1: what works today
⋯
428	429
429	430		\| File \| What it is \|
430	431		\| --- \| --- \|
431		−	\| `docker-compose.yml` \| `g1t` (every core Worker in one workerd on 8787, and the API in a second on 8789), `status`, `gitstore` (bare repositories), `minio` and `minio-setup` (packages, backups and clone packs, with the packs' expiry rule), and `mailpit` (mail). Volumes: `g1t-data`, `g1t-git`, `g1t-packages`, `g1t-status`, `g1t-secrets`. MinIO no longer publishes `minio/minio` or `minio/mc` images; `MINIO_IMAGE` (default `pgsty/minio`, a community build with `mc` in it) is the server. \|
	432	+	\| `docker-compose.yml` \| `g1t` (every core Worker in one workerd on 8787, and the API in a second on 8789), `status`, `gitstore` (bare repositories), `rustfs` (S3-compatible storage for packages, backups and clone packs; `RUSTFS_IMAGE`, default `rustfs/rustfs:1.0.1`), `storage-setup` (the AWS CLI, `AWS_CLI_IMAGE`, default `amazon/aws-cli:2.37.10`: makes the three buckets and puts the packs' bucket's lifecycle rule, which expires `packs/` after 7 days and aborts multipart uploads unfinished after a day; RustFS's scanner applies it), and `mailpit` (mail). Volumes: `g1t-data`, `g1t-git`, `g1t-objects`, `g1t-status`, `g1t-secrets`. \|
432	433		\| `Dockerfile` \| Compiles the ten Rust services to WebAssembly with `worker-build`, as hosted does. Builds the site with React Router. The runtime image has Node, Wrangler, workerd and the built Workers. \|
433	434		\| `Dockerfile.dockerignore` \| Build-context rules for this image only (the root `.dockerignore` leaves out the site). \|
434	435		\| `start.sh` \| Makes the sealing keys once, writes the configs, applies migrations, runs `wrangler dev` with every config on `0.0.0.0:8787`, persisting to `/data/state`, and the API's `wrangler dev` on `0.0.0.0:8789` once the first answers. Starts `scheduler.mjs`. \|
⋯
438	439		\| `workers/artifacts/index.js` \| The `ARTIFACTS` binding, implemented against the git store. \|
439	440		\| `workers/mail/index.js` \| The `EMAIL` binding: logs, then sends to Mailpit. \|
440	441		\| `workers/off/index.js` \| The runner and the context hub when they are off. \|
441		−	\| `smoke.sh` \| The end-to-end check, including the API, pull requests, the merge queue and (with `SCHEDULER_ONCE`) every cron handler. \|
	442	+	\| `smoke.sh` \| The end-to-end check, including the clone pack cache, the API, an npm package published and installed, pull requests, the merge queue and (with `SCHEDULER_ONCE`) every cron handler. \|
442	443
443	444		Workers running: the site; identity, repos, work, events, projects, search,
444	445		billing, security, actions, webhooks, integrations, packages and
⋯
460	461		The workspace context page answered 403 from the off stand-in, as
461	462		intended.
462	463		2. With Docker Compose (2026-10-07, `docker compose up --build`, with
463		−	`API_PORT=18789` because 8789 was taken on this machine). `smoke.sh`
	464	+	`API_PORT=18789` because 8789 was taken on this machine; the store is
	465	+	RustFS 1.0.1, its buckets made by `storage-setup`). `smoke.sh`
464	466		passed every step: the ones above; a second clone answered from the pack
465		−	cache (`Server-Timing: pack;desc=hit`), with the packs in MinIO's
466		−	`g1t-git-packs` and its 7-day rule in place; an access token made in the
467		−	site; `GET /user`, the API index (`mcp_url`, `git_url`), the OAuth
	467	+	cache (`Server-Timing: pack;desc=hit`), with the packs in RustFS's
	468	+	`g1t-git-packs` and its lifecycle rule in place (packs expire after 7
	469	+	days, unfinished uploads are aborted after 1); an access token made in the
	470	+	site; an npm package published to the installation's registry and
	471	+	installed back, its tarball in `g1t-packages`; `GET /user`, the API index (`mcp_url`, `git_url`), the OAuth
468	472		metadata (`issuer` the API's address, `authorization_endpoint` on the
469	473		site), MCP's 401 challenge and `tools/list`; a pull request from a branch
470	474		and one from a fork (`create_pull_request` without a branch: a fork in
⋯
473	477		shown `waiting`, taken out (`unqueue`), the queue turned off and the
474	478		pull request merged; and `scheduler.mjs --once`, every handler `ok`.
475	479		The repository page's clone box and MCP line named the installation's
476		−	own addresses, with no social card tags.
477		−	3. The clone pack cache against MinIO without the stack.
478		−	`node services/repos/dev/clone-check.mjs --s3` (MinIO in Docker):
479		−	misses then hits for full and shallow clones over protocol v2 and v0, a
480		−	miss after the refs version moves, five whole packs in the bucket (12 MB
481		−	each, so uploaded in parts), no unfinished upload, and the expiry rule.
	480	+	own addresses, with no social card tags. By hand against the same
	481	+	stack: `docker push` and `docker pull` of an image with a 20 MB layer
	482	+	(uploaded in 10 MiB parts), the layer read back from `/v2/.../blobs/`
	483	+	with a matching digest, the `g1t-backups` bucket present (empty: the
	484	+	runner cuts bundles), no unfinished upload in any bucket, and the
	485	+	guide's upgrade copy from an old MinIO volume into RustFS.
	486	+	3. The clone pack cache against RustFS without the stack.
	487	+	`node services/repos/dev/clone-check.mjs --s3` (RustFS in Docker, the
	488	+	bucket and lifecycle rule made with the AWS CLI): misses then hits for
	489	+	full and shallow clones over protocol v2 and v0, a miss after the refs
	490	+	version moves, five whole packs in the bucket (12 MB each, multipart
	491	+	objects of 3 parts), each reading back at its listed size, no
	492	+	unfinished upload, and both lifecycle rules.
482	493
483	494		### Not verified, or not working yet
484	495
⋯
521	532		a registry directory, a development feature like the rest. If the API
522	533		starts and a binding says `[not connected]`, restart the container. The
523	534		phase 2 launcher serves both from one workerd.
524		−	- The MinIO image. MinIO stopped publishing `minio/minio` and
525		−	`minio/mc`. The compose file uses a community build (`MINIO_IMAGE`,
526		−	`pgsty/minio`); any S3-compatible store can take its place.
	535	+	- The object store. The compose file runs RustFS (Apache-2.0),
	536	+	pinned to a release tag, and makes its buckets with the AWS CLI
	537	+	(Apache-2.0). Any S3-compatible store can take its place
	538	+	(`S3_ENDPOINT`), given the same buckets and the packs' lifecycle rule.
	539	+	Installations started before 2026-10-07 kept these files in MinIO, in
	540	+	the `g1t-packages` volume; the guide's "Upgrade" section copies them
	541	+	across.
527	542		- Cron goes through Wrangler's local API. `scheduler.mjs` asks
528	543		`/cdn-cgi/local/explorer/api/local/scheduled`, a development endpoint
529	544		that may change between Wrangler releases (pinned by the lockfile).

+2−2View BlameViewed

21	21		// backup's.
22	22		// --repo-id <id> the repository by id (no database needed with --bundles).
23	23		// --bundles <dir> read the bucket from a local copy (`backups/<id>/...`
24		−	// under it, as `mc mirror` or `rclone copy` leave it)
25		−	// instead of R2, e.g. a self-hosted MinIO's.
	24	+	// under it, as `aws s3 sync` or `rclone copy` leave it)
	25	+	// instead of R2, e.g. a self-hosted store's.
26	26		// --live <url or path> the live repository to compare with; default
27	27		// https://g1t.sh/<workspace>/<name>.git.
28	28		// --keep keep the temporary directory, and say where it is.

+1−1View BlameViewed

1	1		//! Where packages' files are kept: the `BlobStore` port (crates/blobstore),
2		−	//! with R2 behind it on Cloudflare and any S3-compatible storage (MinIO in
	2	+	//! with R2 behind it on Cloudflare and any S3-compatible storage (RustFS in
3	3		//! the compose file) when self-hosted. BLOB_STORE chooses: `r2` (the
4	4		//! default) or `s3`.
5	5		//!

+56−33View BlameViewed

11	11		// cd services/repos && node ../../scripts/build-rust-worker.mjs
12	12		// node dev/clone-check.mjs
13	13		//
14		−	// With `--s3`, packs are kept in MinIO instead of local R2, as a
15		−	// self-hosted installation keeps them (PACK_STORE=s3): it starts MinIO in
16		−	// Docker, makes the `g1t-git-packs` bucket with the same expiry rule the
17		−	// compose file gives it, and checks that what was kept is there, whole,
18		−	// with no upload left unfinished.
	14	+	// With `--s3`, packs are kept in RustFS instead of local R2, as a
	15	+	// self-hosted installation keeps them (PACK_STORE=s3): it starts RustFS in
	16	+	// Docker, makes the `g1t-git-packs` bucket with the same lifecycle rule the
	17	+	// compose file gives it (with the AWS CLI, as the compose file does), and
	18	+	// checks that what was kept is there, whole, with no upload left
	19	+	// unfinished.
19	20		//
20	21		// node dev/clone-check.mjs --s3
21	22		//
22		−	// GITSTORE_PORT, REPOS_PORT and MINIO_PORT move it off 8799, 8791 and 9010.
	23	+	// GITSTORE_PORT, REPOS_PORT and S3_PORT move it off 8799, 8791 and 9010.
	24	+	// RUSTFS_IMAGE and AWS_CLI_IMAGE choose other images.
23	25		//
24	26		// Needs node, git (with git-http-backend) and the repository's npm
25	27		// packages; with `--s3`, Docker too.
⋯
56	58		);
57	59		const KEY = "acme--rocket";
58	60		const children = [];
59		−	// --s3: packs in MinIO (see the top).
	61	+	// --s3: packs in RustFS (see the top). The images are the compose file's.
60	62		const S3 = process.argv.includes("--s3");
61		−	const MINIO = "g1t-clone-check-minio";
62		−	const MINIO_PORT = process.env.MINIO_PORT ?? "9010";
63		−	const MINIO_USER = "g1t";
64		−	const MINIO_PASSWORD = "g1t-clone-check-secret";
	63	+	const STORAGE = "g1t-clone-check-rustfs";
	64	+	const S3_PORT = process.env.S3_PORT ?? "9010";
	65	+	const S3_USER = "g1t";
	66	+	const S3_PASSWORD = "g1t-clone-check-secret";
	67	+	const RUSTFS_IMAGE = process.env.RUSTFS_IMAGE ?? "rustfs/rustfs:1.0.1";
	68	+	const AWS_CLI_IMAGE = process.env.AWS_CLI_IMAGE ?? "amazon/aws-cli:2.37.10";
65	69		const PACKS_BUCKET = "g1t-git-packs";
66		−	/** `mc` inside the MinIO container, against itself. */
67		−	const mc = (args, options = {}) => run("docker", ["exec", MINIO, "mc", ...args], options);
	70	+	/** The AWS CLI's `s3api`, in a container on the store's network, against it. */
	71	+	const s3api = (args, options = {}) =>
	72	+	run("docker", [
	73	+	"run", "--rm", "--network", `container:${STORAGE}`,
	74	+	"-e", `AWS_ACCESS_KEY_ID=${S3_USER}`, "-e", `AWS_SECRET_ACCESS_KEY=${S3_PASSWORD}`, "-e", "AWS_DEFAULT_REGION=us-east-1",
	75	+	AWS_CLI_IMAGE, "--endpoint-url", "http://localhost:9000", "--output", "json", "s3api", ...args,
	76	+	], options);
	77	+	/** The same, its answer parsed. */
	78	+	const s3json = (args) => JSON.parse(s3api(args).stdout.trim() \|\| "{}");
68	79		// Wrangler from the repository's packages, run with node: no shell to quote for.
69	80		const WRANGLER = join(dirname(createRequire(join(service, "package.json")).resolve("wrangler/package.json")), "bin/wrangler.js");
70	81
⋯
163	174		});
164	175		sql("INSERT INTO repos (id, namespace, name, is_private, owner_id, default_branch, refs_version) VALUES ('rep_rocket', 'acme', 'rocket', 0, 'usr_dev', 'main', 1)");
165	176
166		−	// MinIO, with the bucket and expiry rule deploy/self-host/docker-compose.yml makes.
	177	+	// RustFS, with the bucket and lifecycle rule deploy/self-host/docker-compose.yml makes.
167	178		const s3Vars = [];
168	179		if (S3) {
169		−	run("docker", ["rm", "-f", MINIO], { allowFail: true });
	180	+	run("docker", ["rm", "-f", STORAGE], { allowFail: true });
170	181		run("docker", [
171		−	"run", "-d", "--rm", "--name", MINIO, "-p", `${MINIO_PORT}:9000`,
172		−	"-e", `MINIO_ROOT_USER=${MINIO_USER}`, "-e", `MINIO_ROOT_PASSWORD=${MINIO_PASSWORD}`,
173		−	process.env.MINIO_IMAGE ?? "pgsty/minio:latest", "server", "/data",
	182	+	"run", "-d", "--rm", "--name", STORAGE, "-p", `${S3_PORT}:9000`,
	183	+	"-e", `RUSTFS_ACCESS_KEY=${S3_USER}`, "-e", `RUSTFS_SECRET_KEY=${S3_PASSWORD}`, "-e", "RUSTFS_CONSOLE_ENABLE=false",
	184	+	RUSTFS_IMAGE,
	185	+	]);
	186	+	await waitFor(`http://localhost:${S3_PORT}/health/ready`, "RustFS");
	187	+	s3api(["create-bucket", "--bucket", PACKS_BUCKET]);
	188	+	s3api([
	189	+	"put-bucket-lifecycle-configuration", "--bucket", PACKS_BUCKET, "--lifecycle-configuration",
	190	+	JSON.stringify({
	191	+	Rules: [
	192	+	{ ID: "expire-packs", Status: "Enabled", Filter: { Prefix: "packs/" }, Expiration: { Days: 7 } },
	193	+	{ ID: "abort-unfinished-uploads", Status: "Enabled", Filter: { Prefix: "" }, AbortIncompleteMultipartUpload: { DaysAfterInitiation: 1 } },
	194	+	],
	195	+	}),
174	196		]);
175		−	await waitFor(`http://localhost:${MINIO_PORT}/minio/health/ready`, "MinIO");
176		−	mc(["alias", "set", "local", "http://localhost:9000", MINIO_USER, MINIO_PASSWORD]);
177		−	mc(["mb", "--ignore-existing", `local/${PACKS_BUCKET}`]);
178		−	mc(["ilm", "rule", "add", "--prefix", "packs/", "--expire-days", "7", `local/${PACKS_BUCKET}`]);
179	197		for (const [name, value] of Object.entries({
180	198		PACK_STORE: "s3",
181	199		PACK_S3_BUCKET: PACKS_BUCKET,
182		−	S3_ENDPOINT: `http://localhost:${MINIO_PORT}`,
	200	+	S3_ENDPOINT: `http://localhost:${S3_PORT}`,
183	201		S3_REGION: "us-east-1",
184		−	S3_ACCESS_KEY_ID: MINIO_USER,
185		−	S3_SECRET_ACCESS_KEY: MINIO_PASSWORD,
	202	+	S3_ACCESS_KEY_ID: S3_USER,
	203	+	S3_SECRET_ACCESS_KEY: S3_PASSWORD,
186	204		})) {
187	205		s3Vars.push("--var", `${name}:${value}`);
188	206		}
⋯
209	227		if (S3) {
210	228		// Fills finish after git has its answer: give the last one a moment.
211	229		await new Promise((resolve) => setTimeout(resolve, 3000));
212		−	const listed = mc(["ls", "--recursive", "--json", `local/${PACKS_BUCKET}/packs/`]).stdout.trim().split("\n").filter(Boolean).map((line) => JSON.parse(line));
213		−	const sizes = listed.map((entry) => entry.size);
	230	+	const listed = s3json(["list-objects-v2", "--bucket", PACKS_BUCKET, "--prefix", "packs/"]).Contents ?? [];
	231	+	const sizes = listed.map((entry) => entry.Size);
214	232		// Nine clones: four kinds twice, each kept once, and one more after the refs moved.
215		−	check("the packs are in MinIO, one per distinct clone", listed.length === 5, `${listed.length}: ${sizes.join(", ")}`);
	233	+	check("the packs are in RustFS, one per distinct clone", listed.length === 5, `${listed.length}: ${sizes.join(", ")}`);
216	234		check("the full clone's pack went up in parts", sizes.some((size) => size > 5 * 1024 * 1024), `largest ${Math.max(...sizes)}`);
217		−	const incomplete = mc(["ls", "--recursive", "--incomplete", `local/${PACKS_BUCKET}`]).stdout.trim();
218		−	check("no upload is left unfinished", incomplete === "", incomplete);
219		−	const rules = mc(["ilm", "rule", "ls", "--json", `local/${PACKS_BUCKET}`]).stdout;
220		−	check("the bucket expires packs after 7 days", /"Days":\s*7/.test(rules) && rules.includes("packs/"));
	235	+	// A multipart object's ETag ends in -<number of parts>.
	236	+	check("the large pack is one multipart object", listed.some((entry) => /-\d+"?$/.test(entry.ETag ?? "")), listed.map((entry) => entry.ETag).join(", "));
	237	+	const short = listed.filter((entry) => s3json(["head-object", "--bucket", PACKS_BUCKET, "--key", entry.Key]).ContentLength !== entry.Size);
	238	+	check("every pack reads back whole", short.length === 0, short.map((entry) => entry.Key).join(", "));
	239	+	const incomplete = s3json(["list-multipart-uploads", "--bucket", PACKS_BUCKET]).Uploads ?? [];
	240	+	check("no upload is left unfinished", incomplete.length === 0, incomplete.map((upload) => upload.Key).join(", "));
	241	+	const rules = s3json(["get-bucket-lifecycle-configuration", "--bucket", PACKS_BUCKET]).Rules ?? [];
	242	+	check("the bucket expires packs after 7 days", rules.some((rule) => rule.Expiration?.Days === 7 && rule.Filter?.Prefix === "packs/"), JSON.stringify(rules));
	243	+	check("the bucket aborts uploads unfinished after a day", rules.some((rule) => rule.AbortIncompleteMultipartUpload?.DaysAfterInitiation === 1));
221	244		}
222	245		} catch (error) {
223	246		console.error(error);
⋯
227	250		if (process.platform === "win32") spawnSync("taskkill", ["/pid", String(child.pid), "/t", "/f"], { stdio: "ignore" });
228	251		else child.kill();
229	252		}
230		−	if (S3) run("docker", ["rm", "-f", MINIO], { allowFail: true });
	253	+	if (S3) run("docker", ["rm", "-f", STORAGE], { allowFail: true });
231	254		try {
232	255		rmSync(work, { recursive: true, force: true });
233	256		} catch {}

+1−1View BlameViewed

7	7		//! again after [`Settings::full_every`] incremental ones, so a restore
8	8		//! never reads a long chain. Bundles and a manifest that lists the chain
9	9		//! are kept in object storage through the `BlobStore` port: the BACKUPS R2
10		−	//! bucket hosted, any S3-compatible store (MinIO in the compose file)
	10	+	//! bucket hosted, any S3-compatible store (RustFS in the compose file)
11	11		//! self-hosted, as BACKUP_STORE says.
12	12		//!
13	13		//! ```text

+1−1View BlameViewed

89	89		/// Where packs are kept, by the names of the service's bindings and
90	90		/// variables: the `GIT_PACKS` R2 bucket on Cloudflare or, when PACK_STORE
91	91		/// is `s3`, the bucket PACK_S3_BUCKET names on the installation's
92		−	/// S3-compatible store (`g1t-git-packs` on MinIO in the self-host compose
	92	+	/// S3-compatible store (`g1t-git-packs` on RustFS in the self-host compose
93	93		/// file). On either an object exists only once it is whole: a `put` makes
94	94		/// it in one write, and a multipart upload is nothing anyone can read
95	95		/// until it is completed.