g1t/docs/SELF_HOSTING.md

562 lines44,635 bytesCodeBlame
1# Self-hosting g1t
2
3The goal (2026-10-05): g1t should not be locked to Cloudflare. Anyone should
4be able to run it on their own machine with `docker compose up`. The free
5core is MIT and self-hostable; managed hosting at g1t.sh is the paid
6product, and it stays on Cloudflare. Self-hosting must never make hosted
7g1t worse, so hosted code paths do not change to make room for it.
8
9This document covers:
10
111. An inventory of every Cloudflare dependency in the code.
122. The design: ports and adapters, with the runtime choice weighed.
133. What phase 1 ships today: `deploy/self-host/`, and what was verified.
144. The phased plan, with estimates.
155. The risks.
16
17The user guide is `apps/docs/src/content/docs/guides/self-hosting.md`.
18
19## The short version
20
21- **g1t is already shaped for this.** Every service talks to every other
22 over plain HTTP: `POST /rpc/<method>` with a JSON body (`crates/kit`,
23 `packages/contracts/src/clients.ts`). Every database is SQLite (D1).
24 Git storage sits behind a `GitStore` port (`services/repos/src/store.rs`),
25 and everything past that port speaks git's smart HTTP to a remote URL
26 with a bearer token.
27- **The fastest credible path is to run the Workers themselves in
28 workerd**, the open-source Workers runtime, not to port them. Phase 1
29 runs every core Worker unchanged (the same WebAssembly and the same
30 bundles that deploy to Cloudflare) in one workerd process under
31 `wrangler dev`. D1, KV and Queues are kept on a volume as SQLite files.
32- **Three things replace Cloudflare-only bindings**, as small Workers bound
33 in their place, with no change to the services:
34 - `ARTIFACTS` becomes an Artifacts-compatible shim in front of a git
35 store, which keeps plain bare repositories on disk and serves them
36 with `git http-backend`.
37 - `EMAIL` becomes a shim that logs each message and hands it to Mailpit,
38 which can relay to any SMTP server.
39 - The runner and the context hub are bound to an "off" Worker, which
40 answers "agents are off" instead of failing.
41- **Proven on this machine with Docker:** sign up, confirm the email
42 through Mailpit, create a workspace and a repository, push and clone over
43 HTTP (the second clone from the clone pack cache in MinIO), open an
44 issue, and browse code, commits and files in the site; then the REST API,
45 OAuth metadata and MCP on their own port, pull requests from a branch and
46 from a fork merged onto `main`, the merge queue taking a pull request and
47 giving it back, and every cron handler the scheduler runs. All of it runs
48 against local storage. See [Phase 1: what works today](#3-phase-1-what-works-today).
49- **Long term:** keep workerd as the runtime, because it is what hosted
50 runs. Replace `wrangler dev` with a production workerd configuration.
51 Move the binding shims into code-level ports in `g1t_kit` and a TS
52 `@g1t/platform` package, so each primitive has a hosted and a
53 self-hosted adapter behind one interface.
54
55## 1. Inventory
56
57The sources are every `wrangler.jsonc` plus a grep of the code. Coupling
58is graded:
59
60- **thin**: one call site or a config switch;
61- **adapter**: already behind a port, or easy to put behind one;
62- **woven**: the logic is shaped around the product.
63
64### By primitive
65
66| Primitive | Where | Coupling | Self-hosted equivalent |
67| --- | --- | --- | --- |
68| **Workers runtime**, service bindings | Every service. Rust through `worker` 0.8 (`#[event(fetch\|queue\|scheduled)]`, `Env`, `Fetcher`); TS as `export default { fetch, queue, scheduled }` | woven (as a runtime), thin (as an API) | **workerd**: the same runtime, open source. Service bindings work as they do hosted. Calls are HTTP (`POST /rpc/<method>`), so a native port could use plain HTTP clients. |
69| **Workers RPC** (JS methods across a binding) | Only `RUNNER`: `RunnerService extends WorkerEntrypoint` (`services/runner/src/index.ts:626`). `apps/web` calls `env.RUNNER.enabled/run/plan/...` directly in 11 routes. | thin | workerd supports it. A native port needs these on `/rpc/*` as well; the runner already has a `fetch` shim for Rust callers. |
70| **D1** | System of record for 13 services. Rust: `env.d1("DB")`; TS: `D1Database`; `db.batch()` in `crates/kit` `rename` | woven (SQL), thin (API) | **SQLite files**. workerd/Miniflare implements D1 on SQLite, and the same `migrations/` apply with `wrangler d1 migrations apply --local`. A native port would need a `Database` port over `rusqlite`/`better-sqlite3`; the SQL is already SQLite, including FTS5. |
71| **KV** | `BLOBS`: Actions artifacts and cache (`apps/api/src/blobs.rs`, `apps/web/app/lib/artifacts.server.ts`). `AVATARS`: `services/identity/src/avatars.rs`, `apps/web/workers/app.ts`, `services/og`. `DOMAINS`: `services/deployments/src/domains.ts`, `services/pages` | thin | Miniflare KV on disk (SQLite plus blob files). Natively: a `BlobStore` port on the filesystem or S3/MinIO. |
72| **Queues**: the event bus | Producer: `services/events` `BUS.sendBatch` (`lib.rs:67`). The consumer writes the log, then fans out to every binding named `SUBSCRIBER_*` (`lib.rs:161`). Twelve consumers, one queue each. Private job queues in search (`g1t-search-jobs`) and context (`g1t-context-jobs`); consumers branch on the queue name. | woven | Miniflare Queues: in-process and persisted, which works today. Natively: a `Bus` port with a SQLite outbox and a poller per subscriber, or NATS/Redis Streams. At-least-once delivery and idempotent consumers are already the contract. |
73| **Durable Objects** | Only `AttemptSandbox` (runner), as the containers library's base class. Uses `ctx.storage.get/put/delete`, `schedule()` (alarm), `idFromName`/`idFromString`, DO RPC (`run`, `destroy`, `noteBlocked`). **Not used:** WebSocket hibernation, raw `alarm()`, `ctx.storage.sql`, `ctx.exports`. | woven, in the runner only | workerd supports Durable Objects (on-disk SQLite). Runner state can move to the sandbox supervisor (phase 2). |
74| **Containers** (`@cloudflare/containers`) | `services/runner`: one sandbox per agent run, Actions job and deploy build. `sleepAfter`, `start({ envVars, enableInternet })`, `onStop`. Image: `services/runner/Dockerfile` (node 24, git, toolchains, Claude Code, `g1t-runner`). | woven | **Docker or Podman** through the socket, with the same image. Wrangler can already run Containers locally through Docker; whether that covers outbound interception has to be tested. |
75| **Outbound interception** (guardrails egress) | `services/runner/src/guard.ts` (`egress()`, `abuse()`), `egress.ts` (`sandboxHosts`, `buildHosts`, `EGRESS_CA = /etc/cloudflare/certs/cloudflare-containers-ca.crt`), `AttemptSandbox.outboundHandlers`, `interceptHttps = true`, `setOutboundHandler("egress", { hosts })`, `setOutboundByHost("sandbox.g1t.internal", "abuse")` (a sandbox reporting it stopped itself for mining; `crates/runner/src/abuse.rs`) | woven | The sandbox joins an internal network with no route out, and gets `HTTP(S)_PROXY` pointing at an allow-list proxy that checks the `CONNECT` host. The CA bundle is then not needed, because nothing is re-signed. Blocked hosts are reported to the runner as `noteBlocked` does today. |
76| **Artifacts** (git storage) | Only `services/repos/src/store.rs` (`ArtifactsStore`, behind the `GitStore`/`GitRepo` traits). Methods used: `create`, `get`; then `info`, `createToken`, `log`, `readCommit`, `readTree`, `readBlob`, `readFile`, `fork`, and `[Symbol.dispose]`. Everything else (push, fetch, landing, catch-up, import, ref listing) is smart HTTP to `info().remote` with `Bearer <token>`: `land.rs`, `catch_up.rs`, `refs.rs`, `import.rs`, `git_http.rs`. | adapter | **Bare repositories on disk plus `git http-backend`.** Built in phase 1: `deploy/self-host/gitstore`. Forks are local clones with hard links. Tokens are HMAC-signed, scoped, and expire. |
77| **Cache API** | `services/repos/src/store.rs` (trees and blobs up to 1 MiB, by hash), `apps/web/workers/app.ts` (avatars), `services/og` | thin, optional | Miniflare's cache, or none. Every use tolerates a miss. |
78| **Vectorize** | `services/context/src/index.ts` (`VECTORS.upsert/deleteByIds/query`); optional, guarded by `if (!AI \|\| !VECTORS)` | thin | **sqlite-vec** (default: one file, next to D1), pgvector or Qdrant behind a `VectorIndex` port; or off, which already degrades to keyword search. |
79| **Workers AI** | `services/context` only: `@cf/baai/bge-base-en-v1.5` embeddings (768 dims) | thin | An OpenAI-compatible `/v1/embeddings` endpoint (Ollama, vLLM, LM Studio, or a hosted API) behind an `Embedder` port. Changing models means re-embedding (a backfill job already exists). |
80| **AI Gateway** | `services/runner/src/model-env.ts` (`modelEnv`), `services/models/src/route.ts:70` (gateway URL, `cf-aig-*` headers), `services/billing/src/keeper.rs` (reads gateway logs to settle) | thin | Optional already: an empty `AI_GATEWAY_ID` goes straight to the provider. Any Anthropic- or OpenAI-compatible base URL works for a workspace's own provider. |
81| **Workers for Platforms** | `services/pages` (dispatcher: `env.APPS.get(script).fetch`), `services/deployments/src/cloudflare.ts` (script and asset upload through the REST API) | woven | Phase 1: off. Later: a self-hosted app host in workerd, using the Worker Loader binding to load uploaded scripts, or a workerd per app (see [Deployments](#deployments)). |
82| **Cloudflare for SaaS** (custom hostnames) | `services/deployments/src/custom-hostnames.ts` (`/zones/{id}/custom_hostnames`) | thin | Caddy with on-demand TLS, asking g1t whether a hostname is allowed. |
83| **Cloudflare REST API** | deployments: script upload, list, delete, assets, GraphQL usage. Billing keeper: AI Gateway logs, `billable-usage`, GraphQL container usage. Ops scripts in `scripts/`. | thin (deployments), woven (keeper pricing) | Deployments: the app-host adapter. Keeper: off when self-hosted, because there is no bill to reconcile. |
84| **Email Sending** | `services/identity/src/email.rs` (`EMAIL.send({to, from, subject, text, html})`); callers: verification, password reset, `admin.rs` limit warnings | thin | Built in phase 1: a shim that logs and hands mail to Mailpit, which relays over SMTP. Later: a `Mailer` port with an SMTP adapter. |
85| **Cloudflare Access** | `apps/sudo/app/lib/access.ts` (verifies `Cf-Access-Jwt-Assertion` against `/cdn-cgi/access/certs`, `ACCESS_AUD`, `STAFF_EMAILS`) | woven, in sudo only | A local admin flag: `G1T_ADMINS` usernames checked against the normal g1t session. Self-hosters rarely need sudo, which is about billing. |
86| **Cron Triggers** | actions (every minute), webhooks (every minute), identity (`*/15`), security (`*/30`), repos and packages (hourly), events (daily), billing (`*/15` and daily), deployments (`*/10`), runner (`*/5`) | thin | workerd runs `scheduled()` when asked, but never on its own. `deploy/self-host/scheduler.mjs` asks: once a minute, inside the g1t container, it runs each due cron through Wrangler's local API (`POST /cdn-cgi/local/explorer/api/local/scheduled?worker=<name>`, answered only on localhost), the same handler Cron Triggers run. The services and crons come from `schedules.json`, which `configs.mjs` writes from each `wrangler.jsonc` for the services in its `SELF_HOST_CRONS`: repos, events, identity, security, webhooks and packages. Not run: actions (it would start scheduled workflows with no runner), billing (Cloudflare and Stripe) and deployments (Cloudflare's API). A handler still running from the minute before is not started again, and one is given up on after ten minutes. `scheduler.mjs --once` runs every cron of every service now and exits non-zero if one failed (smoke.sh uses it). The status page has its own loop (`status.sh`). |
87| **`cloudflare:workers` imports** | `apps/web` (`env` in 15 files), `apps/sudo`, `services/runner` (`WorkerEntrypoint`) | thin | Provided by workerd. A Node port would pass `env` through context instead. |
88| **Static Assets** | `apps/web` (Vite plugin build), `apps/docs`, `apps/sudo` (`run_worker_first`) | thin | workerd serves them. |
89| **`placement`, `observability`, routes, custom domains** | every `wrangler.jsonc` | config only | Dropped by `deploy/self-host/configs.mjs`. |
90| **`cf-ray`** | Used as an audit request id, with a fallback: `services/repos/src/run_access.rs:131`, `apps/api/src/audit.rs:37` | thin | Falls back already. |
91| **R2** | `services/packages` (`BLOBS`: container layers and other package files), `services/repos` (`BACKUPS`: nightly backup bundles; `GIT_PACKS`: the clone pack cache), the API's Actions cache (`ACTIONS_CACHE`), the runner's downloads | thin | **S3-compatible storage**: the `BlobStore` port in `crates/blobstore` has an R2 adapter and an S3 one (`s3.rs`, SigV4 over fetch); each service names its own bucket (`BLOB_STORE`/`S3_BUCKET` for packages, `BACKUP_STORE`/`BACKUP_S3_BUCKET` for backups, `PACK_STORE`/`PACK_S3_BUCKET` for clone packs), run against MinIO in the compose file. |
92| **Not used** | Hyperdrive, Workflows, Analytics Engine, Browser Rendering, Images, Turnstile, Secrets Store, `connect()`, HTMLRewriter, `request.cf` | — | — |
93
94### By service
95
96The deployable units, their stage and what each is built from are listed in
97`deploy/stack.jsonc` (see `docs/DEPLOYING.md`); its `self_host` field is
98what `deploy/self-host/configs.mjs` runs, turns off or leaves out. A test
99checks this table names every unit.
100
101| Service | Runs on | Cloudflare dependencies beyond Workers and D1 | Phase 1 self-hosted |
102| --- | --- | --- | --- |
103| `apps/web` | TS Worker plus assets | KV (`BLOBS`, `AVATARS`), Cache API, `cloudflare:workers` `env`, RPC to `RUNNER` | Runs unchanged |
104| `apps/api` | Rust Worker | KV `BLOBS`, R2 `ACTIONS_CACHE`; `api.g1t.sh`/`mcp.g1t.sh` addresses (now the `API_URL`, `MCP_URL` and `SITE_URL` settings, hosted defaults when unset: `src/addresses.rs`) | Runs in a second workerd on its own port (`API_PORT`, 8789; `self_host: "separate"`), started by `start.sh` once the first is up. Its service bindings reach the other Workers through Wrangler's dev registry. `API_URL` (default: `PUBLIC_URL`'s host on `API_PORT`) is its OAuth issuer; MCP is the path `/mcp` on it (`MCP_URL`). Its KV is its own, apart from the site's: Actions artifacts need the runner, which is off |
105| `apps/sudo` | TS Worker plus assets | Access JWT | Not run |
106| `apps/docs` | Static | — | Not run (docs.g1t.sh serves them) |
107| `apps/status` | TS Worker | Email Sending, cron; bound only to billing | Runs in a process of its own (`status.sh`), so it stays up when the site does not |
108| `services/identity` | Rust | Email Sending, KV `AVATARS` | Runs unchanged; `EMAIL` goes to the mail shim |
109| `services/repos` | Rust | **Artifacts**, **R2** (`BACKUPS`), Cache API, optional KV `GIT_CACHE` with `REPOS_KEY`, optional R2 `GIT_PACKS` | Runs unchanged; `ARTIFACTS` goes to the git store, backups to MinIO's `g1t-backups` bucket (`BACKUP_STORE=s3`), and the clone pack cache to `g1t-git-packs` (`PACK_STORE=s3`: the `PackStore` port in `src/pack_cache.rs` over the shared `BlobStore`, multipart, an object only once whole; `minio-setup` gives the bucket a rule that deletes packs after 7 days, and MinIO removes unfinished uploads after 24 hours). Without `GIT_CACHE` and `REPOS_KEY`, credentials and ref listings are kept per isolate only. Its nightly cron queues backups, but bundles are cut by the runner, which is off in phase 1: none are made yet |
110| `services/work` | Rust | Queue consumer | Runs unchanged |
111| `services/events` | Rust | Queues (producer and fan-out) | Runs unchanged; the off services' queues are not produced to |
112| `services/projects` | TS | Queue consumer | Runs unchanged |
113| `services/search` | Rust | Queues (events and its own jobs); FTS5 | Runs unchanged |
114| `services/billing` | Rust | Cron, Cloudflare REST API (keeper), Stripe | Runs with `FREE_WHILE_BUILDING=true` and no Stripe key: nothing is charged |
115| `services/security` | Rust | Queue, cron | Runs unchanged; its cron through `scheduler.mjs` |
116| `services/actions` | Rust | Queue, cron, `ACTIONS_KEY` | Runs; jobs need the runner, which is off |
117| `services/webhooks` | Rust | Queue, cron, `WEBHOOKS_KEY` | Runs; retries through `scheduler.mjs` |
118| `services/integrations` | Rust | Queue, `INTEGRATIONS_KEY` | Runs unchanged |
119| `services/packages` | Rust | **R2** (`BLOBS`), cron, queue, `PACKAGES_TOKEN_SECRET` | Runs with `BLOB_STORE=s3` against the compose file's MinIO (`deploy/self-host/configs.mjs`); no request size limit (`MAX_REQUEST_BYTES` 0 means none) |
120| `services/deployments` | TS | Workers for Platforms, REST API, KV `DOMAINS`, cron | Runs with no API token: nothing deploys |
121| `services/runner` | TS | **Containers**, Durable Objects, outbound interception, AI Gateway, cron | Off: bound to the off Worker |
122| `services/context` | TS | **Vectorize**, **Workers AI**, Queues | Off: bound to the off Worker |
123| `services/models` | TS | AI Gateway; public at `models.g1t.sh` | Not run (only sandboxes call it) |
124| `services/pages` | TS | Dispatch namespace, wildcard routes, KV | Not run |
125| `services/og` | TS | Cache API | Not run (social cards are optional) |
126| `crates/runner` | native, in the sandbox | Talks to `https://api.g1t.sh` and `https://g1t.sh` (hard-coded in the runner Worker); `wrangler deploy --dry-run` for builds | Phase 2 |
127| `crates/sshd` | native | Not deployed; calls `/_internal/ssh/*` endpoints that do not exist yet | Phase 3 |
128
129### Hard-coded hosted addresses
130
131Self-hosting needs one setting, `PUBLIC_URL`, in place of these, and
132`configs.mjs` derives the rest from it. These are settings now, each with
133the hosted address as its default, so hosted g1t sets nothing:
134
135- The site (`apps/web/app/lib/addresses.ts`): `SITE_URL`, `API_URL`,
136 `MCP_URL` and `OG_URL` (empty: no social card tags). The root loader
137 hands them to the page; `meta.ts`, the clone box, agent setup, the
138 pull request and merge box remotes, the tokens page and the OAuth
139 consent's `iss` read them.
140- The API (`apps/api/src/addresses.rs`): `SITE_URL`, `API_URL` (the OAuth
141 issuer) and `MCP_URL` (the protected resource; a path on the API's host
142 self-hosted).
143- Identity's mail (`services/identity/src/email.rs`): `SITE_URL` for
144 links and the logo, `MAIL_FROM` for the sender. The mail shim still
145 rewrites any `https://g1t.sh` link left in a message.
146
147Still hard-coded, for phase 2:
148
149- `apps/web/workers/app.ts` (`DOCS`); defaults in
150 `components/invites-section.tsx`, `components/runners.tsx`,
151 `lib/invites.ts` and `lib/legal.ts`.
152- `services/runner/src/index.ts` (`G1T_API`, `GIT_REMOTE` and the other
153 remotes handed to sandboxes, in 12 places).
154- `services/billing` (`stripe.rs`, `accounts.rs`, `limits.rs`).
155- `crates/sshd` (`G1T_API` default).
156
157There are about 150 occurrences of `g1t.sh` in non-test code. Most are
158docs links and copy, and need no change.
159
160## 2. Design
161
162### Principles
163
1641. **Hosted is the reference.** Hosted code does not change behaviour to
165 make room for self-hosting. A self-hosted adapter is added beside the
166 hosted one, and the hosted one stays the default.
1672. **Swap at the narrowest seam that exists.** A binding-shaped seam (a
168 Worker that offers the same methods as the Cloudflare binding) needs no
169 code change, and is how phase 1 works. A code-level port (a trait or
170 interface with two adapters) is cleaner and testable, and is the long-term
171 shape. Each primitive moves from the first to the second when it is next
172 touched.
1733. **One runtime, two hosts.** The Workers stay Workers. workerd runs them
174 self-hosted, so one build serves both and there is no second code path
175 to keep correct.
1764. **Off is a real mode.** Every optional subsystem (agents, context,
177 deployments, billing) has an "off" answer that pages already handle.
178
179### The ports
180
181| Port | Hosted adapter | Self-hosted adapter | Lives in | Status |
182| --- | --- | --- | --- | --- |
183| `GitStore` / `GitRepo` | `ArtifactsStore` | Git store (bare repos, `git http-backend`) through an Artifacts-compatible shim; later a `LocalGitStore` that calls the git store's HTTP API from Rust directly | `services/repos/src/store.rs` (exists) | **Built** (binding level) |
184| `Mailer` | Email Sending binding | SMTP (through Mailpit relay now; a direct SMTP adapter later) | `g1t_kit::mail` | **Built** (binding level) |
185| `Sandbox` | `AttemptSandbox` (Containers, DO) | `DockerSandbox`: a supervisor that starts the runner image through the Docker API | `services/runner` (TS) | Phase 2 |
186| `Egress` | Containers outbound handler | Allow-list HTTP(S) proxy on an internal network | `services/runner` | Phase 2 |
187| `ModelRoute` | AI Gateway, or direct | Any Anthropic/OpenAI-compatible base URL | `services/models`, `runner/model-env.ts` | Exists (env-switchable) |
188| `Embedder` | Workers AI | OpenAI-compatible `/v1/embeddings` | `services/context` | Phase 3 |
189| `VectorIndex` | Vectorize | sqlite-vec / pgvector / Qdrant | `services/context` | Phase 3 |
190| `AppHost` | Workers for Platforms plus REST API | workerd app host (Worker Loader) | `services/deployments` | Phase 3 |
191| `Domains` | Cloudflare for SaaS | Caddy on-demand TLS | `services/deployments` | Phase 3 |
192| `AdminAuth` | Cloudflare Access | `G1T_ADMINS` plus the normal session | `apps/sudo` | Phase 4 |
193| `Bus` | Queues | Miniflare Queues now; SQLite outbox later | `services/events`, `g1t_kit` | Works (runtime) |
194| `Database` | D1 | SQLite files through workerd | — | Works (runtime) |
195| `Blobs` | KV | Miniflare KV now; filesystem/S3 later | — | Works (runtime) |
196| `Scheduler` | Cron Triggers | `scheduler.mjs`: a ticker that calls `scheduled()` through Wrangler's local API | `deploy/self-host` | **Built** for the services in `SELF_HOST_CRONS`; each handler checked by `smoke.sh` |
197| `PackStore` | R2 `GIT_PACKS` | S3 (`PACK_STORE=s3`) through `crates/blobstore` | `services/repos/src/pack_cache.rs` | **Built** |
198| `UsageKeeper` | Cloudflare bill plus AI Gateway logs | None (billing off) | `services/billing` | Off |
199
200For Rust, the code-level ports go in `crates/kit` as traits (`g1t_kit::ports`),
201with the Cloudflare adapters next to them. For TypeScript, they go in a new
202`packages/platform` package, with each adapter in its own module so a
203hosted bundle never pulls in a self-hosted adapter. The adapter is chosen at
204startup from one setting, `G1T_MODE=hosted|self`, never per request.
205
206### The runtime: workerd, or native binaries
207
208| | **workerd (phase 1: `wrangler dev`; later a plain workerd config)** | **Native: Rust on axum/hyper, TS on Node** |
209| --- | --- | --- |
210| Effort to first boot | **Done.** About a day, almost all of it shims and config. | Weeks. The Rust services use `worker::*` throughout: `Env`, `Fetcher`, `D1Database`, the `#[event]` macros and `js_sys` interop. Each needs an abstraction layer before it compiles natively. TS needs an `env` provider in place of `cloudflare:workers`, plus Queues, D1 and KV clients. |
211| Fidelity to hosted | **The same runtime and the same bundles.** A bug self-hosted is a bug hosted. | A second implementation of every platform API, with its own bugs. |
212| Performance | Good for one node: WebAssembly in V8, SQLite on local disk. Single-threaded per isolate; plenty for a team. | Better per core, and multi-threaded. That matters only at a scale where people use g1t.sh. |
213| Maintenance | Low. New hosted features run self-hosted for free, unless they add a new Cloudflare-only binding. The config generator then drops it or binds a stand-in. | High. Every feature is built twice, or behind a port that both sides keep honest. |
214| Scale-out | One process, one disk. D1-on-SQLite is single-writer. | Could use Postgres and many processes. |
215| Operational risk | `wrangler dev` is a development tool (see risks). Moving to `workerd serve` with a generated config, or a small Miniflare-API launcher, removes the dev-tool surface. | Conventional. |
216
217**Recommendation.**
218
219- **Phase 1:** workerd under `wrangler dev`, as built in `deploy/self-host/`.
220 It needs no change to any service and is proven end to end.
221- **Phase 2:** replace `wrangler dev` with a launcher that drives Miniflare's
222 API directly, or a generated `workerd` config. It should expose only the
223 site and the API, with no dev endpoints, run cron, and run as a proper
224 service.
225- **Long term:** keep workerd as the runtime and push the remaining
226 Cloudflare-only bindings behind code-level ports. Compile to native only
227 what already is native (the runner binary, sshd), or a service where
228 workerd is a real limit, case by case.
229
230A full native port is not worth it. It would double the maintenance of
231every feature, which is the opposite of what makes self-hosting
232sustainable for a small team.
233
234### Sandboxes (phase 2)
235
236- **Engine.** Docker or Podman through the socket, mounted into a small
237 supervisor (`g1t-sandboxd`), not into the workerd container. The
238 supervisor exposes the runner's `Sandbox` port over HTTP: start with
239 env, stop, status, and the exit report that becomes `onStop`. It uses
240 the **same image** (`services/runner/Dockerfile`).
241- **Runner.** The runner Worker runs in workerd with `AttemptSandbox`
242 replaced by a `DockerSandbox` adapter. Run state that lives in
243 `ctx.storage` moves to the supervisor's SQLite; `schedule()`/`timeUp`
244 becomes a supervisor timer.
245- **Egress guardrails.** Each restricted sandbox joins an internal Docker
246 network with no default route. Its `HTTP_PROXY`/`HTTPS_PROXY` point at an
247 allow-list proxy (for example a small Go or Node `CONNECT` proxy) that
248 admits the hosts `sandboxHosts()` computes and reports refusals.
249 Nothing is intercepted, so no CA is injected. `EGRESS=off` puts the
250 sandbox on a normal network.
251- **Addresses.** `G1T_API`, `GIT_REMOTE` and the rest become
252 `PUBLIC_URL`-derived settings. Sandboxes reach the site and API on the
253 compose network.
254- **Shortcut to evaluate first.** Wrangler can already run Containers
255 classes locally through Docker. If its local Containers support
256 `setOutboundHandler`, the runner could run nearly unchanged. Test this
257 before building the supervisor.
258
259### Git storage
260
261Artifacts gives g1t: named repositories, scoped short-lived tokens, a smart
262HTTP remote, typed reads (commit, tree, blob, file, log), copy-on-write
263forks, and jurisdictions. g1t uses everything except jurisdictions.
264
265The self-hosted git store (`deploy/self-host/gitstore/server.mjs`, about 450
266lines of Node with no dependencies) provides the same:
267
268| Artifacts | Git store |
269| --- | --- |
270| `create(name, { setDefaultBranch, description })` | `git init --bare --initial-branch`, plus `g1t.json` beside it for metadata |
271| `get(name)` then `info()` | Metadata, `HEAD`, and the last push time; `remote` is `GITSTORE_URL/git/<name>.git` |
272| `createToken(scope, ttl)` | HMAC-SHA256 over `{ key, scope, expiry }` with the shared secret |
273| Smart HTTP remote | `git http-backend`; a read token cannot push |
274| `log`, `readCommit`, `readTree`, `readBlob`, `readFile` | `git rev-list --first-parent`, `cat-file`, `ls-tree` |
275| `fork(name, { defaultBranchOnly })` | `git clone --bare [--single-branch]` with hard-linked objects |
276
277Hosted is untouched: `ArtifactsStore` is still the only adapter compiled
278into `services/repos`. Self-hosted, the `ARTIFACTS` binding is a service
279binding to `deploy/self-host/workers/artifacts`, which offers Artifacts'
280methods and calls the git store. The git store can later gain `git gc`
281scheduling and object-store-backed packs for large installations.
282
283The same git store is hosted g1t's cold fallback for an Artifacts outage
284(docs/ARTIFACTS.md, R12). For that it takes namespaced keys as well as
285plain ones (`g1t-us-1/acme--rocket`, kept at
286`<GITSTORE_ROOT>/g1t-us-1/acme--rocket.git` and served at
287`/git/g1t-us-1/acme--rocket.git`, the shape Artifacts gives remotes), and
288`GITSTORE_READ_ONLY=1` refuses pushes, write tokens and making, forking or
289deleting repositories. `services/repos/src/fallback.rs` calls its API from
290Rust, the start of the `LocalGitStore` adapter: a namespace named in
291`GIT_FALLBACK_NAMESPACES` is served from it, with the shim out of the path.
292Self-hosted installations keep using the shim, with plain keys; nothing
293changes for them.
294
295### Search and context
296
297- **Site search** (`services/search`) is D1 FTS5. It works self-hosted as
298 is (verified: `/search?q=hello` answers 200).
299- **Context hub** (`services/context`) needs an `Embedder` and a
300 `VectorIndex`:
301 - Default: **sqlite-vec** in the context service's own SQLite, with
302 embeddings from an **OpenAI-compatible endpoint**. Ollama's
303 `nomic-embed-text` has the same 768 dimensions as today's
304 `bge-base-en-v1.5`.
305 - Alternatives: pgvector or Qdrant, for installations that already run
306 them.
307 - Off: the service already degrades to keyword search when `AI` or
308 `VECTORS` is missing (`index.ts:919`), so phase 3 can first run context
309 with neither bound.
310
311### Models
312
313Already portable. The model proxy (`services/models`) and the runner's
314model environment take any Anthropic-compatible base URL, and an empty
315`AI_GATEWAY_ID` skips AI Gateway. Self-hosted:
316
317- a workspace connects its own provider under Integrations (Anthropic, or
318 any Anthropic-compatible endpoint, including a local gateway in front of
319 OpenAI-compatible models);
320- "hosted models" are off, because there is no g1t key to spend.
321
322### Deployments
323
324- **Phase 1–2: off.** `services/deployments` runs, but with no
325 `CLOUDFLARE_API_TOKEN` nothing deploys. Its pages and settings still
326 render.
327- **Phase 3: an app host on workerd.** The self-hosted `pages` dispatcher
328 uses workerd's **Worker Loader** binding to load each uploaded app's
329 modules on demand, keyed by deployment id, from the blob store. Static
330 assets are served from the same store. Builds run in Docker sandboxes
331 (phase 2), which already bundle with `wrangler deploy --dry-run`.
332 Wildcard hosts (`*.apps.example.com`) and custom domains go through
333 Caddy with on-demand TLS. The alternative, one workerd process per app,
334 is simpler to isolate but heavier.
335
336### Billing and the feature map
337
338Self-hosted billing is **off by default**: no Stripe, no usage limits,
339no keeper. Billing still runs, because the shell reads `billing.account`
340on every page, but with `FREE_WHILE_BUILDING=true` and no Stripe key.
341
342| Feature | Hosted (g1t.sh) | Self-hosted default | Self-hosted, when turned on |
343| --- | --- | --- | --- |
344| Accounts, workspaces, repos, git over HTTP | On | On | — |
345| Issues, pull requests, review | On | On | — |
346| Merge queue | On | Takes pull requests; testing and landing them needs sandboxes | Phase 2 |
347| Bringing a pull request up to date before it lands (catch-up) | On | Off: needs a sandbox | Phase 2 |
348| Clone pack cache | R2 | MinIO (`g1t-git-packs`) | — |
349| Site search (FTS5) | On | On | — |
350| Email | Email Sending | Mailpit, logged | SMTP relay |
351| Webhooks, integrations | On | On (retries through `scheduler.mjs`) | — |
352| g1t's agent | On | Off | Phase 2: Docker sandboxes plus your own model provider |
353| Guardrails egress | Containers interception | n/a | Phase 2: allow-list proxy |
354| Hosted models (g1t's key) | On (billed) | Off | Never: bring your own |
355| Context hub semantic search | Vectorize plus Workers AI | Off | Phase 3: sqlite-vec plus an OpenAI-compatible embedder |
356| Deployments | Workers for Platforms | Off | Phase 3: workerd app host |
357| Custom domains | Cloudflare for SaaS | Off | Phase 3: Caddy on-demand TLS |
358| Billing, limits, Stripe, keeper | On | Off | Not planned |
359| sudo (staff console) | Access | Off | Phase 4: `G1T_ADMINS` |
360| Git over SSH | Not yet | Off | Phase 3 (`crates/sshd`, which is native already) |
361| REST API, OAuth, MCP | On | On, on `API_PORT` | — |
362| CLI | On | Off | Phase 2 |
363
364### Auth, Access and email
365
366- `apps/sudo` checks Cloudflare Access. Self-hosted, it should check a
367 normal g1t session against `G1T_ADMINS`. That needs an `AdminAuth` port in
368 `workers/app.ts` with two adapters. It is phase 4, because sudo is about
369 billing.
370- Sessions are random tokens stored hashed in D1, with no signing key, so
371 nothing to configure. The cookie is `Secure`: fine on `localhost`, but any
372 other address needs HTTPS. The compose file should gain an optional Caddy
373 service in phase 2.
374- **Email verification is required** before creating anything, and there is
375 no bypass in code. That is why the proof ships a working mail path
376 (Mailpit) rather than "email off". An admin "mark verified" or
377 `G1T_SKIP_EMAIL_VERIFICATION` belongs with `G1T_ADMINS`.
378
379### Configuration, upgrades and backups
380
381- **Today:** environment variables in the compose file (`PUBLIC_URL`,
382 `G1T_PORT`, `API_PORT`, `API_URL`, `MCP_URL`, `MAIL_URL`, `MAIL_FROM`,
383 the S3 store and its buckets). Keys (`ACTIONS_KEY`,
384 `INTEGRATIONS_KEY`, `WEBHOOKS_KEY`, the git store secret) are generated on
385 first start and kept on volumes.
386- **Phase 2:** one `g1t.toml`, read by the launcher and turned into
387 bindings and variables:
388
389 ```toml
390 public_url = "https://git.example.com"
391
392 [mail]
393 smtp = "smtp://user:pass@smtp.example.com:587"
394 from = "g1t <git@example.com>"
395
396 [agents] # off when absent
397 docker = "unix:///var/run/docker.sock"
398 egress = "enforce"
399
400 [context] # off when absent
401 embeddings = "http://ollama:11434/v1"
402 model = "nomic-embed-text"
403
404 [admins]
405 usernames = ["alice"]
406 ```
407
408- **Upgrades.** The container applies every service's D1 migrations to its
409 SQLite file on start (`wrangler d1 migrations apply --local`). Applied
410 migrations are recorded in `d1_migrations`, so this is idempotent. Hosted
411 and self-hosted run the same migration files, which keeps them
412 forward-compatible. Rule to keep: migrations stay additive, or come with
413 a backfill a self-hoster's start can run.
414- **Backups.** Phase 1: stop, then tar the `g1t-data` and `g1t-git`
415 volumes (documented in the guide). Phase 2: online backups with
416 `sqlite3 .backup` per database, or Litestream for continuous
417 replication. The repositories get hosted g1t's nightly bundles
418 (docs/ARTIFACTS.md, R11) once the runner runs: the storage is already
419 configured (`BACKUP_STORE=s3`, the `g1t-backups` bucket that
420 `minio-setup` makes, `BACKUP_S3_BUCKET` to choose another), and the
421 restore drill reads a copy of that bucket
422 (`mc mirror local/g1t-backups ./copy`, then
423 `node scripts/ops/backup-restore-drill.mjs --bundles ./copy --repo-id <id> --live <bare repository>`).
424
425## 3. Phase 1: what works today
426
427Everything is in `deploy/self-host/`:
428
429| File | What it is |
430| --- | --- |
431| `docker-compose.yml` | `g1t` (every core Worker in one workerd on 8787, and the API in a second on 8789), `status`, `gitstore` (bare repositories), `minio` and `minio-setup` (packages, backups and clone packs, with the packs' expiry rule), and `mailpit` (mail). Volumes: `g1t-data`, `g1t-git`, `g1t-packages`, `g1t-status`, `g1t-secrets`. MinIO no longer publishes `minio/minio` or `minio/mc` images; `MINIO_IMAGE` (default `pgsty/minio`, a community build with `mc` in it) is the server. |
432| `Dockerfile` | Compiles the ten Rust services to WebAssembly with `worker-build`, as hosted does. Builds the site with React Router. The runtime image has Node, Wrangler, workerd and the built Workers. |
433| `Dockerfile.dockerignore` | Build-context rules for this image only (the root `.dockerignore` leaves out the site). |
434| `start.sh` | Makes the sealing keys once, writes the configs, applies migrations, runs `wrangler dev` with every config on `0.0.0.0:8787`, persisting to `/data/state`, and the API's `wrangler dev` on `0.0.0.0:8789` once the first answers. Starts `scheduler.mjs`. |
435| `scheduler.mjs` | The cron ticker (see Cron Triggers above). |
436| `configs.mjs` | Derives each self-hosted Wrangler config from the hosted `wrangler.jsonc`. It drops routes, account and placement, rebinds `ARTIFACTS`/`EMAIL` and the off services, and rewrites hosted URLs. Derived, so it cannot drift. |
437| `gitstore/server.mjs`, `gitstore/Dockerfile` | The git store. |
438| `workers/artifacts/index.js` | The `ARTIFACTS` binding, implemented against the git store. |
439| `workers/mail/index.js` | The `EMAIL` binding: logs, then sends to Mailpit. |
440| `workers/off/index.js` | The runner and the context hub when they are off. |
441| `smoke.sh` | The end-to-end check, including the API, pull requests, the merge queue and (with `SCHEDULER_ONCE`) every cron handler. |
442
443Workers running: the site; identity, repos, work, events, projects, search,
444billing, security, actions, webhooks, integrations, packages and
445deployments; the artifacts, mail and two off stand-ins; and, in a second
446workerd, the API.
447
448### Verified
449
450On this machine (Windows 11, Docker Desktop 29.8, engine on Linux):
451
4521. **Without Docker, with local processes.** The git store ran under Node
453 on Windows and every Worker ran under `wrangler dev`, using the configs
454 from `configs.mjs` and the existing Rust builds. `smoke.sh` passed every
455 step: sign-up, confirmation through the logged link, workspace, repo,
456 `git push` and `git clone` over HTTP, issue, and the code, blob and
457 commits pages. Seventeen other pages answered 200: home, workspace,
458 repo overview, issues, pulls, settings, agents, actions, deployments,
459 security, people, usage, explore, search, account settings and tree.
460 The workspace context page answered 403 from the off stand-in, as
461 intended.
4622. **With Docker Compose** (2026-10-07, `docker compose up --build`, with
463 `API_PORT=18789` because 8789 was taken on this machine). `smoke.sh`
464 passed every step: the ones above; a second clone answered from the pack
465 cache (`Server-Timing: pack;desc=hit`), with the packs in MinIO's
466 `g1t-git-packs` and its 7-day rule in place; an access token made in the
467 site; `GET /user`, the API index (`mcp_url`, `git_url`), the OAuth
468 metadata (`issuer` the API's address, `authorization_endpoint` on the
469 site), MCP's 401 challenge and `tools/list`; a pull request from a branch
470 and one from a fork (`create_pull_request` without a branch: a fork in
471 the git store, pushed to with the token, marked ready) merged onto
472 `main`; the merge queue turned on, a pull request merged into it and
473 shown `waiting`, taken out (`unqueue`), the queue turned off and the
474 pull request merged; and `scheduler.mjs --once`, every handler `ok`.
475 The repository page's clone box and MCP line named the installation's
476 own addresses, with no social card tags.
4773. **The clone pack cache against MinIO without the stack.**
478 `node services/repos/dev/clone-check.mjs --s3` (MinIO in Docker):
479 misses then hits for full and shallow clones over protocol v2 and v0, a
480 miss after the refs version moves, five whole packs in the bucket (12 MB
481 each, so uploaded in parts), no unfinished upload, and the expiry rule.
482
483### Not verified, or not working yet
484
485- The merge queue past `waiting`, and catch-up (bringing a pull request
486 up to date before it lands): both need a sandbox, and the runner is off.
487- Actions schedules (`on: schedule`), and billing's and deployments' crons.
488- The CLI, and git over SSH.
489- An OAuth sign-in from start to finish (the metadata and issuer are
490 checked, the consent flow is not).
491- Anything on an address other than `localhost` without HTTPS (the
492 session cookie is `Secure`).
493- Restart durability beyond one restart, upgrades across schema changes,
494 and backup and restore.
495
496## 4. Phased plan
497
498Estimates are for one engineer who knows the codebase, working with
499agents. They include docs and tests.
500
501| Phase | Scope | Estimate |
502| --- | --- | --- |
503| **1. Core forge** | **Done:** compose stack, git store, Artifacts/Email shims, off stand-ins, config generator, smoke test, guide; the API (REST, MCP, OAuth) on its own port with a `PUBLIC_URL`-derived issuer; `PUBLIC_URL`-derived settings in identity mail, the site's meta tags, clone box and agent setup, and the API; the cron ticker; the clone pack cache on S3; pull requests from branches and forks and the merge queue's enqueue and removal in `smoke.sh`. **Left:** optional Caddy for HTTPS; a CI job that builds the images and runs `smoke.sh`; the remaining hard-coded addresses listed above. | 2–3 days left |
504| **2. Agents with Docker sandboxes** | Test Wrangler's local Containers first. Otherwise: `g1t-sandboxd` supervisor, `DockerSandbox` adapter in the runner, egress allow-list proxy and internal network, runner addresses from `PUBLIC_URL`, models through a workspace's own provider, `g1t.toml` and a launcher that replaces `wrangler dev`. | 2–3 weeks |
505| **3. Search, context and deployments** | `Embedder` (OpenAI-compatible) and `VectorIndex` (sqlite-vec first) ports in context; app host on workerd with Worker Loader; Caddy on-demand TLS for app and custom domains; git over SSH through `crates/sshd` plus the missing `/_internal/ssh/*` endpoints. | 3–4 weeks |
506| **4. Parity and upgrade path** | Code-level ports in `g1t_kit` / `@g1t/platform` replacing the binding shims (`LocalGitStore` in Rust, `Mailer` with SMTP); `AdminAuth` for sudo; online backups (Litestream or `.backup`); versioned releases with published images; an upgrade test in CI that migrates a snapshot of the previous release; a self-host column in the docs for every feature. | 3–4 weeks |
507
508Total to parity: about 10–13 weeks. Phase 1 alone is already a credible
509"run it yourself" for the core forge.
510
511## 5. Risks
512
513- **`wrangler dev` is a development tool.** It exposes Miniflare's dev
514 endpoints (`/cdn-cgi/...`, including a local data explorer) on the same
515 port as the site. Treat phase 1 as **localhost or a trusted private
516 network only** until the launcher in phase 2 replaces it. Its flags and
517 behaviour can also change between Wrangler releases. Pin the Wrangler
518 version, as the lockfile already does.
519- **The API reaches the other Workers through Wrangler's dev registry.**
520 Two `wrangler dev` processes in one container find each other through
521 a registry directory, a development feature like the rest. If the API
522 starts and a binding says `[not connected]`, restart the container. The
523 phase 2 launcher serves both from one workerd.
524- **The MinIO image.** MinIO stopped publishing `minio/minio` and
525 `minio/mc`. The compose file uses a community build (`MINIO_IMAGE`,
526 `pgsty/minio`); any S3-compatible store can take its place.
527- **Cron goes through Wrangler's local API.** `scheduler.mjs` asks
528 `/cdn-cgi/local/explorer/api/local/scheduled`, a development endpoint
529 that may change between Wrangler releases (pinned by the lockfile).
530 Actions schedules, billing and deployments are not run.
531- **New Cloudflare-only bindings break self-hosting silently.**
532 `configs.mjs` passes unknown keys through untouched. A new binding type
533 could make `wrangler dev` reach for a remote resource (for example
534 `remote: true`, AI or Vectorize) and prompt for a login. Mitigation: CI
535 that builds the compose stack and runs `smoke.sh` on every change, and an
536 allow-list in `configs.mjs` that fails on unknown binding types.
537- **Artifacts semantics drift.** The shim copies the Artifacts methods g1t
538 uses today. If repos starts using another method (`import`,
539 `listTokens`, `revokeToken`), self-hosted fails at runtime. Mitigation: a
540 contract test that runs `services/repos` against the git store, and the
541 long-term `LocalGitStore` port.
542- **Error codes over RPC.** `ArtifactsStore::create` and `fork` tolerate
543 `ALREADY_EXISTS` by reading the thrown error's `code`. Workers RPC may not
544 carry custom error properties across a service binding. If it does not,
545 retrying a half-finished create fails self-hosted where it would succeed
546 hosted. This was not seen in testing, because creates were never retried.
547- **Single node, single writer.** SQLite (D1 local) and one workerd
548 process suit a team, not a large organisation. Scaling out means
549 Postgres behind a `Database` port, which is a large change and is not
550 planned.
551- **The `Secure` cookie.** It needs HTTPS anywhere but `localhost`. A LAN
552 install over plain HTTP cannot sign in.
553- **Building from a working tree that is mid-change.** The image compiles
554 every Rust service, including ones other work is changing. A service that
555 does not compile breaks the whole image. Released images (phase 4) fix
556 this.
557- **Image size and build time.** The first build compiles ten Rust crates
558 to WebAssembly and installs the site's dependencies. Expect minutes and
559 several GB. Published images remove this for users.
560- **Hard-coded hosted URLs.** The few left (listed above) point at the
561 hosted service until they read a setting: the docs link, invite and
562 runner defaults, and the runner's remotes.