Running g1t yourself: the design, a docker compose proof, and a guide to what works today
16 files+1976−00/16 viewed
| 123 | 123 | { label: 'Audit log', slug: 'guides/audit-log' }, | |
| 124 | 124 | { label: 'Usage and billing', slug: 'guides/usage-and-billing' }, | |
| 125 | 125 | { label: 'Git', slug: 'guides/git' }, | |
| 126 | + | { label: 'Run g1t yourself', slug: 'guides/self-hosting' }, | |
| 126 | 127 | ], | |
| 127 | 128 | }, | |
| 128 | 129 | { |
| 1 | + | --- | |
| 2 | + | title: Run g1t yourself | |
| 3 | + | description: Start the core forge on your own machine with Docker Compose. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | g1t is MIT licensed. You can run the core forge on your own machine: | |
| 7 | + | accounts, workspaces, repositories, git over HTTP, issues and pull | |
| 8 | + | requests, and the site to browse them. Your repositories are plain bare | |
| 9 | + | git repositories on a Docker volume. | |
| 10 | + | ||
| 11 | + | This is an early version. It is for trying g1t out and for small teams on | |
| 12 | + | a private network, not yet for an installation on the open internet. | |
| 13 | + | ||
| 14 | + | ## What works and what is off | |
| 15 | + | ||
| 16 | + | | Feature | Self-hosted | | |
| 17 | + | | --- | --- | | |
| 18 | + | | Sign up, sign in, email confirmation | Works. Mail goes to the bundled Mailpit inbox. | | |
| 19 | + | | Workspaces, members, access tokens | Works | | |
| 20 | + | | Repositories: create, push and clone over HTTP, browse code, commits | Works | | |
| 21 | + | | Issues, comments, labels | Works | | |
| 22 | + | | Site search | Works | | |
| 23 | + | | Webhooks, integrations | Run, but scheduled retries do not (see below) | | |
| 24 | + | | g1t agents, plans, reviews by agents | Off | | |
| 25 | + | | Context hub search | Off | | |
| 26 | + | | Deployments on `g1t.page` | Off | | |
| 27 | + | | Billing | Off. Nothing is charged, and no usage limit stops work. | | |
| 28 | + | | Git over SSH, the REST API, MCP and the `g1t` CLI | Not available yet | | |
| 29 | + | | Scheduled jobs (webhook retries, Actions schedules) | Not run yet | | |
| 30 | + | ||
| 31 | + | ## Before you start | |
| 32 | + | ||
| 33 | + | - Docker with Compose v2 (`docker compose version`). | |
| 34 | + | - About 4 GB of free disk space for the images. | |
| 35 | + | - Ports 8787 and 8025 free on your machine. | |
| 36 | + | ||
| 37 | + | ## Start g1t | |
| 38 | + | ||
| 39 | + | 1. Get the source: | |
| 40 | + | ||
| 41 | + | ```sh | |
| 42 | + | git clone https://g1t.sh/syntaqx/g1t.git | |
| 43 | + | cd g1t | |
| 44 | + | ``` | |
| 45 | + | ||
| 46 | + | 2. Build and start it. The first build compiles every service and takes a | |
| 47 | + | while: | |
| 48 | + | ||
| 49 | + | ```sh | |
| 50 | + | docker compose -f deploy/self-host/docker-compose.yml up --build -d | |
| 51 | + | ``` | |
| 52 | + | ||
| 53 | + | 3. Open [http://localhost:8787](http://localhost:8787) and create an | |
| 54 | + | account. | |
| 55 | + | 4. Open the Mailpit inbox at [http://localhost:8025](http://localhost:8025) | |
| 56 | + | and follow the link in the confirmation email. | |
| 57 | + | 5. Create a workspace, then a repository. | |
| 58 | + | ||
| 59 | + | ## Push and clone | |
| 60 | + | ||
| 61 | + | The remote is the site's address, then the workspace and repository: | |
| 62 | + | ||
| 63 | + | ```sh | |
| 64 | + | git remote add origin http://localhost:8787/<workspace>/<repo>.git | |
| 65 | + | git push -u origin main | |
| 66 | + | ``` | |
| 67 | + | ||
| 68 | + | Git asks for a username and password: use your g1t username and password, | |
| 69 | + | or an access token, as described in [Git](/guides/git/#authentication). | |
| 70 | + | Public repositories clone without signing in: | |
| 71 | + | ||
| 72 | + | ```sh | |
| 73 | + | git clone http://localhost:8787/<workspace>/<repo>.git | |
| 74 | + | ``` | |
| 75 | + | ||
| 76 | + | ## Check an installation | |
| 77 | + | ||
| 78 | + | `deploy/self-host/smoke.sh` signs up a new account, confirms it through | |
| 79 | + | Mailpit, makes a workspace and a repository, pushes, clones, opens an issue | |
| 80 | + | and reads the code back through the site: | |
| 81 | + | ||
| 82 | + | ```sh | |
| 83 | + | bash deploy/self-host/smoke.sh | |
| 84 | + | ``` | |
| 85 | + | ||
| 86 | + | It prints `All checks passed` when every step worked. | |
| 87 | + | ||
| 88 | + | ## Settings | |
| 89 | + | ||
| 90 | + | Set these in the environment, or in a `.env` file next to | |
| 91 | + | `docker-compose.yml`: | |
| 92 | + | ||
| 93 | + | | Variable | Default | What it does | | |
| 94 | + | | --- | --- | --- | | |
| 95 | + | | `PUBLIC_URL` | `http://localhost:8787` | The address people use. Links in email point here. | | |
| 96 | + | | `G1T_PORT` | `8787` | The port the site is published on | | |
| 97 | + | | `MAILPIT_PORT` | `8025` | The port of the Mailpit inbox | | |
| 98 | + | | `MAIL_FROM` | `g1t <noreply@localhost>` | The sender of g1t's email | | |
| 99 | + | | `MAIL_URL` | `http://mailpit:8025` | The Mailpit server g1t sends mail through | | |
| 100 | + | ||
| 101 | + | To deliver email to real inboxes, have Mailpit relay it through your SMTP | |
| 102 | + | server. The settings are in `docker-compose.yml`, under `mailpit`. | |
| 103 | + | ||
| 104 | + | Sign-in cookies are marked `Secure`. Browsers accept them on | |
| 105 | + | `http://localhost`. On any other address, put g1t behind HTTPS (a reverse | |
| 106 | + | proxy such as Caddy or nginx with a certificate) and set `PUBLIC_URL` to | |
| 107 | + | the `https://` address. | |
| 108 | + | ||
| 109 | + | ## Where your data lives | |
| 110 | + | ||
| 111 | + | | Volume | Holds | | |
| 112 | + | | --- | --- | | |
| 113 | + | | `g1t_g1t-data` | Accounts, workspaces, issues and every other record, as SQLite files; the keys that seal stored secrets (`keys.env`) | | |
| 114 | + | | `g1t_g1t-git` | Your repositories, one bare git repository each | | |
| 115 | + | | `g1t_g1t-secrets` | The key the site and the git store share | | |
| 116 | + | ||
| 117 | + | To back up, stop g1t and copy the volumes: | |
| 118 | + | ||
| 119 | + | ```sh | |
| 120 | + | docker compose -f deploy/self-host/docker-compose.yml stop | |
| 121 | + | docker run --rm -v g1t_g1t-data:/data -v g1t_g1t-git:/git -v "$PWD":/backup \ | |
| 122 | + | debian:bookworm-slim tar czf /backup/g1t-backup.tgz /data /git | |
| 123 | + | docker compose -f deploy/self-host/docker-compose.yml start | |
| 124 | + | ``` | |
| 125 | + | ||
| 126 | + | Keep `keys.env` with the backup. Without it, saved webhook, integration and | |
| 127 | + | Actions secrets cannot be opened. | |
| 128 | + | ||
| 129 | + | ## Upgrade | |
| 130 | + | ||
| 131 | + | Pull the new source and rebuild. Database changes are applied on start, | |
| 132 | + | and changes already applied are skipped: | |
| 133 | + | ||
| 134 | + | ```sh | |
| 135 | + | git pull | |
| 136 | + | docker compose -f deploy/self-host/docker-compose.yml up --build -d | |
| 137 | + | ``` | |
| 138 | + | ||
| 139 | + | ## Stop and remove | |
| 140 | + | ||
| 141 | + | ```sh | |
| 142 | + | docker compose -f deploy/self-host/docker-compose.yml down # keeps your data | |
| 143 | + | docker compose -f deploy/self-host/docker-compose.yml down -v # deletes it | |
| 144 | + | ``` |
| 1 | + | # Wrangler configs written by configs.mjs. | |
| 2 | + | .generated/ |
| 1 | + | # Self-hosted g1t: every Worker of the core forge in one workerd, through | |
| 2 | + | # `wrangler dev`, with D1, KV and Queues kept on the /data volume. | |
| 3 | + | # Build context: the repository root (see docker-compose.yml). | |
| 4 | + | ||
| 5 | + | # ── The Rust services, compiled to WebAssembly as they are for Workers ── | |
| 6 | + | FROM rust:1-slim-bookworm AS rust | |
| 7 | + | RUN apt-get update \ | |
| 8 | + | && apt-get install -y --no-install-recommends curl ca-certificates pkg-config libssl-dev \ | |
| 9 | + | && rm -rf /var/lib/apt/lists/* \ | |
| 10 | + | && rustup target add wasm32-unknown-unknown \ | |
| 11 | + | && cargo install -q worker-build@0.8.7 --locked | |
| 12 | + | WORKDIR /src | |
| 13 | + | COPY Cargo.toml Cargo.lock ./ | |
| 14 | + | COPY apps/api apps/api | |
| 15 | + | COPY crates crates | |
| 16 | + | COPY services services | |
| 17 | + | # The same build hosted g1t deploys (each wrangler.jsonc's build command). | |
| 18 | + | # No cache mount for target/: cargo trusts file times, and a cached build | |
| 19 | + | # from a newer tree would be taken as fresh for an older one. | |
| 20 | + | RUN --mount=type=cache,target=/usr/local/cargo/registry \ | |
| 21 | + | for service in identity repos work events search billing security actions webhooks integrations; do \ | |
| 22 | + | (cd services/$service && worker-build --release) || exit 1; \ | |
| 23 | + | done | |
| 24 | + | ||
| 25 | + | # ── The site, built by React Router; the TypeScript services' packages ── | |
| 26 | + | FROM node:24-bookworm-slim AS node | |
| 27 | + | WORKDIR /app | |
| 28 | + | COPY . . | |
| 29 | + | RUN npm ci --no-audit --no-fund \ | |
| 30 | + | --include-workspace-root \ | |
| 31 | + | -w @g1t/web -w @g1t/projects -w @g1t/deployments -w @g1t/contracts -w @g1t/theme \ | |
| 32 | + | && npm run build -w @g1t/web | |
| 33 | + | ||
| 34 | + | # ── Runtime ── | |
| 35 | + | FROM node:24-bookworm-slim | |
| 36 | + | RUN apt-get update \ | |
| 37 | + | && apt-get install -y --no-install-recommends ca-certificates \ | |
| 38 | + | && rm -rf /var/lib/apt/lists/* | |
| 39 | + | WORKDIR /app | |
| 40 | + | COPY --from=node --chown=node:node /app /app | |
| 41 | + | COPY --from=rust /src/services /tmp/rust-services | |
| 42 | + | RUN for service in identity repos work events search billing security actions webhooks integrations; do \ | |
| 43 | + | cp -r /tmp/rust-services/$service/build services/$service/build; \ | |
| 44 | + | done \ | |
| 45 | + | && rm -rf /tmp/rust-services \ | |
| 46 | + | && mkdir -p /data && chown node:node /data | |
| 47 | + | ENV WRANGLER_SEND_METRICS=false \ | |
| 48 | + | PUBLIC_URL=http://localhost:8787 \ | |
| 49 | + | GITSTORE_URL=http://gitstore:8080 \ | |
| 50 | + | G1T_DATA=/data | |
| 51 | + | VOLUME /data | |
| 52 | + | EXPOSE 8787 | |
| 53 | + | USER node | |
| 54 | + | CMD ["bash", "deploy/self-host/start.sh"] |
| 1 | + | # For deploy/self-host/Dockerfile only (BuildKit reads <Dockerfile>.dockerignore). | |
| 2 | + | # Unlike the root .dockerignore, the site and the packages are needed here. | |
| 3 | + | .git | |
| 4 | + | .credentials | |
| 5 | + | .g1t | |
| 6 | + | .claude | |
| 7 | + | .vscode | |
| 8 | + | .env* | |
| 9 | + | **/.dev.vars* | |
| 10 | + | **/node_modules | |
| 11 | + | **/target | |
| 12 | + | **/build | |
| 13 | + | **/dist | |
| 14 | + | **/.wrangler | |
| 15 | + | **/.astro | |
| 16 | + | deploy/self-host/.generated | |
| 17 | + | docs | |
| 18 | + | apps/docs | |
| 19 | + | apps/sudo |
| 1 | + | #!/usr/bin/env node | |
| 2 | + | // Writes the Wrangler configs a self-hosted g1t runs with, derived from the | |
| 3 | + | // hosted ones, so the two never drift apart. | |
| 4 | + | // | |
| 5 | + | // Each hosted service's wrangler.jsonc is read and changed only where | |
| 6 | + | // Cloudflare-only things live: | |
| 7 | + | // | |
| 8 | + | // - account, routes, placement, observability and builds are dropped; | |
| 9 | + | // - ARTIFACTS (git storage) becomes a service binding to workers/artifacts, | |
| 10 | + | // which keeps repositories in the git store (gitstore/server.mjs); | |
| 11 | + | // - EMAIL (Email Sending) becomes a service binding to workers/mail; | |
| 12 | + | // - services that are off in this phase (agents, the context hub, the | |
| 13 | + | // g1t.page dispatcher, model proxy) are bound to workers/off instead, and | |
| 14 | + | // events stop queueing work for them; | |
| 15 | + | // - URLs that name g1t.sh name PUBLIC_URL instead, and billing is free. | |
| 16 | + | // | |
| 17 | + | // Usage: node configs.mjs [outDir] | |
| 18 | + | // Environment: PUBLIC_URL, GITSTORE_URL, GITSTORE_SECRET, MAIL_URL, | |
| 19 | + | // ACTIONS_KEY, INTEGRATIONS_KEY, WEBHOOKS_KEY. | |
| 20 | + | // | |
| 21 | + | // The output is for `wrangler dev` (see start.sh): every Worker in one | |
| 22 | + | // workerd, the site first, with D1, KV and Queues kept on disk. | |
| 23 | + | ||
| 24 | + | import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; | |
| 25 | + | import { dirname, join, relative, resolve } from "node:path"; | |
| 26 | + | import { fileURLToPath } from "node:url"; | |
| 27 | + | ||
| 28 | + | const here = dirname(fileURLToPath(import.meta.url)); | |
| 29 | + | const root = resolve(here, "../.."); | |
| 30 | + | const out = resolve(process.argv[2] ?? join(here, ".generated")); | |
| 31 | + | mkdirSync(out, { recursive: true }); | |
| 32 | + | ||
| 33 | + | const PUBLIC_URL = (process.env.PUBLIC_URL ?? "http://localhost:8787").replace(/\/$/, ""); | |
| 34 | + | ||
| 35 | + | /** Services that run, in the order Wrangler is given them (the site first). */ | |
| 36 | + | export const RUNNING = [ | |
| 37 | + | { name: "g1t", dir: "apps/web", web: true }, | |
| 38 | + | { name: "g1t-identity", dir: "services/identity" }, | |
| 39 | + | { name: "g1t-repos", dir: "services/repos" }, | |
| 40 | + | { name: "g1t-work", dir: "services/work" }, | |
| 41 | + | { name: "g1t-events", dir: "services/events" }, | |
| 42 | + | { name: "g1t-projects", dir: "services/projects" }, | |
| 43 | + | { name: "g1t-search", dir: "services/search" }, | |
| 44 | + | { name: "g1t-billing", dir: "services/billing" }, | |
| 45 | + | { name: "g1t-security", dir: "services/security" }, | |
| 46 | + | { name: "g1t-actions", dir: "services/actions" }, | |
| 47 | + | { name: "g1t-webhooks", dir: "services/webhooks" }, | |
| 48 | + | { name: "g1t-integrations", dir: "services/integrations" }, | |
| 49 | + | { name: "g1t-deployments", dir: "services/deployments" }, | |
| 50 | + | ]; | |
| 51 | + | ||
| 52 | + | /** Services that are off in phase 1, and what the off Worker calls them. */ | |
| 53 | + | const OFF = { | |
| 54 | + | "g1t-runner": "Agents", | |
| 55 | + | "g1t-context": "Context search and memory", | |
| 56 | + | }; | |
| 57 | + | ||
| 58 | + | /** Sealing keys, by the service that holds each (hosted: Wrangler secrets). */ | |
| 59 | + | const SECRETS = { | |
| 60 | + | "g1t-actions": "ACTIONS_KEY", | |
| 61 | + | "g1t-integrations": "INTEGRATIONS_KEY", | |
| 62 | + | "g1t-webhooks": "WEBHOOKS_KEY", | |
| 63 | + | }; | |
| 64 | + | ||
| 65 | + | /** Queues whose consumers are off: events stops sending to them. */ | |
| 66 | + | const OFF_QUEUES = new Set(["g1t-events-runner", "g1t-events-context"]); | |
| 67 | + | ||
| 68 | + | /** Strips comments and trailing commas from JSONC. Strings are respected. */ | |
| 69 | + | function parseJsonc(text) { | |
| 70 | + | let result = ""; | |
| 71 | + | let inString = false; | |
| 72 | + | for (let i = 0; i < text.length; i++) { | |
| 73 | + | const char = text[i]; | |
| 74 | + | if (inString) { | |
| 75 | + | result += char; | |
| 76 | + | if (char === "\\") result += text[++i]; | |
| 77 | + | else if (char === '"') inString = false; | |
| 78 | + | } else if (char === '"') { | |
| 79 | + | inString = true; | |
| 80 | + | result += char; | |
| 81 | + | } else if (char === "/" && text[i + 1] === "/") { | |
| 82 | + | while (i < text.length && text[i] !== "\n") i++; | |
| 83 | + | result += "\n"; | |
| 84 | + | } else if (char === "/" && text[i + 1] === "*") { | |
| 85 | + | i = text.indexOf("*/", i + 2) + 1; | |
| 86 | + | } else { | |
| 87 | + | result += char; | |
| 88 | + | } | |
| 89 | + | } | |
| 90 | + | return JSON.parse(result.replace(/,(\s*[}\]])/g, "$1")); | |
| 91 | + | } | |
| 92 | + | ||
| 93 | + | const rel = (path) => relative(out, resolve(root, path)).replaceAll("\\", "/"); | |
| 94 | + | ||
| 95 | + | function hostedUrl(value) { | |
| 96 | + | return typeof value === "string" ? value.replace(/https:\/\/(api\.)?g1t\.sh/g, PUBLIC_URL) : value; | |
| 97 | + | } | |
| 98 | + | ||
| 99 | + | function selfHosted(service) { | |
| 100 | + | const hosted = parseJsonc(readFileSync(join(root, service.dir, "wrangler.jsonc"), "utf8")); | |
| 101 | + | const config = { | |
| 102 | + | name: hosted.name, | |
| 103 | + | compatibility_date: hosted.compatibility_date, | |
| 104 | + | compatibility_flags: hosted.compatibility_flags, | |
| 105 | + | rules: hosted.rules, | |
| 106 | + | vars: {}, | |
| 107 | + | }; | |
| 108 | + | ||
| 109 | + | if (service.web) { | |
| 110 | + | // The site as React Router built it (apps/web/build), not its sources. | |
| 111 | + | config.main = rel(`${service.dir}/build/server/index.js`); | |
| 112 | + | config.no_bundle = true; | |
| 113 | + | config.rules = [{ type: "ESModule", globs: ["**/*.js", "**/*.mjs"] }]; | |
| 114 | + | config.assets = { directory: rel(`${service.dir}/build/client`) }; | |
| 115 | + | } else { | |
| 116 | + | config.main = rel(join(service.dir, hosted.main)); | |
| 117 | + | } | |
| 118 | + | ||
| 119 | + | for (const [key, value] of Object.entries(hosted.vars ?? {})) config.vars[key] = hostedUrl(value); | |
| 120 | + | ||
| 121 | + | if (hosted.d1_databases) { | |
| 122 | + | config.d1_databases = hosted.d1_databases.map((db) => ({ | |
| 123 | + | binding: db.binding, | |
| 124 | + | database_name: db.database_name, | |
| 125 | + | database_id: db.database_id, | |
| 126 | + | migrations_dir: rel(join(service.dir, db.migrations_dir ?? "migrations")), | |
| 127 | + | })); | |
| 128 | + | } | |
| 129 | + | if (hosted.kv_namespaces) config.kv_namespaces = hosted.kv_namespaces.map(({ binding, id }) => ({ binding, id })); | |
| 130 | + | if (hosted.triggers) config.triggers = hosted.triggers; | |
| 131 | + | ||
| 132 | + | if (hosted.queues) { | |
| 133 | + | config.queues = {}; | |
| 134 | + | if (hosted.queues.producers) { | |
| 135 | + | config.queues.producers = hosted.queues.producers.filter((producer) => !OFF_QUEUES.has(producer.queue)); | |
| 136 | + | } | |
| 137 | + | if (hosted.queues.consumers) config.queues.consumers = hosted.queues.consumers; | |
| 138 | + | } | |
| 139 | + | ||
| 140 | + | config.services = (hosted.services ?? []).map((binding) => | |
| 141 | + | OFF[binding.service] ? { binding: binding.binding, service: offName(binding.service) } : binding, | |
| 142 | + | ); | |
| 143 | + | ||
| 144 | + | // Cloudflare-only bindings, and what stands in for them. | |
| 145 | + | if (hosted.artifacts) { | |
| 146 | + | for (const artifacts of hosted.artifacts) { | |
| 147 | + | config.services.push({ binding: artifacts.binding, service: "g1t-artifacts" }); | |
| 148 | + | } | |
| 149 | + | } | |
| 150 | + | if (hosted.send_email) { | |
| 151 | + | for (const email of hosted.send_email) config.services.push({ binding: email.name, service: "g1t-mail" }); | |
| 152 | + | } | |
| 153 | + | ||
| 154 | + | // Secrets the hosted services hold, given here from the environment, each | |
| 155 | + | // only to the service that uses it. | |
| 156 | + | const secret = SECRETS[hosted.name]; | |
| 157 | + | if (secret && process.env[secret]) config.vars[secret] = process.env[secret]; | |
| 158 | + | ||
| 159 | + | // Self-hosted g1t charges nothing: billing records usage at cost and never | |
| 160 | + | // stops work for it. | |
| 161 | + | if (hosted.name === "g1t-billing") config.vars.FREE_WHILE_BUILDING = "true"; | |
| 162 | + | // Nothing to deploy to: deployments are off (no Cloudflare API token). | |
| 163 | + | if (hosted.name === "g1t-deployments") delete config.vars.CUSTOM_HOSTNAMES_ZONE_ID; | |
| 164 | + | ||
| 165 | + | return config; | |
| 166 | + | } | |
| 167 | + | ||
| 168 | + | function offName(service) { | |
| 169 | + | return `${service}-off`; | |
| 170 | + | } | |
| 171 | + | ||
| 172 | + | function write(name, config) { | |
| 173 | + | const path = join(out, `${name}.json`); | |
| 174 | + | writeFileSync(path, `${JSON.stringify(config, null, 2)}\n`); | |
| 175 | + | return path; | |
| 176 | + | } | |
| 177 | + | ||
| 178 | + | const files = []; | |
| 179 | + | for (const service of RUNNING) files.push(write(service.name, selfHosted(service))); | |
| 180 | + | ||
| 181 | + | const compatibility_date = "2026-09-26"; | |
| 182 | + | files.push( | |
| 183 | + | write("g1t-artifacts", { | |
| 184 | + | name: "g1t-artifacts", | |
| 185 | + | main: rel("deploy/self-host/workers/artifacts/index.js"), | |
| 186 | + | compatibility_date, | |
| 187 | + | vars: { | |
| 188 | + | GITSTORE_URL: process.env.GITSTORE_URL ?? "http://gitstore:8080", | |
| 189 | + | GITSTORE_SECRET: process.env.GITSTORE_SECRET ?? "", | |
| 190 | + | }, | |
| 191 | + | }), | |
| 192 | + | ); | |
| 193 | + | files.push( | |
| 194 | + | write("g1t-mail", { | |
| 195 | + | name: "g1t-mail", | |
| 196 | + | main: rel("deploy/self-host/workers/mail/index.js"), | |
| 197 | + | compatibility_date, | |
| 198 | + | vars: { | |
| 199 | + | PUBLIC_URL, | |
| 200 | + | MAIL_URL: process.env.MAIL_URL ?? "", | |
| 201 | + | MAIL_FROM: process.env.MAIL_FROM ?? "", | |
| 202 | + | }, | |
| 203 | + | }), | |
| 204 | + | ); | |
| 205 | + | for (const [service, feature] of Object.entries(OFF)) { | |
| 206 | + | files.push( | |
| 207 | + | write(offName(service), { | |
| 208 | + | name: offName(service), | |
| 209 | + | main: rel("deploy/self-host/workers/off/index.js"), | |
| 210 | + | compatibility_date, | |
| 211 | + | vars: { OFF_NAME: feature }, | |
| 212 | + | }), | |
| 213 | + | ); | |
| 214 | + | } | |
| 215 | + | ||
| 216 | + | // The order Wrangler takes them in: the site first, as the one that serves. | |
| 217 | + | writeFileSync(join(out, "workers.txt"), `${files.map((file) => relative(out, file)).join("\n")}\n`); | |
| 218 | + | console.log(`Wrote ${files.length} configs to ${out}`); |
| 1 | + | # Self-hosted g1t, phase 1: the core forge on your own machine. | |
| 2 | + | # | |
| 3 | + | # docker compose -f deploy/self-host/docker-compose.yml up --build | |
| 4 | + | # | |
| 5 | + | # Then open http://localhost:8787. Mail (the confirmation link at sign-up) | |
| 6 | + | # lands in Mailpit at http://localhost:8025. | |
| 7 | + | # | |
| 8 | + | # What runs: the site and every core service in one workerd (g1t), git | |
| 9 | + | # repositories as bare repos on a volume (gitstore), and Mailpit for mail. | |
| 10 | + | # Agents, deployments, context search and billing are off. See | |
| 11 | + | # docs/SELF_HOSTING.md. | |
| 12 | + | name: g1t | |
| 13 | + | ||
| 14 | + | services: | |
| 15 | + | g1t: | |
| 16 | + | build: | |
| 17 | + | context: ../.. | |
| 18 | + | dockerfile: deploy/self-host/Dockerfile | |
| 19 | + | ports: | |
| 20 | + | - "${G1T_PORT:-8787}:8787" | |
| 21 | + | environment: | |
| 22 | + | # Where people reach this installation. Links in mail point here. | |
| 23 | + | PUBLIC_URL: ${PUBLIC_URL:-http://localhost:8787} | |
| 24 | + | GITSTORE_URL: http://gitstore:8080 | |
| 25 | + | GITSTORE_SECRET_FILE: /secrets/gitstore | |
| 26 | + | MAIL_URL: ${MAIL_URL:-http://mailpit:8025} | |
| 27 | + | MAIL_FROM: ${MAIL_FROM:-g1t <noreply@localhost>} | |
| 28 | + | volumes: | |
| 29 | + | - g1t-data:/data | |
| 30 | + | - g1t-secrets:/secrets:ro | |
| 31 | + | depends_on: | |
| 32 | + | gitstore: | |
| 33 | + | condition: service_healthy | |
| 34 | + | mailpit: | |
| 35 | + | condition: service_started | |
| 36 | + | restart: unless-stopped | |
| 37 | + | ||
| 38 | + | gitstore: | |
| 39 | + | build: | |
| 40 | + | context: ./gitstore | |
| 41 | + | environment: | |
| 42 | + | GITSTORE_URL: http://gitstore:8080 | |
| 43 | + | GITSTORE_SECRET_FILE: /secrets/gitstore | |
| 44 | + | volumes: | |
| 45 | + | - g1t-git:/data/git | |
| 46 | + | - g1t-secrets:/secrets | |
| 47 | + | # Not published: only the g1t container reaches it. | |
| 48 | + | restart: unless-stopped | |
| 49 | + | ||
| 50 | + | mailpit: | |
| 51 | + | image: axllent/mailpit:latest | |
| 52 | + | ports: | |
| 53 | + | - "${MAILPIT_PORT:-8025}:8025" | |
| 54 | + | # To deliver for real, relay through your SMTP server: | |
| 55 | + | # environment: | |
| 56 | + | # MP_SMTP_RELAY_HOST: smtp.example.com | |
| 57 | + | # MP_SMTP_RELAY_PORT: "587" | |
| 58 | + | # MP_SMTP_RELAY_USERNAME: ... | |
| 59 | + | # MP_SMTP_RELAY_PASSWORD: ... | |
| 60 | + | # MP_SMTP_RELAY_ALL: "true" | |
| 61 | + | restart: unless-stopped | |
| 62 | + | ||
| 63 | + | volumes: | |
| 64 | + | g1t-data: | |
| 65 | + | g1t-git: | |
| 66 | + | g1t-secrets: |
| 1 | + | # g1t's git store for self-hosting: bare repositories on a volume, served | |
| 2 | + | # by git itself. See server.mjs. | |
| 3 | + | FROM node:24-bookworm-slim | |
| 4 | + | RUN apt-get update \ | |
| 5 | + | && apt-get install -y --no-install-recommends git ca-certificates \ | |
| 6 | + | && rm -rf /var/lib/apt/lists/* | |
| 7 | + | WORKDIR /app | |
| 8 | + | COPY server.mjs ./ | |
| 9 | + | ENV GITSTORE_ROOT=/data/git GITSTORE_PORT=8080 | |
| 10 | + | # Owned by the user it runs as, so new volumes are writable. | |
| 11 | + | RUN mkdir -p /data/git /secrets && chown node:node /data/git /secrets | |
| 12 | + | VOLUME /data/git | |
| 13 | + | EXPOSE 8080 | |
| 14 | + | USER node | |
| 15 | + | HEALTHCHECK --interval=5s --timeout=3s --retries=20 \ | |
| 16 | + | CMD node -e "fetch('http://127.0.0.1:8080/healthz').then(r=>process.exit(r.ok?0:1),()=>process.exit(1))" | |
| 17 | + | CMD ["node", "server.mjs"] |
| 1 | + | // g1t's git store for self-hosting: plain bare repositories on disk. | |
| 2 | + | // | |
| 3 | + | // Hosted g1t keeps repositories in Cloudflare Artifacts. This server does | |
| 4 | + | // the same job with nothing but git: one bare repository per store key | |
| 5 | + | // under GITSTORE_ROOT, git's own smart HTTP (git http-backend) for clones, | |
| 6 | + | // fetches and pushes, and a small JSON API for the reads the repos service | |
| 7 | + | // makes (commits, trees, blobs, files) and for creating and forking. | |
| 8 | + | // | |
| 9 | + | // It is reached only by the Artifacts-compatible shim (workers/artifacts), | |
| 10 | + | // which the repos service is bound to in place of the Artifacts binding, and | |
| 11 | + | // by the repos service itself for git's smart HTTP. Nothing else should be | |
| 12 | + | // able to reach it: the API takes a shared secret, and git requests a | |
| 13 | + | // short-lived token the shim minted with the same secret. | |
| 14 | + | // | |
| 15 | + | // No dependencies beyond Node and git. | |
| 16 | + | ||
| 17 | + | import { spawn } from "node:child_process"; | |
| 18 | + | import { createHmac, randomBytes, randomUUID, timingSafeEqual } from "node:crypto"; | |
| 19 | + | import { existsSync, mkdirSync, readFileSync, statSync, utimesSync, writeFileSync } from "node:fs"; | |
| 20 | + | import { createServer } from "node:http"; | |
| 21 | + | import { dirname, join } from "node:path"; | |
| 22 | + | ||
| 23 | + | const ROOT = process.env.GITSTORE_ROOT ?? "/data/git"; | |
| 24 | + | const PORT = Number(process.env.GITSTORE_PORT ?? 8080); | |
| 25 | + | const SECRET = loadSecret(); | |
| 26 | + | // How the repos service reaches this server; it becomes each repository's | |
| 27 | + | // `remote`, exactly as Artifacts hands one out. | |
| 28 | + | const PUBLIC_URL = (process.env.GITSTORE_URL ?? `http://localhost:${PORT}`).replace(/\/$/, ""); | |
| 29 | + | ||
| 30 | + | /** | |
| 31 | + | * The secret shared with the Artifacts shim: GITSTORE_SECRET, or else the | |
| 32 | + | * one in GITSTORE_SECRET_FILE, made on first start. The compose file shares | |
| 33 | + | * that file with the g1t container, so nobody has to choose one. | |
| 34 | + | */ | |
| 35 | + | function loadSecret() { | |
| 36 | + | if (process.env.GITSTORE_SECRET) return process.env.GITSTORE_SECRET; | |
| 37 | + | const file = process.env.GITSTORE_SECRET_FILE; | |
| 38 | + | if (!file) return ""; | |
| 39 | + | if (!existsSync(file)) { | |
| 40 | + | mkdirSync(dirname(file), { recursive: true }); | |
| 41 | + | writeFileSync(file, randomBytes(32).toString("hex"), { mode: 0o600 }); | |
| 42 | + | } | |
| 43 | + | return readFileSync(file, "utf8").trim(); | |
| 44 | + | } | |
| 45 | + | ||
| 46 | + | if (SECRET.length < 16) { | |
| 47 | + | console.error("Set GITSTORE_SECRET (16 characters or more) or GITSTORE_SECRET_FILE."); | |
| 48 | + | process.exit(1); | |
| 49 | + | } | |
| 50 | + | mkdirSync(ROOT, { recursive: true }); | |
| 51 | + | ||
| 52 | + | const KEY = /^[A-Za-z0-9_][A-Za-z0-9._-]{0,199}$/; | |
| 53 | + | const HASH = /^[0-9a-f]{40}$/; | |
| 54 | + | ||
| 55 | + | class StoreError extends Error { | |
| 56 | + | constructor(code, message, status = 400) { | |
| 57 | + | super(message); | |
| 58 | + | this.code = code; | |
| 59 | + | this.status = status; | |
| 60 | + | } | |
| 61 | + | } | |
| 62 | + | ||
| 63 | + | function repoDir(key) { | |
| 64 | + | if (!KEY.test(key) || key.includes("..")) { | |
| 65 | + | throw new StoreError("INVALID_REPO_NAME", `invalid repository name: ${key}`); | |
| 66 | + | } | |
| 67 | + | return join(ROOT, `${key}.git`); | |
| 68 | + | } | |
| 69 | + | ||
| 70 | + | function exists(key) { | |
| 71 | + | return existsSync(join(repoDir(key), "HEAD")); | |
| 72 | + | } | |
| 73 | + | ||
| 74 | + | function requireRepo(key) { | |
| 75 | + | if (!exists(key)) throw new StoreError("NOT_FOUND", `no repository ${key}`, 404); | |
| 76 | + | return repoDir(key); | |
| 77 | + | } | |
| 78 | + | ||
| 79 | + | /** Runs git and resolves with its stdout as a Buffer. */ | |
| 80 | + | function git(args, { cwd, input, allowFail = false } = {}) { | |
| 81 | + | return new Promise((resolve, reject) => { | |
| 82 | + | const child = spawn("git", args, { cwd, stdio: ["pipe", "pipe", "pipe"] }); | |
| 83 | + | const out = []; | |
| 84 | + | const err = []; | |
| 85 | + | child.stdout.on("data", (chunk) => out.push(chunk)); | |
| 86 | + | child.stderr.on("data", (chunk) => err.push(chunk)); | |
| 87 | + | child.on("error", reject); | |
| 88 | + | child.on("close", (code) => { | |
| 89 | + | if (code !== 0 && !allowFail) { | |
| 90 | + | reject(new StoreError("INTERNAL_ERROR", `git ${args[0]} failed: ${Buffer.concat(err)}`, 500)); | |
| 91 | + | } else { | |
| 92 | + | resolve({ code, stdout: Buffer.concat(out) }); | |
| 93 | + | } | |
| 94 | + | }); | |
| 95 | + | child.stdin.end(input ?? undefined); | |
| 96 | + | }); | |
| 97 | + | } | |
| 98 | + | ||
| 99 | + | // ── Metadata kept beside each repository ──────────────────────────────── | |
| 100 | + | ||
| 101 | + | function metaPath(key) { | |
| 102 | + | return join(repoDir(key), "g1t.json"); | |
| 103 | + | } | |
| 104 | + | ||
| 105 | + | function readMeta(key) { | |
| 106 | + | try { | |
| 107 | + | return JSON.parse(readFileSync(metaPath(key), "utf8")); | |
| 108 | + | } catch { | |
| 109 | + | return {}; | |
| 110 | + | } | |
| 111 | + | } | |
| 112 | + | ||
| 113 | + | function writeMeta(key, meta) { | |
| 114 | + | writeFileSync(metaPath(key), JSON.stringify(meta, null, 2)); | |
| 115 | + | } | |
| 116 | + | ||
| 117 | + | async function info(key) { | |
| 118 | + | const dir = requireRepo(key); | |
| 119 | + | const meta = readMeta(key); | |
| 120 | + | const head = (await git(["symbolic-ref", "--short", "HEAD"], { cwd: dir, allowFail: true })).stdout | |
| 121 | + | .toString() | |
| 122 | + | .trim(); | |
| 123 | + | let lastPushAt = null; | |
| 124 | + | try { | |
| 125 | + | lastPushAt = statSync(join(dir, "g1t-pushed")).mtime.toISOString(); | |
| 126 | + | } catch {} | |
| 127 | + | return { | |
| 128 | + | id: meta.id ?? key, | |
| 129 | + | name: key, | |
| 130 | + | description: meta.description ?? null, | |
| 131 | + | defaultBranch: head || "main", | |
| 132 | + | createdAt: meta.createdAt ?? new Date(0).toISOString(), | |
| 133 | + | updatedAt: lastPushAt ?? meta.createdAt ?? new Date(0).toISOString(), | |
| 134 | + | lastPushAt, | |
| 135 | + | source: meta.source ?? null, | |
| 136 | + | readOnly: Boolean(meta.readOnly), | |
| 137 | + | remote: `${PUBLIC_URL}/git/${key}.git`, | |
| 138 | + | }; | |
| 139 | + | } | |
| 140 | + | ||
| 141 | + | async function create(key, { description, defaultBranch, readOnly, source } = {}) { | |
| 142 | + | const dir = repoDir(key); | |
| 143 | + | if (exists(key)) throw new StoreError("ALREADY_EXISTS", `${key} already exists`, 409); | |
| 144 | + | mkdirSync(dir, { recursive: true }); | |
| 145 | + | await git(["init", "--bare", "--quiet", `--initial-branch=${defaultBranch || "main"}`, dir]); | |
| 146 | + | await configure(dir); | |
| 147 | + | writeMeta(key, { | |
| 148 | + | id: randomUUID(), | |
| 149 | + | description: description ?? null, | |
| 150 | + | createdAt: new Date().toISOString(), | |
| 151 | + | readOnly: Boolean(readOnly), | |
| 152 | + | source: source ?? null, | |
| 153 | + | }); | |
| 154 | + | return info(key); | |
| 155 | + | } | |
| 156 | + | ||
| 157 | + | async function configure(dir) { | |
| 158 | + | // Pushes arrive through git http-backend; the token has already been | |
| 159 | + | // checked, so receive-pack is allowed for every write-scoped request. | |
| 160 | + | await git(["config", "http.receivepack", "true"], { cwd: dir }); | |
| 161 | + | await git(["config", "receive.denyNonFastForwards", "false"], { cwd: dir }); | |
| 162 | + | await git(["config", "uploadpack.allowAnySHA1InWant", "true"], { cwd: dir }); | |
| 163 | + | } | |
| 164 | + | ||
| 165 | + | async function fork(key, target, { description, readOnly, defaultBranchOnly = true } = {}) { | |
| 166 | + | const source = requireRepo(key); | |
| 167 | + | const dir = repoDir(target); | |
| 168 | + | if (exists(target)) throw new StoreError("ALREADY_EXISTS", `${target} already exists`, 409); | |
| 169 | + | const args = ["clone", "--bare", "--quiet", "--no-tags"]; | |
| 170 | + | if (defaultBranchOnly) args.push("--single-branch"); | |
| 171 | + | // A local clone hard-links the objects: cheap, and independent of the | |
| 172 | + | // source from then on. | |
| 173 | + | args.push(source, dir); | |
| 174 | + | await git(args); | |
| 175 | + | await git(["remote", "remove", "origin"], { cwd: dir, allowFail: true }); | |
| 176 | + | await configure(dir); | |
| 177 | + | writeMeta(target, { | |
| 178 | + | id: randomUUID(), | |
| 179 | + | description: description ?? readMeta(key).description ?? null, | |
| 180 | + | createdAt: new Date().toISOString(), | |
| 181 | + | readOnly: Boolean(readOnly), | |
| 182 | + | source: `artifacts:${key}`, | |
| 183 | + | }); | |
| 184 | + | return info(target); | |
| 185 | + | } | |
| 186 | + | ||
| 187 | + | // ── Reading objects ───────────────────────────────────────────────────── | |
| 188 | + | ||
| 189 | + | async function objectType(dir, spec) { | |
| 190 | + | const { code, stdout } = await git(["cat-file", "-t", "--", spec], { cwd: dir, allowFail: true }); | |
| 191 | + | return code === 0 ? stdout.toString().trim() : null; | |
| 192 | + | } | |
| 193 | + | ||
| 194 | + | function person(line) { | |
| 195 | + | // `Name <email> 1700000000 +0000` | |
| 196 | + | const match = /^(.*) <([^>]*)> (\d+) [+-]\d{4}$/.exec(line); | |
| 197 | + | return match ? { name: match[1], email: match[2], at: Number(match[3]) } : { name: line, email: "", at: 0 }; | |
| 198 | + | } | |
| 199 | + | ||
| 200 | + | function parseCommit(hash, raw) { | |
| 201 | + | const text = raw.toString("utf8"); | |
| 202 | + | const split = text.indexOf("\n\n"); | |
| 203 | + | const headers = (split === -1 ? text : text.slice(0, split)).split("\n"); | |
| 204 | + | let message = split === -1 ? "" : text.slice(split + 2); | |
| 205 | + | if (message.endsWith("\n")) message = message.slice(0, -1); | |
| 206 | + | const commit = { hash, treeHash: "", message, parents: [], author: null, committer: null }; | |
| 207 | + | for (const header of headers) { | |
| 208 | + | const space = header.indexOf(" "); | |
| 209 | + | const name = header.slice(0, space); | |
| 210 | + | const value = header.slice(space + 1); | |
| 211 | + | if (name === "tree") commit.treeHash = value; | |
| 212 | + | else if (name === "parent") commit.parents.push(value); | |
| 213 | + | else if (name === "author") commit.author = person(value); | |
| 214 | + | else if (name === "committer") commit.committer = person(value); | |
| 215 | + | } | |
| 216 | + | const author = commit.author ?? { name: "", email: "", at: 0 }; | |
| 217 | + | const committer = commit.committer ?? author; | |
| 218 | + | return { | |
| 219 | + | hash, | |
| 220 | + | treeHash: commit.treeHash, | |
| 221 | + | message: commit.message, | |
| 222 | + | author: { name: author.name, email: author.email }, | |
| 223 | + | committer: { name: committer.name, email: committer.email }, | |
| 224 | + | parents: commit.parents, | |
| 225 | + | authoredAt: author.at, | |
| 226 | + | committedAt: committer.at, | |
| 227 | + | }; | |
| 228 | + | } | |
| 229 | + | ||
| 230 | + | async function readCommit(key, hash) { | |
| 231 | + | const dir = requireRepo(key); | |
| 232 | + | if (!HASH.test(hash)) return null; | |
| 233 | + | if ((await objectType(dir, hash)) !== "commit") return null; | |
| 234 | + | return parseCommit(hash, (await git(["cat-file", "commit", hash], { cwd: dir })).stdout); | |
| 235 | + | } | |
| 236 | + | ||
| 237 | + | async function log(key, { ref = "HEAD", limit = 50, offset = 0 } = {}) { | |
| 238 | + | const dir = requireRepo(key); | |
| 239 | + | if (typeof ref !== "string" || ref.startsWith("-")) return []; | |
| 240 | + | const count = Math.max(1, Math.min(Number(limit) || 50, 1000)); | |
| 241 | + | const skip = Math.max(0, Number(offset) || 0); | |
| 242 | + | const listed = await git( | |
| 243 | + | ["rev-list", "--first-parent", `--max-count=${count}`, `--skip=${skip}`, ref, "--"], | |
| 244 | + | { cwd: dir, allowFail: true }, | |
| 245 | + | ); | |
| 246 | + | if (listed.code !== 0) return []; | |
| 247 | + | const hashes = listed.stdout.toString().split("\n").filter(Boolean); | |
| 248 | + | const commits = []; | |
| 249 | + | for (const hash of hashes) { | |
| 250 | + | commits.push(parseCommit(hash, (await git(["cat-file", "commit", hash], { cwd: dir })).stdout)); | |
| 251 | + | } | |
| 252 | + | return commits; | |
| 253 | + | } | |
| 254 | + | ||
| 255 | + | const TYPES = { "040000": "tree", "100644": "blob", "100755": "exec", "120000": "symlink", "160000": "gitlink" }; | |
| 256 | + | ||
| 257 | + | async function readTree(key, hash) { | |
| 258 | + | const dir = requireRepo(key); | |
| 259 | + | if (!HASH.test(hash)) return null; | |
| 260 | + | if ((await objectType(dir, hash)) !== "tree") return null; | |
| 261 | + | const { stdout } = await git(["ls-tree", "-z", hash], { cwd: dir }); | |
| 262 | + | return stdout | |
| 263 | + | .toString("utf8") | |
| 264 | + | .split("\0") | |
| 265 | + | .filter(Boolean) | |
| 266 | + | .map((line) => { | |
| 267 | + | const tab = line.indexOf("\t"); | |
| 268 | + | const [mode, , object] = line.slice(0, tab).split(" "); | |
| 269 | + | return { | |
| 270 | + | name: line.slice(tab + 1), | |
| 271 | + | mode: mode === "040000" ? "40000" : mode, | |
| 272 | + | hash: object, | |
| 273 | + | type: TYPES[mode] ?? "blob", | |
| 274 | + | }; | |
| 275 | + | }); | |
| 276 | + | } | |
| 277 | + | ||
| 278 | + | async function readBlob(key, hash) { | |
| 279 | + | const dir = requireRepo(key); | |
| 280 | + | if (!HASH.test(hash)) return null; | |
| 281 | + | if ((await objectType(dir, hash)) !== "blob") return null; | |
| 282 | + | return (await git(["cat-file", "blob", hash], { cwd: dir })).stdout; | |
| 283 | + | } | |
| 284 | + | ||
| 285 | + | async function readFile(key, ref, path) { | |
| 286 | + | const dir = requireRepo(key); | |
| 287 | + | if (!ref || !path || ref.startsWith("-") || ref.includes(":")) return null; | |
| 288 | + | const spec = `${ref}:${path.replace(/^\/+/, "")}`; | |
| 289 | + | if ((await objectType(dir, spec)) !== "blob") return null; | |
| 290 | + | return (await git(["cat-file", "blob", spec], { cwd: dir })).stdout; | |
| 291 | + | } | |
| 292 | + | ||
| 293 | + | // ── Tokens for git's smart HTTP ───────────────────────────────────────── | |
| 294 | + | ||
| 295 | + | function sign(payload) { | |
| 296 | + | return createHmac("sha256", SECRET).update(payload).digest("base64url"); | |
| 297 | + | } | |
| 298 | + | ||
| 299 | + | function mintToken(key, scope = "write", ttl = 86400) { | |
| 300 | + | const seconds = Math.max(60, Math.min(Number(ttl) || 86400, 31536000)); | |
| 301 | + | const expires = Math.floor(Date.now() / 1000) + seconds; | |
| 302 | + | const id = randomUUID(); | |
| 303 | + | const payload = Buffer.from(JSON.stringify({ k: key, s: scope, e: expires, i: id })).toString("base64url"); | |
| 304 | + | return { | |
| 305 | + | id, | |
| 306 | + | plaintext: `${payload}.${sign(payload)}`, | |
| 307 | + | scope, | |
| 308 | + | expiresAt: new Date(expires * 1000).toISOString(), | |
| 309 | + | }; | |
| 310 | + | } | |
| 311 | + | ||
| 312 | + | function checkToken(token, key) { | |
| 313 | + | const [payload, signature] = String(token ?? "").split("."); | |
| 314 | + | if (!payload || !signature) return null; | |
| 315 | + | const expected = Buffer.from(sign(payload)); | |
| 316 | + | const given = Buffer.from(signature); | |
| 317 | + | if (expected.length !== given.length || !timingSafeEqual(expected, given)) return null; | |
| 318 | + | const claims = JSON.parse(Buffer.from(payload, "base64url").toString()); | |
| 319 | + | if (claims.k !== key || claims.e < Date.now() / 1000) return null; | |
| 320 | + | return claims; | |
| 321 | + | } | |
| 322 | + | ||
| 323 | + | function bearer(request) { | |
| 324 | + | const header = request.headers.authorization ?? ""; | |
| 325 | + | if (/^bearer /i.test(header)) return header.slice(7).trim(); | |
| 326 | + | if (/^basic /i.test(header)) { | |
| 327 | + | // A git client given the token as a password: `x:<token>`. | |
| 328 | + | const decoded = Buffer.from(header.slice(6).trim(), "base64").toString(); | |
| 329 | + | return decoded.slice(decoded.indexOf(":") + 1); | |
| 330 | + | } | |
| 331 | + | return null; | |
| 332 | + | } | |
| 333 | + | ||
| 334 | + | // ── Smart HTTP through git http-backend ───────────────────────────────── | |
| 335 | + | ||
| 336 | + | function smartHttp(request, response, key, rest, query) { | |
| 337 | + | if (!exists(key)) return send(response, 404, "not found"); | |
| 338 | + | const claims = checkToken(bearer(request), key); | |
| 339 | + | if (!claims) { | |
| 340 | + | response.writeHead(401, { "www-authenticate": 'Basic realm="g1t-gitstore"' }); | |
| 341 | + | return response.end("unauthorized"); | |
| 342 | + | } | |
| 343 | + | const service = rest === "info/refs" ? new URLSearchParams(query).get("service") : rest; | |
| 344 | + | if (service === "git-receive-pack" && claims.s !== "write") return send(response, 403, "read-only token"); | |
| 345 | + | if (service !== "git-upload-pack" && service !== "git-receive-pack") return send(response, 404, "not found"); | |
| 346 | + | ||
| 347 | + | const env = { | |
| 348 | + | PATH: process.env.PATH, | |
| 349 | + | GIT_PROJECT_ROOT: ROOT, | |
| 350 | + | GIT_HTTP_EXPORT_ALL: "1", | |
| 351 | + | REQUEST_METHOD: request.method, | |
| 352 | + | PATH_INFO: `/${key}.git/${rest}`, | |
| 353 | + | QUERY_STRING: query, | |
| 354 | + | CONTENT_TYPE: request.headers["content-type"] ?? "", | |
| 355 | + | REMOTE_USER: "g1t", | |
| 356 | + | REMOTE_ADDR: request.socket.remoteAddress ?? "", | |
| 357 | + | }; | |
| 358 | + | if (request.headers["git-protocol"]) env.GIT_PROTOCOL = request.headers["git-protocol"]; | |
| 359 | + | if (request.headers["content-encoding"]) env.HTTP_CONTENT_ENCODING = request.headers["content-encoding"]; | |
| 360 | + | if (request.headers["content-length"]) env.CONTENT_LENGTH = request.headers["content-length"]; | |
| 361 | + | ||
| 362 | + | const child = spawn("git", ["http-backend"], { env, stdio: ["pipe", "pipe", "pipe"] }); | |
| 363 | + | request.pipe(child.stdin); | |
| 364 | + | child.stderr.on("data", (chunk) => process.stderr.write(chunk)); | |
| 365 | + | ||
| 366 | + | // CGI: headers, a blank line, then the body. | |
| 367 | + | let buffered = Buffer.alloc(0); | |
| 368 | + | let headersDone = false; | |
| 369 | + | child.stdout.on("data", (chunk) => { | |
| 370 | + | if (headersDone) return response.write(chunk); | |
| 371 | + | buffered = Buffer.concat([buffered, chunk]); | |
| 372 | + | let end = buffered.indexOf("\r\n\r\n"); | |
| 373 | + | let gap = 4; | |
| 374 | + | if (end === -1) { | |
| 375 | + | end = buffered.indexOf("\n\n"); | |
| 376 | + | gap = 2; | |
| 377 | + | } | |
| 378 | + | if (end === -1) return; | |
| 379 | + | headersDone = true; | |
| 380 | + | let status = 200; | |
| 381 | + | const headers = {}; | |
| 382 | + | for (const line of buffered.slice(0, end).toString().split(/\r?\n/)) { | |
| 383 | + | const colon = line.indexOf(":"); | |
| 384 | + | if (colon === -1) continue; | |
| 385 | + | const name = line.slice(0, colon).trim().toLowerCase(); | |
| 386 | + | const value = line.slice(colon + 1).trim(); | |
| 387 | + | if (name === "status") status = Number.parseInt(value, 10); | |
| 388 | + | else headers[name] = value; | |
| 389 | + | } | |
| 390 | + | response.writeHead(status, headers); | |
| 391 | + | response.write(buffered.slice(end + gap)); | |
| 392 | + | }); | |
| 393 | + | child.on("close", (code) => { | |
| 394 | + | if (!headersDone) { | |
| 395 | + | send(response, 500, "git http-backend failed"); | |
| 396 | + | return; | |
| 397 | + | } | |
| 398 | + | if (service === "git-receive-pack" && request.method === "POST" && code === 0) { | |
| 399 | + | const marker = join(repoDir(key), "g1t-pushed"); | |
| 400 | + | try { | |
| 401 | + | utimesSync(marker, new Date(), new Date()); | |
| 402 | + | } catch { | |
| 403 | + | writeFileSync(marker, ""); | |
| 404 | + | } | |
| 405 | + | } | |
| 406 | + | response.end(); | |
| 407 | + | }); | |
| 408 | + | } | |
| 409 | + | ||
| 410 | + | // ── HTTP ──────────────────────────────────────────────────────────────── | |
| 411 | + | ||
| 412 | + | function send(response, status, body, headers = {}) { | |
| 413 | + | const isBuffer = Buffer.isBuffer(body); | |
| 414 | + | const payload = isBuffer ? body : typeof body === "string" ? body : JSON.stringify(body); | |
| 415 | + | response.writeHead(status, { | |
| 416 | + | "content-type": isBuffer ? "application/octet-stream" : typeof body === "string" ? "text/plain" : "application/json", | |
| 417 | + | ...headers, | |
| 418 | + | }); | |
| 419 | + | response.end(payload); | |
| 420 | + | } | |
| 421 | + | ||
| 422 | + | async function readJson(request) { | |
| 423 | + | const chunks = []; | |
| 424 | + | for await (const chunk of request) chunks.push(chunk); | |
| 425 | + | const text = Buffer.concat(chunks).toString(); | |
| 426 | + | return text ? JSON.parse(text) : {}; | |
| 427 | + | } | |
| 428 | + | ||
| 429 | + | function authorized(request) { | |
| 430 | + | const given = Buffer.from(request.headers["x-gitstore-secret"] ?? ""); | |
| 431 | + | const expected = Buffer.from(SECRET); | |
| 432 | + | return given.length === expected.length && timingSafeEqual(given, expected); | |
| 433 | + | } | |
| 434 | + | ||
| 435 | + | async function api(request, response, parts, params) { | |
| 436 | + | if (!authorized(request)) return send(response, 401, { code: "UNAUTHORIZED", message: "bad secret" }); | |
| 437 | + | const method = request.method; | |
| 438 | + | // POST /api/repos create | |
| 439 | + | if (parts.length === 0 && method === "POST") { | |
| 440 | + | const body = await readJson(request); | |
| 441 | + | return send(response, 200, await create(body.name, body)); | |
| 442 | + | } | |
| 443 | + | const [key, action, arg] = parts; | |
| 444 | + | if (method === "GET" && !action) return send(response, 200, await info(key)); | |
| 445 | + | if (method === "POST" && action === "tokens") { | |
| 446 | + | requireRepo(key); | |
| 447 | + | const body = await readJson(request); | |
| 448 | + | return send(response, 200, mintToken(key, body.scope, body.ttl)); | |
| 449 | + | } | |
| 450 | + | if (method === "POST" && action === "fork") { | |
| 451 | + | const body = await readJson(request); | |
| 452 | + | return send(response, 200, await fork(key, body.name, body)); | |
| 453 | + | } | |
| 454 | + | if (method === "GET" && action === "commits") { | |
| 455 | + | return send(response, 200, await readCommit(key, arg)); | |
| 456 | + | } | |
| 457 | + | if (method === "GET" && action === "log") { | |
| 458 | + | return send(response, 200, await log(key, Object.fromEntries(params))); | |
| 459 | + | } | |
| 460 | + | if (method === "GET" && action === "trees") { | |
| 461 | + | return send(response, 200, await readTree(key, arg)); | |
| 462 | + | } | |
| 463 | + | if (method === "GET" && (action === "blobs" || action === "file")) { | |
| 464 | + | const bytes = | |
| 465 | + | action === "blobs" ? await readBlob(key, arg) : await readFile(key, params.get("ref"), params.get("path")); | |
| 466 | + | return bytes ? send(response, 200, bytes) : send(response, 404, { code: "NOT_FOUND", message: "no such object" }); | |
| 467 | + | } | |
| 468 | + | return send(response, 404, { code: "NOT_FOUND", message: "no such route" }); | |
| 469 | + | } | |
| 470 | + | ||
| 471 | + | const server = createServer(async (request, response) => { | |
| 472 | + | const url = new URL(request.url, "http://gitstore"); | |
| 473 | + | try { | |
| 474 | + | if (url.pathname === "/healthz") return send(response, 200, "ok"); | |
| 475 | + | const git = /^\/git\/([^/]+)\.git\/(info\/refs|git-upload-pack|git-receive-pack)$/.exec(url.pathname); | |
| 476 | + | if (git) return smartHttp(request, response, decodeURIComponent(git[1]), git[2], url.search.slice(1)); | |
| 477 | + | if (url.pathname === "/api/repos" || url.pathname.startsWith("/api/repos/")) { | |
| 478 | + | const parts = url.pathname.slice("/api/repos".length).split("/").filter(Boolean).map(decodeURIComponent); | |
| 479 | + | return await api(request, response, parts, url.searchParams); | |
| 480 | + | } | |
| 481 | + | send(response, 404, "not found"); | |
| 482 | + | } catch (error) { | |
| 483 | + | const status = error instanceof StoreError ? error.status : 500; | |
| 484 | + | const code = error instanceof StoreError ? error.code : "INTERNAL_ERROR"; | |
| 485 | + | if (status >= 500) console.error(error); | |
| 486 | + | if (!response.headersSent) send(response, status, { code, message: error.message }); | |
| 487 | + | else response.end(); | |
| 488 | + | } | |
| 489 | + | }); | |
| 490 | + | ||
| 491 | + | server.listen(PORT, () => { | |
| 492 | + | console.log(`g1t gitstore: ${ROOT} on :${PORT} (remote ${PUBLIC_URL})`); | |
| 493 | + | }); | |
| 494 | + | ||
| 495 | + | for (const signal of ["SIGINT", "SIGTERM"]) { | |
| 496 | + | process.on(signal, () => server.close(() => process.exit(0))); | |
| 497 | + | } |
| 1 | + | #!/usr/bin/env bash | |
| 2 | + | # End-to-end check of a self-hosted g1t: sign up, confirm the email, make a | |
| 3 | + | # workspace and a repository, push and clone over HTTP, open an issue, and | |
| 4 | + | # read the code back through the site. | |
| 5 | + | # | |
| 6 | + | # ./smoke.sh # against the compose stack's defaults | |
| 7 | + | # G1T_URL=http://localhost:8787 MAIL_LOG=wrangler.log ./smoke.sh | |
| 8 | + | # | |
| 9 | + | # The confirmation link is read from Mailpit (MAILPIT_URL, the default) or, | |
| 10 | + | # with MAIL_LOG set, from a log the mail Worker printed it to. | |
| 11 | + | set -euo pipefail | |
| 12 | + | ||
| 13 | + | G1T_URL="${G1T_URL:-http://localhost:8787}" | |
| 14 | + | MAILPIT_URL="${MAILPIT_URL:-http://localhost:8025}" | |
| 15 | + | MAIL_LOG="${MAIL_LOG:-}" | |
| 16 | + | RUN="$(date +%s)" | |
| 17 | + | USER_NAME="smoke${RUN}" | |
| 18 | + | EMAIL="${USER_NAME}@example.com" | |
| 19 | + | PASSWORD="correct-horse-${RUN}" | |
| 20 | + | WORKSPACE="ws${RUN}" | |
| 21 | + | REPO="hello" | |
| 22 | + | WORK="$(mktemp -d)" | |
| 23 | + | JAR="$WORK/cookies" | |
| 24 | + | trap 'rm -rf "$WORK"' EXIT | |
| 25 | + | ||
| 26 | + | step() { printf '\n== %s\n' "$*"; } | |
| 27 | + | fail() { printf 'FAILED: %s\n' "$*" >&2; exit 1; } | |
| 28 | + | ||
| 29 | + | # A form POST as a browser sends it, with the Origin the site checks. | |
| 30 | + | post() { | |
| 31 | + | local path="$1"; shift | |
| 32 | + | curl -sS -o "$WORK/body" -w '%{http_code} %{redirect_url}' -b "$JAR" -c "$JAR" \ | |
| 33 | + | -H "Origin: $G1T_URL" "$@" "$G1T_URL$path" | |
| 34 | + | } | |
| 35 | + | get() { | |
| 36 | + | curl -sS -o "$WORK/body" -w '%{http_code}' -b "$JAR" -c "$JAR" "$G1T_URL$1" | |
| 37 | + | } | |
| 38 | + | ||
| 39 | + | step "site answers at $G1T_URL" | |
| 40 | + | [ "$(get /)" = 200 ] || fail "GET / did not answer 200" | |
| 41 | + | ||
| 42 | + | step "sign up as $USER_NAME" | |
| 43 | + | out="$(post /register --data-urlencode "username=$USER_NAME" --data-urlencode "email=$EMAIL" --data-urlencode "password=$PASSWORD")" | |
| 44 | + | echo "$out" | |
| 45 | + | case "$out" in 30[23]*) ;; *) fail "register: $out $(head -c 300 "$WORK/body")" ;; esac | |
| 46 | + | # curl keeps Secure cookies only for https or localhost; carry it by hand. | |
| 47 | + | grep -q g1t_session "$JAR" || fail "no session cookie" | |
| 48 | + | ||
| 49 | + | step "confirm the email" | |
| 50 | + | link="" | |
| 51 | + | for _ in $(seq 1 20); do | |
| 52 | + | if [ -n "$MAIL_LOG" ]; then | |
| 53 | + | link="$(grep -ao "[a-z]*://[^ \"<]*/verify?token=[0-9a-zA-Z_-]*" "$MAIL_LOG" | tail -1 || true)" | |
| 54 | + | else | |
| 55 | + | id="$(curl -sS "$MAILPIT_URL/api/v1/search?query=to:$EMAIL" | sed -n 's/.*"ID":"\([^"]*\)".*/\1/p' | head -1)" | |
| 56 | + | [ -n "$id" ] && link="$(curl -sS "$MAILPIT_URL/api/v1/message/$id" | grep -ao '[a-z]*://[^ "<\\]*/verify?token=[0-9a-zA-Z_-]*' | head -1 || true)" | |
| 57 | + | fi | |
| 58 | + | [ -n "$link" ] && break | |
| 59 | + | sleep 1 | |
| 60 | + | done | |
| 61 | + | [ -n "$link" ] || fail "no confirmation email arrived" | |
| 62 | + | echo "$link" | |
| 63 | + | [ "$(get "/verify?${link#*\?}")" = 200 ] || fail "verify" | |
| 64 | + | grep -q "$USER_NAME" "$WORK/body" || fail "verify page does not name the account" | |
| 65 | + | ||
| 66 | + | step "create workspace $WORKSPACE" | |
| 67 | + | out="$(post /workspaces/new --data-urlencode "slug=$WORKSPACE" --data-urlencode "displayName=Smoke $RUN")" | |
| 68 | + | echo "$out" | |
| 69 | + | case "$out" in 30[23]*) ;; *) fail "workspace: $out $(head -c 300 "$WORK/body")" ;; esac | |
| 70 | + | ||
| 71 | + | step "create repository $WORKSPACE/$REPO" | |
| 72 | + | out="$(post /new --data-urlencode "workspace=$WORKSPACE" --data-urlencode "name=$REPO" --data-urlencode "description=Self-host smoke test" --data-urlencode "visibility=public" --data-urlencode "source=empty")" | |
| 73 | + | echo "$out" | |
| 74 | + | case "$out" in 30[23]*) ;; *) fail "repo: $out $(head -c 300 "$WORK/body")" ;; esac | |
| 75 | + | ||
| 76 | + | step "push over HTTP" | |
| 77 | + | remote="${G1T_URL/:\/\//://$USER_NAME:$PASSWORD@}/$WORKSPACE/$REPO.git" | |
| 78 | + | git init -q -b main "$WORK/src" | |
| 79 | + | ( | |
| 80 | + | cd "$WORK/src" | |
| 81 | + | git config user.name "Smoke Test" | |
| 82 | + | git config user.email "$EMAIL" | |
| 83 | + | # A throwaway commit: never signed, whatever the global config says. | |
| 84 | + | git config commit.gpgsign false | |
| 85 | + | printf '# hello\n\nPushed to a self-hosted g1t.\n' > README.md | |
| 86 | + | mkdir -p src && printf 'fn main() {\n println!("hello from g1t");\n}\n' > src/main.rs | |
| 87 | + | git add . && git commit -qm "First commit" | |
| 88 | + | git -c credential.helper= push -q "$remote" main | |
| 89 | + | ) | |
| 90 | + | echo "pushed $(git -C "$WORK/src" rev-parse --short HEAD)" | |
| 91 | + | ||
| 92 | + | step "clone over HTTP" | |
| 93 | + | git -c credential.helper= clone -q "$G1T_URL/$WORKSPACE/$REPO.git" "$WORK/clone" | |
| 94 | + | diff -q "$WORK/src/README.md" "$WORK/clone/README.md" || fail "clone differs" | |
| 95 | + | echo "clone matches" | |
| 96 | + | ||
| 97 | + | step "open an issue" | |
| 98 | + | out="$(post "/$WORKSPACE/$REPO/issues/new" --data-urlencode "title=It works" --data-urlencode "body=Opened by smoke.sh")" | |
| 99 | + | echo "$out" | |
| 100 | + | case "$out" in 30[23]*/issues/1) ;; *) fail "issue: $out $(head -c 300 "$WORK/body")" ;; esac | |
| 101 | + | [ "$(get "/$WORKSPACE/$REPO/issues/1")" = 200 ] || fail "issue page" | |
| 102 | + | grep -q "It works" "$WORK/body" || fail "issue page does not show the title" | |
| 103 | + | ||
| 104 | + | step "browse code in the site" | |
| 105 | + | [ "$(get "/$WORKSPACE/$REPO/code")" = 200 ] || fail "code page" | |
| 106 | + | grep -q "README.md" "$WORK/body" || fail "code page does not list README.md" | |
| 107 | + | [ "$(get "/$WORKSPACE/$REPO/blob/main/src/main.rs")" = 200 ] || fail "blob page" | |
| 108 | + | grep -q "hello from g1t" "$WORK/body" || fail "blob page does not show the file" | |
| 109 | + | [ "$(get "/$WORKSPACE/$REPO/commits")" = 200 ] || fail "commits page" | |
| 110 | + | grep -q "First commit" "$WORK/body" || fail "commits page does not show the commit" | |
| 111 | + | ||
| 112 | + | printf '\nAll checks passed: %s/%s/%s\n' "$G1T_URL" "$WORKSPACE" "$REPO" |
| 1 | + | #!/usr/bin/env bash | |
| 2 | + | # Starts self-hosted g1t inside its container: makes the keys it needs once, | |
| 3 | + | # writes the Wrangler configs, brings every database up to date, and runs | |
| 4 | + | # every Worker in one workerd on :8787. | |
| 5 | + | set -euo pipefail | |
| 6 | + | ||
| 7 | + | cd "$(dirname "$0")" | |
| 8 | + | DATA="${G1T_DATA:-/data}" | |
| 9 | + | STATE="$DATA/state" | |
| 10 | + | KEYS="$DATA/keys.env" | |
| 11 | + | GENERATED="$DATA/generated" | |
| 12 | + | WRANGLER="$(cd ../.. && pwd)/node_modules/.bin/wrangler" | |
| 13 | + | mkdir -p "$STATE" | |
| 14 | + | ||
| 15 | + | # Keys that seal secrets at rest (actions, integrations, webhooks). Made on | |
| 16 | + | # first start and kept on the volume: losing them loses those secrets. | |
| 17 | + | if [ ! -f "$KEYS" ]; then | |
| 18 | + | umask 077 | |
| 19 | + | { | |
| 20 | + | echo "ACTIONS_KEY=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')" | |
| 21 | + | echo "INTEGRATIONS_KEY=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')" | |
| 22 | + | echo "WEBHOOKS_KEY=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')" | |
| 23 | + | } > "$KEYS" | |
| 24 | + | fi | |
| 25 | + | set -a | |
| 26 | + | # shellcheck disable=SC1090 | |
| 27 | + | . "$KEYS" | |
| 28 | + | set +a | |
| 29 | + | ||
| 30 | + | # The git store's shared secret: given, or the file the git store made. | |
| 31 | + | if [ -z "${GITSTORE_SECRET:-}" ] && [ -n "${GITSTORE_SECRET_FILE:-}" ]; then | |
| 32 | + | for _ in $(seq 1 60); do [ -s "$GITSTORE_SECRET_FILE" ] && break; sleep 1; done | |
| 33 | + | GITSTORE_SECRET="$(cat "$GITSTORE_SECRET_FILE")" | |
| 34 | + | export GITSTORE_SECRET | |
| 35 | + | fi | |
| 36 | + | [ -n "${GITSTORE_SECRET:-}" ] || { echo "No GITSTORE_SECRET or GITSTORE_SECRET_FILE." >&2; exit 1; } | |
| 37 | + | ||
| 38 | + | node configs.mjs "$GENERATED" | |
| 39 | + | cd "$GENERATED" | |
| 40 | + | ||
| 41 | + | # Migrations: the same files D1 gets, applied to the SQLite files on the | |
| 42 | + | # volume. Applied ones are recorded, so this is safe on every start. | |
| 43 | + | for config in $(cat workers.txt); do | |
| 44 | + | for database in $(node -e "for (const d of require('./$config').d1_databases ?? []) console.log(d.database_name)"); do | |
| 45 | + | echo "Migrating $database" | |
| 46 | + | "$WRANGLER" d1 migrations apply "$database" --local --persist-to "$STATE" -c "$config" >/dev/null | |
| 47 | + | done | |
| 48 | + | done | |
| 49 | + | ||
| 50 | + | args=() | |
| 51 | + | while read -r config; do args+=(-c "$config"); done < workers.txt | |
| 52 | + | echo "g1t is starting on ${PUBLIC_URL:-http://localhost:8787}" | |
| 53 | + | exec "$WRANGLER" dev "${args[@]}" \ | |
| 54 | + | --ip 0.0.0.0 --port 8787 \ | |
| 55 | + | --persist-to "$STATE" \ | |
| 56 | + | --show-interactive-dev-session=false |
| 1 | + | // The Artifacts binding, for self-hosted g1t. | |
| 2 | + | // | |
| 3 | + | // The repos service is written against Cloudflare Artifacts' Workers | |
| 4 | + | // binding (services/repos/src/store.rs). Self-hosted, its ARTIFACTS binding | |
| 5 | + | // is a service binding to this Worker instead, which offers the same | |
| 6 | + | // methods and keeps the repositories in the git store (gitstore/server.mjs): | |
| 7 | + | // plain bare repositories on disk. Hosted g1t never runs this. | |
| 8 | + | // | |
| 9 | + | // Only what g1t calls is implemented: create and get on the namespace; | |
| 10 | + | // info, createToken, log, readCommit, readTree, readBlob, readFile and fork | |
| 11 | + | // on a repository. | |
| 12 | + | ||
| 13 | + | import { RpcTarget, WorkerEntrypoint } from "cloudflare:workers"; | |
| 14 | + | ||
| 15 | + | class ArtifactsError extends Error { | |
| 16 | + | constructor(code, message) { | |
| 17 | + | super(message); | |
| 18 | + | this.name = "ArtifactsError"; | |
| 19 | + | this.code = code; | |
| 20 | + | } | |
| 21 | + | } | |
| 22 | + | ||
| 23 | + | async function store(env, path, init = {}) { | |
| 24 | + | const base = (env.GITSTORE_URL ?? "http://gitstore:8080").replace(/\/$/, ""); | |
| 25 | + | const response = await fetch(`${base}/api/repos${path}`, { | |
| 26 | + | ...init, | |
| 27 | + | headers: { | |
| 28 | + | "x-gitstore-secret": env.GITSTORE_SECRET ?? "", | |
| 29 | + | ...(init.body ? { "content-type": "application/json" } : {}), | |
| 30 | + | }, | |
| 31 | + | }); | |
| 32 | + | return response; | |
| 33 | + | } | |
| 34 | + | ||
| 35 | + | async function json(response) { | |
| 36 | + | if (response.ok) return response.json(); | |
| 37 | + | let code = "INTERNAL_ERROR"; | |
| 38 | + | let message = `git store answered ${response.status}`; | |
| 39 | + | try { | |
| 40 | + | const body = await response.json(); | |
| 41 | + | code = body.code ?? code; | |
| 42 | + | message = body.message ?? message; | |
| 43 | + | } catch {} | |
| 44 | + | throw new ArtifactsError(code, message); | |
| 45 | + | } | |
| 46 | + | ||
| 47 | + | /** Bytes as something with `arrayBuffer()`, the way a Blob is read. */ | |
| 48 | + | async function bytes(response) { | |
| 49 | + | if (response.status === 404) return null; | |
| 50 | + | if (!response.ok) await json(response); | |
| 51 | + | return new Response(await response.arrayBuffer(), { | |
| 52 | + | headers: { "content-type": response.headers.get("content-type") ?? "application/octet-stream" }, | |
| 53 | + | }); | |
| 54 | + | } | |
| 55 | + | ||
| 56 | + | class Repo extends RpcTarget { | |
| 57 | + | #env; | |
| 58 | + | #name; | |
| 59 | + | ||
| 60 | + | constructor(env, name) { | |
| 61 | + | super(); | |
| 62 | + | this.#env = env; | |
| 63 | + | this.#name = name; | |
| 64 | + | } | |
| 65 | + | ||
| 66 | + | #path(rest = "") { | |
| 67 | + | return `/${encodeURIComponent(this.#name)}${rest}`; | |
| 68 | + | } | |
| 69 | + | ||
| 70 | + | async info() { | |
| 71 | + | return json(await store(this.#env, this.#path())); | |
| 72 | + | } | |
| 73 | + | ||
| 74 | + | async createToken(scope = "write", ttl = 86400) { | |
| 75 | + | return json( | |
| 76 | + | await store(this.#env, this.#path("/tokens"), { | |
| 77 | + | method: "POST", | |
| 78 | + | body: JSON.stringify({ scope, ttl }), | |
| 79 | + | }), | |
| 80 | + | ); | |
| 81 | + | } | |
| 82 | + | ||
| 83 | + | async log(opts = {}) { | |
| 84 | + | const query = new URLSearchParams(); | |
| 85 | + | for (const [key, value] of Object.entries(opts ?? {})) { | |
| 86 | + | if (value !== undefined && value !== null) query.set(key, String(value)); | |
| 87 | + | } | |
| 88 | + | return json(await store(this.#env, this.#path(`/log?${query}`))); | |
| 89 | + | } | |
| 90 | + | ||
| 91 | + | async readCommit(hash) { | |
| 92 | + | return json(await store(this.#env, this.#path(`/commits/${encodeURIComponent(hash)}`))); | |
| 93 | + | } | |
| 94 | + | ||
| 95 | + | async readTree(hash) { | |
| 96 | + | return json(await store(this.#env, this.#path(`/trees/${encodeURIComponent(hash)}`))); | |
| 97 | + | } | |
| 98 | + | ||
| 99 | + | async readBlob(hash) { | |
| 100 | + | return bytes(await store(this.#env, this.#path(`/blobs/${encodeURIComponent(hash)}`))); | |
| 101 | + | } | |
| 102 | + | ||
| 103 | + | async readFile({ ref, path }) { | |
| 104 | + | const query = new URLSearchParams({ ref, path }); | |
| 105 | + | return bytes(await store(this.#env, this.#path(`/file?${query}`))); | |
| 106 | + | } | |
| 107 | + | ||
| 108 | + | async fork(name, opts = {}) { | |
| 109 | + | const created = await json( | |
| 110 | + | await store(this.#env, this.#path("/fork"), { | |
| 111 | + | method: "POST", | |
| 112 | + | body: JSON.stringify({ ...opts, name }), | |
| 113 | + | }), | |
| 114 | + | ); | |
| 115 | + | const token = await new Repo(this.#env, name).createToken("write"); | |
| 116 | + | return { ...created, token: token.plaintext }; | |
| 117 | + | } | |
| 118 | + | } | |
| 119 | + | ||
| 120 | + | export default class Artifacts extends WorkerEntrypoint { | |
| 121 | + | async create(name, opts = {}) { | |
| 122 | + | const created = await json( | |
| 123 | + | await store(this.env, "", { | |
| 124 | + | method: "POST", | |
| 125 | + | body: JSON.stringify({ | |
| 126 | + | name, | |
| 127 | + | description: opts?.description, | |
| 128 | + | defaultBranch: opts?.setDefaultBranch, | |
| 129 | + | readOnly: opts?.readOnly, | |
| 130 | + | }), | |
| 131 | + | }), | |
| 132 | + | ); | |
| 133 | + | const token = await new Repo(this.env, name).createToken("write"); | |
| 134 | + | return { ...created, token: token.plaintext }; | |
| 135 | + | } | |
| 136 | + | ||
| 137 | + | async get(name) { | |
| 138 | + | // Artifacts answers NOT_FOUND here for a repository that does not exist. | |
| 139 | + | await json(await store(this.env, `/${encodeURIComponent(name)}`)); | |
| 140 | + | return new Repo(this.env, name); | |
| 141 | + | } | |
| 142 | + | ||
| 143 | + | async fetch() { | |
| 144 | + | return new Response("The Artifacts binding for self-hosted g1t. Bind to it; do not browse it.", { | |
| 145 | + | status: 404, | |
| 146 | + | }); | |
| 147 | + | } | |
| 148 | + | } |
| 1 | + | // The Email Sending binding, for self-hosted g1t. | |
| 2 | + | // | |
| 3 | + | // The identity service sends mail through Cloudflare Email Sending: | |
| 4 | + | // `env.EMAIL.send({ to, from, subject, text, html })`. Self-hosted, EMAIL is | |
| 5 | + | // a service binding to this Worker, which offers the same method and: | |
| 6 | + | // | |
| 7 | + | // - always prints the message to the log, so a self-hoster with no mail | |
| 8 | + | // server can still confirm an address by following the link; | |
| 9 | + | // - with MAIL_URL set, hands it to a Mailpit server's send API | |
| 10 | + | // (`POST <MAIL_URL>/api/v1/send`). Mailpit keeps it in its inbox, or | |
| 11 | + | // relays it to a real SMTP server when it is configured to. | |
| 12 | + | // | |
| 13 | + | // Links in messages name g1t.sh, where hosted g1t lives; they are rewritten | |
| 14 | + | // to PUBLIC_URL so they come back to this installation. | |
| 15 | + | ||
| 16 | + | import { WorkerEntrypoint } from "cloudflare:workers"; | |
| 17 | + | ||
| 18 | + | const HOSTED = /https:\/\/g1t\.sh/g; | |
| 19 | + | ||
| 20 | + | function address(value) { | |
| 21 | + | const match = /^(.*)<([^>]+)>\s*$/.exec(value ?? ""); | |
| 22 | + | return match ? { Name: match[1].trim(), Email: match[2].trim() } : { Email: String(value ?? "") }; | |
| 23 | + | } | |
| 24 | + | ||
| 25 | + | export default class Mail extends WorkerEntrypoint { | |
| 26 | + | async send(message) { | |
| 27 | + | const site = (this.env.PUBLIC_URL ?? "http://localhost:8787").replace(/\/$/, ""); | |
| 28 | + | const text = String(message?.text ?? "").replace(HOSTED, site); | |
| 29 | + | const html = String(message?.html ?? "").replace(HOSTED, site); | |
| 30 | + | const from = this.env.MAIL_FROM || message?.from; | |
| 31 | + | const to = Array.isArray(message?.to) ? message.to : [message?.to]; | |
| 32 | + | ||
| 33 | + | console.log(`[mail] to ${to.join(", ")}: ${message?.subject}\n${text}`); | |
| 34 | + | ||
| 35 | + | if (this.env.MAIL_URL) { | |
| 36 | + | const response = await fetch(`${this.env.MAIL_URL.replace(/\/$/, "")}/api/v1/send`, { | |
| 37 | + | method: "POST", | |
| 38 | + | headers: { "content-type": "application/json" }, | |
| 39 | + | body: JSON.stringify({ | |
| 40 | + | From: address(from), | |
| 41 | + | To: to.map(address), | |
| 42 | + | Subject: message?.subject ?? "", | |
| 43 | + | Text: text, | |
| 44 | + | HTML: html, | |
| 45 | + | }), | |
| 46 | + | }); | |
| 47 | + | if (!response.ok) { | |
| 48 | + | throw new Error(`mail server answered ${response.status}: ${await response.text()}`); | |
| 49 | + | } | |
| 50 | + | } | |
| 51 | + | return { messageId: crypto.randomUUID() }; | |
| 52 | + | } | |
| 53 | + | ||
| 54 | + | async fetch() { | |
| 55 | + | return new Response("The Email Sending binding for self-hosted g1t.", { status: 404 }); | |
| 56 | + | } | |
| 57 | + | } |
| 1 | + | // A service that is turned off on this installation. | |
| 2 | + | // | |
| 3 | + | // Self-hosted phase 1 runs the core forge only. The site and the services | |
| 4 | + | // still hold bindings to the runner (agents), the context hub and the | |
| 5 | + | // deployments' builders; this Worker stands in for each of them, so a | |
| 6 | + | // page that asks "are agents on here?" is told no instead of failing. | |
| 7 | + | // | |
| 8 | + | // - As the runner (RunnerApi in packages/contracts/src/runner.ts): agents | |
| 9 | + | // are not enabled, no model is reachable, and starting anything fails with | |
| 10 | + | // a message that says why. | |
| 11 | + | // - For the JSON protocol the Rust services speak (`POST /rpc/<method>`): | |
| 12 | + | // every method answers with a failed Result. | |
| 13 | + | // | |
| 14 | + | // OFF_NAME names the feature in those messages, such as "Agents". | |
| 15 | + | ||
| 16 | + | import { WorkerEntrypoint } from "cloudflare:workers"; | |
| 17 | + | ||
| 18 | + | function off(env) { | |
| 19 | + | const name = env.OFF_NAME ?? "This feature"; | |
| 20 | + | return { | |
| 21 | + | ok: false, | |
| 22 | + | error: { code: "forbidden", message: `${name} are off on this installation of g1t.` }, | |
| 23 | + | }; | |
| 24 | + | } | |
| 25 | + | ||
| 26 | + | export default class Off extends WorkerEntrypoint { | |
| 27 | + | // ── The runner's methods, as the site calls them ── | |
| 28 | + | async enabled() { | |
| 29 | + | return false; | |
| 30 | + | } | |
| 31 | + | async modelAccess() { | |
| 32 | + | return { own: null, hosted: false, trial: null }; | |
| 33 | + | } | |
| 34 | + | async instructions() { | |
| 35 | + | return off(this.env); | |
| 36 | + | } | |
| 37 | + | async run() { | |
| 38 | + | return off(this.env); | |
| 39 | + | } | |
| 40 | + | async plan() { | |
| 41 | + | return off(this.env); | |
| 42 | + | } | |
| 43 | + | async applyPlan() { | |
| 44 | + | return off(this.env); | |
| 45 | + | } | |
| 46 | + | async update() { | |
| 47 | + | return off(this.env); | |
| 48 | + | } | |
| 49 | + | async review() { | |
| 50 | + | return off(this.env); | |
| 51 | + | } | |
| 52 | + | async recheck() { | |
| 53 | + | return off(this.env); | |
| 54 | + | } | |
| 55 | + | async stopRun() { | |
| 56 | + | return off(this.env); | |
| 57 | + | } | |
| 58 | + | ||
| 59 | + | // ── The JSON protocol ── | |
| 60 | + | async fetch(request) { | |
| 61 | + | const { pathname } = new URL(request.url); | |
| 62 | + | if (request.method === "POST" && pathname.startsWith("/rpc/")) { | |
| 63 | + | return Response.json(off(this.env)); | |
| 64 | + | } | |
| 65 | + | return new Response("Off on this installation.", { status: 404 }); | |
| 66 | + | } | |
| 67 | + | ||
| 68 | + | // Events and schedules for a service that is off are dropped. | |
| 69 | + | async queue(batch) { | |
| 70 | + | batch.ackAll(); | |
| 71 | + | } | |
| 72 | + | async scheduled() {} | |
| 73 | + | } |
| 1184 | 1184 | | Transcripts and logs | R2 | | |
| 1185 | 1185 | | Email, bot protection, keys | Email Sending, Turnstile, Secrets Store | | |
| 1186 | 1186 | ||
| 1187 | + | ## Running g1t yourself | |
| 1188 | + | ||
| 1189 | + | g1t.sh runs on Cloudflare, and that does not change. The core is MIT and | |
| 1190 | + | must also run on anyone's own machine with `docker compose up`. A free | |
| 1191 | + | core people can self-host is what makes paid hosting worth trusting. | |
| 1192 | + | Self-hosting never makes hosted worse: hosted code paths keep their | |
| 1193 | + | behaviour, and a self-hosted adapter sits beside the hosted one. The | |
| 1194 | + | inventory of every Cloudflare dependency, the design and the risks are in | |
| 1195 | + | [SELF_HOSTING.md](SELF_HOSTING.md). | |
| 1196 | + | ||
| 1197 | + | The approach: the Workers stay Workers, and self-hosted they run in | |
| 1198 | + | workerd, the open-source Workers runtime. D1, KV and Queues are SQLite on a | |
| 1199 | + | volume, with the same migrations. Cloudflare-only bindings are replaced | |
| 1200 | + | by stand-ins: | |
| 1201 | + | ||
| 1202 | + | - Artifacts becomes bare repositories served by `git http-backend`; | |
| 1203 | + | - Email Sending becomes SMTP, through Mailpit; | |
| 1204 | + | - services that are off answer "off" instead of failing. | |
| 1205 | + | ||
| 1206 | + | | Phase | Scope | Estimate | | |
| 1207 | + | | --- | --- | --- | | |
| 1208 | + | | 1. Core forge | Done: `deploy/self-host/` (compose, git store, binding stand-ins, smoke test) and the "Run g1t yourself" guide. Left: the API on its own port, `PUBLIC_URL` in place of hard-coded hosts, cron, pull requests in the smoke test, CI that runs it. | 1–1.5 weeks left | | |
| 1209 | + | | 2. Agents | Docker sandboxes with the same runner image, an egress allow-list proxy for guardrails, `g1t.toml`, a launcher in place of `wrangler dev` | 2–3 weeks | | |
| 1210 | + | | 3. Search, context, deployments | sqlite-vec plus an OpenAI-compatible embedder; an app host on workerd; Caddy for app and custom domains; SSH | 3–4 weeks | | |
| 1211 | + | | 4. Parity and upgrades | Code-level ports in `g1t_kit` and `@g1t/platform`, sudo without Access, online backups, released images, an upgrade test in CI | 3–4 weeks | | |
| 1212 | + | ||
| 1187 | 1213 | ## The submission | |
| 1188 | 1214 | ||
| 1189 | 1215 | - **g1t is built on g1t.** This repository is hosted on g1t.sh, its features |
| 1 | + | # Self-hosting g1t | |
| 2 | + | ||
| 3 | + | The goal (2026-10-05): g1t should not be locked to Cloudflare. Anyone should | |
| 4 | + | be able to run it on their own machine with `docker compose up`. The free | |
| 5 | + | core is MIT and self-hostable; managed hosting at g1t.sh is the paid | |
| 6 | + | product, and it stays on Cloudflare. Self-hosting must never make hosted | |
| 7 | + | g1t worse, so hosted code paths do not change to make room for it. | |
| 8 | + | ||
| 9 | + | This document covers: | |
| 10 | + | ||
| 11 | + | 1. An inventory of every Cloudflare dependency in the code. | |
| 12 | + | 2. The design: ports and adapters, with the runtime choice weighed. | |
| 13 | + | 3. What phase 1 ships today: `deploy/self-host/`, and what was verified. | |
| 14 | + | 4. The phased plan, with estimates. | |
| 15 | + | 5. The risks. | |
| 16 | + | ||
| 17 | + | The user guide is `apps/docs/src/content/docs/guides/self-hosting.md`. | |
| 18 | + | ||
| 19 | + | ## The short version | |
| 20 | + | ||
| 21 | + | - **g1t is already shaped for this.** Every service talks to every other | |
| 22 | + | over plain HTTP: `POST /rpc/<method>` with a JSON body (`crates/kit`, | |
| 23 | + | `packages/contracts/src/clients.ts`). Every database is SQLite (D1). | |
| 24 | + | Git storage sits behind a `GitStore` port (`services/repos/src/store.rs`), | |
| 25 | + | and everything past that port speaks git's smart HTTP to a remote URL | |
| 26 | + | with a bearer token. | |
| 27 | + | - **The fastest credible path is to run the Workers themselves in | |
| 28 | + | workerd**, the open-source Workers runtime, not to port them. Phase 1 | |
| 29 | + | runs every core Worker unchanged (the same WebAssembly and the same | |
| 30 | + | bundles that deploy to Cloudflare) in one workerd process under | |
| 31 | + | `wrangler dev`. D1, KV and Queues are kept on a volume as SQLite files. | |
| 32 | + | - **Three things replace Cloudflare-only bindings**, as small Workers bound | |
| 33 | + | in their place, with no change to the services: | |
| 34 | + | - `ARTIFACTS` becomes an Artifacts-compatible shim in front of a git | |
| 35 | + | store, which keeps plain bare repositories on disk and serves them | |
| 36 | + | with `git http-backend`. | |
| 37 | + | - `EMAIL` becomes a shim that logs each message and hands it to Mailpit, | |
| 38 | + | which can relay to any SMTP server. | |
| 39 | + | - The runner and the context hub are bound to an "off" Worker, which | |
| 40 | + | answers "agents are off" instead of failing. | |
| 41 | + | - **Proven on this machine with Docker:** sign up, confirm the email | |
| 42 | + | through Mailpit, create a workspace and a repository, push and clone over | |
| 43 | + | HTTP, open an issue, and browse code, commits and files in the site. All | |
| 44 | + | of it runs against local storage. See [Phase 1: what works today](#phase-1-what-works-today). | |
| 45 | + | - **Long term:** keep workerd as the runtime, because it is what hosted | |
| 46 | + | runs. Replace `wrangler dev` with a production workerd configuration. | |
| 47 | + | Move the binding shims into code-level ports in `g1t_kit` and a TS | |
| 48 | + | `@g1t/platform` package, so each primitive has a hosted and a | |
| 49 | + | self-hosted adapter behind one interface. | |
| 50 | + | ||
| 51 | + | ## 1. Inventory | |
| 52 | + | ||
| 53 | + | The sources are every `wrangler.jsonc` plus a grep of the code. Coupling | |
| 54 | + | is graded: | |
| 55 | + | ||
| 56 | + | - **thin**: one call site or a config switch; | |
| 57 | + | - **adapter**: already behind a port, or easy to put behind one; | |
| 58 | + | - **woven**: the logic is shaped around the product. | |
| 59 | + | ||
| 60 | + | ### By primitive | |
| 61 | + | ||
| 62 | + | | Primitive | Where | Coupling | Self-hosted equivalent | | |
| 63 | + | | --- | --- | --- | --- | | |
| 64 | + | | **Workers runtime**, service bindings | Every service. Rust through `worker` 0.8 (`#[event(fetch\|queue\|scheduled)]`, `Env`, `Fetcher`); TS as `export default { fetch, queue, scheduled }` | woven (as a runtime), thin (as an API) | **workerd**: the same runtime, open source. Service bindings work as they do hosted. Calls are HTTP (`POST /rpc/<method>`), so a native port could use plain HTTP clients. | | |
| 65 | + | | **Workers RPC** (JS methods across a binding) | Only `RUNNER`: `RunnerService extends WorkerEntrypoint` (`services/runner/src/index.ts:626`). `apps/web` calls `env.RUNNER.enabled/run/plan/...` directly in 11 routes. | thin | workerd supports it. A native port needs these on `/rpc/*` as well; the runner already has a `fetch` shim for Rust callers. | | |
| 66 | + | | **D1** | System of record for 13 services. Rust: `env.d1("DB")`; TS: `D1Database`; `db.batch()` in `crates/kit` `rename` | woven (SQL), thin (API) | **SQLite files**. workerd/Miniflare implements D1 on SQLite, and the same `migrations/` apply with `wrangler d1 migrations apply --local`. A native port would need a `Database` port over `rusqlite`/`better-sqlite3`; the SQL is already SQLite, including FTS5. | | |
| 67 | + | | **KV** | `BLOBS`: Actions artifacts and cache (`apps/api/src/blobs.rs`, `apps/web/app/lib/artifacts.server.ts`). `AVATARS`: `services/identity/src/avatars.rs`, `apps/web/workers/app.ts`, `services/og`. `DOMAINS`: `services/deployments/src/domains.ts`, `services/pages` | thin | Miniflare KV on disk (SQLite plus blob files). Natively: a `BlobStore` port on the filesystem or S3/MinIO. | | |
| 68 | + | | **Queues**: the event bus | Producer: `services/events` `BUS.sendBatch` (`lib.rs:67`). The consumer writes the log, then fans out to every binding named `SUBSCRIBER_*` (`lib.rs:161`). Twelve consumers, one queue each. Private job queues in search (`g1t-search-jobs`) and context (`g1t-context-jobs`); consumers branch on the queue name. | woven | Miniflare Queues: in-process and persisted, which works today. Natively: a `Bus` port with a SQLite outbox and a poller per subscriber, or NATS/Redis Streams. At-least-once delivery and idempotent consumers are already the contract. | | |
| 69 | + | | **Durable Objects** | Only `AttemptSandbox` (runner), as the containers library's base class. Uses `ctx.storage.get/put/delete`, `schedule()` (alarm), `idFromName`/`idFromString`, DO RPC (`run`, `destroy`, `noteBlocked`). **Not used:** WebSocket hibernation, raw `alarm()`, `ctx.storage.sql`, `ctx.exports`. | woven, in the runner only | workerd supports Durable Objects (on-disk SQLite). Runner state can move to the sandbox supervisor (phase 2). | | |
| 70 | + | | **Containers** (`@cloudflare/containers`) | `services/runner`: one sandbox per agent run, Actions job and deploy build. `sleepAfter`, `start({ envVars, enableInternet })`, `onStop`. Image: `services/runner/Dockerfile` (node 24, git, toolchains, Claude Code, `g1t-runner`). | woven | **Docker or Podman** through the socket, with the same image. Wrangler can already run Containers locally through Docker; whether that covers outbound interception has to be tested. | | |
| 71 | + | | **Outbound interception** (guardrails egress) | `services/runner/src/guard.ts` (`egress()`), `egress.ts` (`sandboxHosts`, `EGRESS_CA = /etc/cloudflare/certs/cloudflare-containers-ca.crt`), `AttemptSandbox.outboundHandlers`, `interceptHttps = true`, `setOutboundHandler("egress", { hosts })` | woven | The sandbox joins an internal network with no route out, and gets `HTTP(S)_PROXY` pointing at an allow-list proxy that checks the `CONNECT` host. The CA bundle is then not needed, because nothing is re-signed. Blocked hosts are reported to the runner as `noteBlocked` does today. | | |
| 72 | + | | **Artifacts** (git storage) | Only `services/repos/src/store.rs` (`ArtifactsStore`, behind the `GitStore`/`GitRepo` traits). Methods used: `create`, `get`; then `info`, `createToken`, `log`, `readCommit`, `readTree`, `readBlob`, `readFile`, `fork`, and `[Symbol.dispose]`. Everything else (push, fetch, landing, catch-up, import, ref listing) is smart HTTP to `info().remote` with `Bearer <token>`: `land.rs`, `catch_up.rs`, `refs.rs`, `import.rs`, `git_http.rs`. | adapter | **Bare repositories on disk plus `git http-backend`.** Built in phase 1: `deploy/self-host/gitstore`. Forks are local clones with hard links. Tokens are HMAC-signed, scoped, and expire. | | |
| 73 | + | | **Cache API** | `services/repos/src/store.rs` (trees and blobs up to 1 MiB, by hash), `apps/web/workers/app.ts` (avatars), `services/og` | thin, optional | Miniflare's cache, or none. Every use tolerates a miss. | | |
| 74 | + | | **Vectorize** | `services/context/src/index.ts` (`VECTORS.upsert/deleteByIds/query`); optional, guarded by `if (!AI \|\| !VECTORS)` | thin | **sqlite-vec** (default: one file, next to D1), pgvector or Qdrant behind a `VectorIndex` port; or off, which already degrades to keyword search. | | |
| 75 | + | | **Workers AI** | `services/context` only: `@cf/baai/bge-base-en-v1.5` embeddings (768 dims) | thin | An OpenAI-compatible `/v1/embeddings` endpoint (Ollama, vLLM, LM Studio, or a hosted API) behind an `Embedder` port. Changing models means re-embedding (a backfill job already exists). | | |
| 76 | + | | **AI Gateway** | `services/runner/src/model-env.ts:56`, `services/models/src/route.ts:70` (gateway URL, `cf-aig-*` headers), `services/billing/src/keeper.rs` (reads gateway logs to settle) | thin | Optional already: an empty `AI_GATEWAY_ID` goes straight to the provider. Any Anthropic- or OpenAI-compatible base URL works for a workspace's own provider. | | |
| 77 | + | | **Workers for Platforms** | `services/pages` (dispatcher: `env.APPS.get(script).fetch`), `services/deployments/src/cloudflare.ts` (script and asset upload through the REST API) | woven | Phase 1: off. Later: a self-hosted app host in workerd, using the Worker Loader binding to load uploaded scripts, or a workerd per app (see [Deployments](#deployments)). | | |
| 78 | + | | **Cloudflare for SaaS** (custom hostnames) | `services/deployments/src/custom-hostnames.ts` (`/zones/{id}/custom_hostnames`) | thin | Caddy with on-demand TLS, asking g1t whether a hostname is allowed. | | |
| 79 | + | | **Cloudflare REST API** | deployments: script upload, list, delete, assets, GraphQL usage. Billing keeper: AI Gateway logs, `billable-usage`, GraphQL container usage. Ops scripts in `scripts/`. | thin (deployments), woven (keeper pricing) | Deployments: the app-host adapter. Keeper: off when self-hosted, because there is no bill to reconcile. | | |
| 80 | + | | **Email Sending** | `services/identity/src/email.rs` (`EMAIL.send({to, from, subject, text, html})`); callers: verification, password reset, `admin.rs` limit warnings | thin | Built in phase 1: a shim that logs and hands mail to Mailpit, which relays over SMTP. Later: a `Mailer` port with an SMTP adapter. | | |
| 81 | + | | **Cloudflare Access** | `apps/sudo/app/lib/access.ts` (verifies `Cf-Access-Jwt-Assertion` against `/cdn-cgi/access/certs`, `ACCESS_AUD`, `STAFF_EMAILS`) | woven, in sudo only | A local admin flag: `G1T_ADMINS` usernames checked against the normal g1t session. Self-hosters rarely need sudo, which is about billing. | | |
| 82 | + | | **Cron Triggers** | actions (every minute), webhooks (every minute), security (`*/30`), billing (`*/15` and daily), deployments (`*/10`), runner (`*/5`) | thin | workerd runs `scheduled()` when asked. Phase 1 does not yet fire them; phase 2 adds a scheduler that does (see the risks). | | |
| 83 | + | | **`cloudflare:workers` imports** | `apps/web` (`env` in 15 files), `apps/sudo`, `services/runner` (`WorkerEntrypoint`) | thin | Provided by workerd. A Node port would pass `env` through context instead. | | |
| 84 | + | | **Static Assets** | `apps/web` (Vite plugin build), `apps/docs`, `apps/sudo` (`run_worker_first`) | thin | workerd serves them. | | |
| 85 | + | | **`placement`, `observability`, routes, custom domains** | every `wrangler.jsonc` | config only | Dropped by `deploy/self-host/configs.mjs`. | | |
| 86 | + | | **`cf-ray`** | Used as an audit request id, with a fallback: `services/repos/src/run_access.rs:131`, `apps/api/src/audit.rs:37` | thin | Falls back already. | | |
| 87 | + | | **Not used** | R2, Hyperdrive, Workflows, Analytics Engine, Browser Rendering, Images, Turnstile, Secrets Store, `connect()`, HTMLRewriter, `request.cf` | — | — | | |
| 88 | + | ||
| 89 | + | ### By service | |
| 90 | + | ||
| 91 | + | | Service | Runs on | Cloudflare dependencies beyond Workers and D1 | Phase 1 self-hosted | | |
| 92 | + | | --- | --- | --- | --- | | |
| 93 | + | | `apps/web` | TS Worker plus assets | KV (`BLOBS`, `AVATARS`), Cache API, `cloudflare:workers` `env`, RPC to `RUNNER` | Runs unchanged | | |
| 94 | + | | `apps/api` | Rust Worker | KV `BLOBS`; hard-coded `api.g1t.sh`/`mcp.g1t.sh` issuer | Not started yet (phase 2) | | |
| 95 | + | | `apps/sudo` | TS Worker plus assets | Access JWT | Not run | | |
| 96 | + | | `apps/docs` | Static | — | Not run (docs.g1t.sh serves them) | | |
| 97 | + | | `services/identity` | Rust | Email Sending, KV `AVATARS` | Runs unchanged; `EMAIL` goes to the mail shim | | |
| 98 | + | | `services/repos` | Rust | **Artifacts**, Cache API | Runs unchanged; `ARTIFACTS` goes to the git store | | |
| 99 | + | | `services/work` | Rust | Queue consumer | Runs unchanged | | |
| 100 | + | | `services/events` | Rust | Queues (producer and fan-out) | Runs unchanged; the off services' queues are not produced to | | |
| 101 | + | | `services/projects` | TS | Queue consumer | Runs unchanged | | |
| 102 | + | | `services/search` | Rust | Queues (events and its own jobs); FTS5 | Runs unchanged | | |
| 103 | + | | `services/billing` | Rust | Cron, Cloudflare REST API (keeper), Stripe | Runs with `FREE_WHILE_BUILDING=true` and no Stripe key: nothing is charged | | |
| 104 | + | | `services/security` | Rust | Queue, cron | Runs unchanged (cron not fired) | | |
| 105 | + | | `services/actions` | Rust | Queue, cron, `ACTIONS_KEY` | Runs; jobs need the runner, which is off | | |
| 106 | + | | `services/webhooks` | Rust | Queue, cron, `WEBHOOKS_KEY` | Runs; first delivery works, retries need cron | | |
| 107 | + | | `services/integrations` | Rust | Queue, `INTEGRATIONS_KEY` | Runs unchanged | | |
| 108 | + | | `services/deployments` | TS | Workers for Platforms, REST API, KV `DOMAINS`, cron | Runs with no API token: nothing deploys | | |
| 109 | + | | `services/runner` | TS | **Containers**, Durable Objects, outbound interception, AI Gateway, cron | Off: bound to the off Worker | | |
| 110 | + | | `services/context` | TS | **Vectorize**, **Workers AI**, Queues | Off: bound to the off Worker | | |
| 111 | + | | `services/models` | TS | AI Gateway; public at `models.g1t.sh` | Not run (only sandboxes call it) | | |
| 112 | + | | `services/pages` | TS | Dispatch namespace, wildcard routes, KV | Not run | | |
| 113 | + | | `services/og` | TS | Cache API | Not run (social cards are optional) | | |
| 114 | + | | `crates/runner` | native, in the sandbox | Talks to `https://api.g1t.sh` and `https://g1t.sh` (hard-coded in the runner Worker); `wrangler deploy --dry-run` for builds | Phase 2 | | |
| 115 | + | | `crates/sshd` | native | Not deployed; calls `/_internal/ssh/*` endpoints that do not exist yet | Phase 3 | | |
| 116 | + | ||
| 117 | + | ### Hard-coded hosted addresses | |
| 118 | + | ||
| 119 | + | Self-hosting needs one setting, `PUBLIC_URL`, in place of these. Phase 1 | |
| 120 | + | gets around the ones on its path: the mail shim rewrites `https://g1t.sh` | |
| 121 | + | links in mail, and `configs.mjs` rewrites `SITE_URL`/`API_URL`/`SITE` | |
| 122 | + | variables. The rest are listed here so phase 2 can make them settings: | |
| 123 | + | ||
| 124 | + | - `apps/web/app/lib/meta.ts` (`SITE`, `OG`); `clone-box.tsx` (clone URL, | |
| 125 | + | `mcp.g1t.sh`); `workers/app.ts` (`DOCS`). | |
| 126 | + | - `apps/api/src/lib.rs` (`API`); `oauth.rs` (issuer, MCP resource). | |
| 127 | + | - `services/identity/src/email.rs` (`SITE`, `FROM`). | |
| 128 | + | - `services/runner/src/index.ts` (`G1T_API`, `GIT_REMOTE` and the other | |
| 129 | + | remotes handed to sandboxes, in 12 places). | |
| 130 | + | - `services/billing` (`stripe.rs`, `accounts.rs`, `limits.rs`). | |
| 131 | + | - `crates/sshd` (`G1T_API` default). | |
| 132 | + | ||
| 133 | + | There are about 150 occurrences of `g1t.sh` in non-test code. Most are | |
| 134 | + | docs links and copy, and need no change. | |
| 135 | + | ||
| 136 | + | ## 2. Design | |
| 137 | + | ||
| 138 | + | ### Principles | |
| 139 | + | ||
| 140 | + | 1. **Hosted is the reference.** Hosted code does not change behaviour to | |
| 141 | + | make room for self-hosting. A self-hosted adapter is added beside the | |
| 142 | + | hosted one, and the hosted one stays the default. | |
| 143 | + | 2. **Swap at the narrowest seam that exists.** A binding-shaped seam (a | |
| 144 | + | Worker that offers the same methods as the Cloudflare binding) needs no | |
| 145 | + | code change, and is how phase 1 works. A code-level port (a trait or | |
| 146 | + | interface with two adapters) is cleaner and testable, and is the long-term | |
| 147 | + | shape. Each primitive moves from the first to the second when it is next | |
| 148 | + | touched. | |
| 149 | + | 3. **One runtime, two hosts.** The Workers stay Workers. workerd runs them | |
| 150 | + | self-hosted, so one build serves both and there is no second code path | |
| 151 | + | to keep correct. | |
| 152 | + | 4. **Off is a real mode.** Every optional subsystem (agents, context, | |
| 153 | + | deployments, billing) has an "off" answer that pages already handle. | |
| 154 | + | ||
| 155 | + | ### The ports | |
| 156 | + | ||
| 157 | + | | Port | Hosted adapter | Self-hosted adapter | Lives in | Status | | |
| 158 | + | | --- | --- | --- | --- | --- | | |
| 159 | + | | `GitStore` / `GitRepo` | `ArtifactsStore` | Git store (bare repos, `git http-backend`) through an Artifacts-compatible shim; later a `LocalGitStore` that calls the git store's HTTP API from Rust directly | `services/repos/src/store.rs` (exists) | **Built** (binding level) | | |
| 160 | + | | `Mailer` | Email Sending binding | SMTP (through Mailpit relay now; a direct SMTP adapter later) | `g1t_kit::mail` | **Built** (binding level) | | |
| 161 | + | | `Sandbox` | `AttemptSandbox` (Containers, DO) | `DockerSandbox`: a supervisor that starts the runner image through the Docker API | `services/runner` (TS) | Phase 2 | | |
| 162 | + | | `Egress` | Containers outbound handler | Allow-list HTTP(S) proxy on an internal network | `services/runner` | Phase 2 | | |
| 163 | + | | `ModelRoute` | AI Gateway, or direct | Any Anthropic/OpenAI-compatible base URL | `services/models`, `runner/model-env.ts` | Exists (env-switchable) | | |
| 164 | + | | `Embedder` | Workers AI | OpenAI-compatible `/v1/embeddings` | `services/context` | Phase 3 | | |
| 165 | + | | `VectorIndex` | Vectorize | sqlite-vec / pgvector / Qdrant | `services/context` | Phase 3 | | |
| 166 | + | | `AppHost` | Workers for Platforms plus REST API | workerd app host (Worker Loader) | `services/deployments` | Phase 3 | | |
| 167 | + | | `Domains` | Cloudflare for SaaS | Caddy on-demand TLS | `services/deployments` | Phase 3 | | |
| 168 | + | | `AdminAuth` | Cloudflare Access | `G1T_ADMINS` plus the normal session | `apps/sudo` | Phase 4 | | |
| 169 | + | | `Bus` | Queues | Miniflare Queues now; SQLite outbox later | `services/events`, `g1t_kit` | Works (runtime) | | |
| 170 | + | | `Database` | D1 | SQLite files through workerd | — | Works (runtime) | | |
| 171 | + | | `Blobs` | KV | Miniflare KV now; filesystem/S3 later | — | Works (runtime) | | |
| 172 | + | | `Scheduler` | Cron Triggers | A ticker that calls `scheduled()` | `deploy/self-host` | Phase 2 | | |
| 173 | + | | `UsageKeeper` | Cloudflare bill plus AI Gateway logs | None (billing off) | `services/billing` | Off | | |
| 174 | + | ||
| 175 | + | For Rust, the code-level ports go in `crates/kit` as traits (`g1t_kit::ports`), | |
| 176 | + | with the Cloudflare adapters next to them. For TypeScript, they go in a new | |
| 177 | + | `packages/platform` package, with each adapter in its own module so a | |
| 178 | + | hosted bundle never pulls in a self-hosted adapter. The adapter is chosen at | |
| 179 | + | startup from one setting, `G1T_MODE=hosted|self`, never per request. | |
| 180 | + | ||
| 181 | + | ### The runtime: workerd, or native binaries | |
| 182 | + | ||
| 183 | + | | | **workerd (phase 1: `wrangler dev`; later a plain workerd config)** | **Native: Rust on axum/hyper, TS on Node** | | |
| 184 | + | | --- | --- | --- | | |
| 185 | + | | Effort to first boot | **Done.** About a day, almost all of it shims and config. | Weeks. The Rust services use `worker::*` throughout: `Env`, `Fetcher`, `D1Database`, the `#[event]` macros and `js_sys` interop. Each needs an abstraction layer before it compiles natively. TS needs an `env` provider in place of `cloudflare:workers`, plus Queues, D1 and KV clients. | | |
| 186 | + | | Fidelity to hosted | **The same runtime and the same bundles.** A bug self-hosted is a bug hosted. | A second implementation of every platform API, with its own bugs. | | |
| 187 | + | | Performance | Good for one node: WebAssembly in V8, SQLite on local disk. Single-threaded per isolate; plenty for a team. | Better per core, and multi-threaded. That matters only at a scale where people use g1t.sh. | | |
| 188 | + | | Maintenance | Low. New hosted features run self-hosted for free, unless they add a new Cloudflare-only binding. The config generator then drops it or binds a stand-in. | High. Every feature is built twice, or behind a port that both sides keep honest. | | |
| 189 | + | | Scale-out | One process, one disk. D1-on-SQLite is single-writer. | Could use Postgres and many processes. | | |
| 190 | + | | Operational risk | `wrangler dev` is a development tool (see risks). Moving to `workerd serve` with a generated config, or a small Miniflare-API launcher, removes the dev-tool surface. | Conventional. | | |
| 191 | + | ||
| 192 | + | **Recommendation.** | |
| 193 | + | ||
| 194 | + | - **Phase 1:** workerd under `wrangler dev`, as built in `deploy/self-host/`. | |
| 195 | + | It needs no change to any service and is proven end to end. | |
| 196 | + | - **Phase 2:** replace `wrangler dev` with a launcher that drives Miniflare's | |
| 197 | + | API directly, or a generated `workerd` config. It should expose only the | |
| 198 | + | site and the API, with no dev endpoints, run cron, and run as a proper | |
| 199 | + | service. | |
| 200 | + | - **Long term:** keep workerd as the runtime and push the remaining | |
| 201 | + | Cloudflare-only bindings behind code-level ports. Compile to native only | |
| 202 | + | what already is native (the runner binary, sshd), or a service where | |
| 203 | + | workerd is a real limit, case by case. | |
| 204 | + | ||
| 205 | + | A full native port is not worth it. It would double the maintenance of | |
| 206 | + | every feature, which is the opposite of what makes self-hosting | |
| 207 | + | sustainable for a small team. | |
| 208 | + | ||
| 209 | + | ### Sandboxes (phase 2) | |
| 210 | + | ||
| 211 | + | - **Engine.** Docker or Podman through the socket, mounted into a small | |
| 212 | + | supervisor (`g1t-sandboxd`), not into the workerd container. The | |
| 213 | + | supervisor exposes the runner's `Sandbox` port over HTTP: start with | |
| 214 | + | env, stop, status, and the exit report that becomes `onStop`. It uses | |
| 215 | + | the **same image** (`services/runner/Dockerfile`). | |
| 216 | + | - **Runner.** The runner Worker runs in workerd with `AttemptSandbox` | |
| 217 | + | replaced by a `DockerSandbox` adapter. Run state that lives in | |
| 218 | + | `ctx.storage` moves to the supervisor's SQLite; `schedule()`/`timeUp` | |
| 219 | + | becomes a supervisor timer. | |
| 220 | + | - **Egress guardrails.** Each restricted sandbox joins an internal Docker | |
| 221 | + | network with no default route. Its `HTTP_PROXY`/`HTTPS_PROXY` point at an | |
| 222 | + | allow-list proxy (for example a small Go or Node `CONNECT` proxy) that | |
| 223 | + | admits the hosts `sandboxHosts()` computes and reports refusals. | |
| 224 | + | Nothing is intercepted, so no CA is injected. `EGRESS=off` puts the | |
| 225 | + | sandbox on a normal network. | |
| 226 | + | - **Addresses.** `G1T_API`, `GIT_REMOTE` and the rest become | |
| 227 | + | `PUBLIC_URL`-derived settings. Sandboxes reach the site and API on the | |
| 228 | + | compose network. | |
| 229 | + | - **Shortcut to evaluate first.** Wrangler can already run Containers | |
| 230 | + | classes locally through Docker. If its local Containers support | |
| 231 | + | `setOutboundHandler`, the runner could run nearly unchanged. Test this | |
| 232 | + | before building the supervisor. | |
| 233 | + | ||
| 234 | + | ### Git storage | |
| 235 | + | ||
| 236 | + | Artifacts gives g1t: named repositories, scoped short-lived tokens, a smart | |
| 237 | + | HTTP remote, typed reads (commit, tree, blob, file, log), copy-on-write | |
| 238 | + | forks, and jurisdictions. g1t uses everything except jurisdictions. | |
| 239 | + | ||
| 240 | + | The self-hosted git store (`deploy/self-host/gitstore/server.mjs`, about 450 | |
| 241 | + | lines of Node with no dependencies) provides the same: | |
| 242 | + | ||
| 243 | + | | Artifacts | Git store | | |
| 244 | + | | --- | --- | | |
| 245 | + | | `create(name, { setDefaultBranch, description })` | `git init --bare --initial-branch`, plus `g1t.json` beside it for metadata | | |
| 246 | + | | `get(name)` then `info()` | Metadata, `HEAD`, and the last push time; `remote` is `GITSTORE_URL/git/<name>.git` | | |
| 247 | + | | `createToken(scope, ttl)` | HMAC-SHA256 over `{ key, scope, expiry }` with the shared secret | | |
| 248 | + | | Smart HTTP remote | `git http-backend`; a read token cannot push | | |
| 249 | + | | `log`, `readCommit`, `readTree`, `readBlob`, `readFile` | `git rev-list --first-parent`, `cat-file`, `ls-tree` | | |
| 250 | + | | `fork(name, { defaultBranchOnly })` | `git clone --bare [--single-branch]` with hard-linked objects | | |
| 251 | + | ||
| 252 | + | Hosted is untouched: `ArtifactsStore` is still the only adapter compiled | |
| 253 | + | into `services/repos`. Self-hosted, the `ARTIFACTS` binding is a service | |
| 254 | + | binding to `deploy/self-host/workers/artifacts`, which offers Artifacts' | |
| 255 | + | methods and calls the git store. Long term, a `LocalGitStore` adapter in | |
| 256 | + | Rust should call the git store's API directly, which removes the shim. The | |
| 257 | + | git store can later gain `git gc` scheduling and object-store-backed packs | |
| 258 | + | for large installations. | |
| 259 | + | ||
| 260 | + | ### Search and context | |
| 261 | + | ||
| 262 | + | - **Site search** (`services/search`) is D1 FTS5. It works self-hosted as | |
| 263 | + | is (verified: `/search?q=hello` answers 200). | |
| 264 | + | - **Context hub** (`services/context`) needs an `Embedder` and a | |
| 265 | + | `VectorIndex`: | |
| 266 | + | - Default: **sqlite-vec** in the context service's own SQLite, with | |
| 267 | + | embeddings from an **OpenAI-compatible endpoint**. Ollama's | |
| 268 | + | `nomic-embed-text` has the same 768 dimensions as today's | |
| 269 | + | `bge-base-en-v1.5`. | |
| 270 | + | - Alternatives: pgvector or Qdrant, for installations that already run | |
| 271 | + | them. | |
| 272 | + | - Off: the service already degrades to keyword search when `AI` or | |
| 273 | + | `VECTORS` is missing (`index.ts:919`), so phase 3 can first run context | |
| 274 | + | with neither bound. | |
| 275 | + | ||
| 276 | + | ### Models | |
| 277 | + | ||
| 278 | + | Already portable. The model proxy (`services/models`) and the runner's | |
| 279 | + | model environment take any Anthropic-compatible base URL, and an empty | |
| 280 | + | `AI_GATEWAY_ID` skips AI Gateway. Self-hosted: | |
| 281 | + | ||
| 282 | + | - a workspace connects its own provider under Integrations (Anthropic, or | |
| 283 | + | any Anthropic-compatible endpoint, including a local gateway in front of | |
| 284 | + | OpenAI-compatible models); | |
| 285 | + | - "hosted models" are off, because there is no g1t key to spend. | |
| 286 | + | ||
| 287 | + | ### Deployments | |
| 288 | + | ||
| 289 | + | - **Phase 1–2: off.** `services/deployments` runs, but with no | |
| 290 | + | `CLOUDFLARE_API_TOKEN` nothing deploys. Its pages and settings still | |
| 291 | + | render. | |
| 292 | + | - **Phase 3: an app host on workerd.** The self-hosted `pages` dispatcher | |
| 293 | + | uses workerd's **Worker Loader** binding to load each uploaded app's | |
| 294 | + | modules on demand, keyed by deployment id, from the blob store. Static | |
| 295 | + | assets are served from the same store. Builds run in Docker sandboxes | |
| 296 | + | (phase 2), which already bundle with `wrangler deploy --dry-run`. | |
| 297 | + | Wildcard hosts (`*.apps.example.com`) and custom domains go through | |
| 298 | + | Caddy with on-demand TLS. The alternative, one workerd process per app, | |
| 299 | + | is simpler to isolate but heavier. | |
| 300 | + | ||
| 301 | + | ### Billing and the feature map | |
| 302 | + | ||
| 303 | + | Self-hosted billing is **off by default**: no Stripe, no usage limits, | |
| 304 | + | no keeper. Billing still runs, because the shell reads `billing.account` | |
| 305 | + | on every page, but with `FREE_WHILE_BUILDING=true` and no Stripe key. | |
| 306 | + | ||
| 307 | + | | Feature | Hosted (g1t.sh) | Self-hosted default | Self-hosted, when turned on | | |
| 308 | + | | --- | --- | --- | --- | | |
| 309 | + | | Accounts, workspaces, repos, git over HTTP | On | On | — | | |
| 310 | + | | Issues, pull requests, review, merge queue | On | On | — | | |
| 311 | + | | Site search (FTS5) | On | On | — | | |
| 312 | + | | Email | Email Sending | Mailpit, logged | SMTP relay | | |
| 313 | + | | Webhooks, integrations | On | On (no scheduled retries yet) | — | | |
| 314 | + | | g1t agents | On | Off | Phase 2: Docker sandboxes plus your own model provider | | |
| 315 | + | | Guardrails egress | Containers interception | n/a | Phase 2: allow-list proxy | | |
| 316 | + | | Hosted models (g1t's key) | On (billed) | Off | Never: bring your own | | |
| 317 | + | | Context hub semantic search | Vectorize plus Workers AI | Off | Phase 3: sqlite-vec plus an OpenAI-compatible embedder | | |
| 318 | + | | Deployments | Workers for Platforms | Off | Phase 3: workerd app host | | |
| 319 | + | | Custom domains | Cloudflare for SaaS | Off | Phase 3: Caddy on-demand TLS | | |
| 320 | + | | Billing, limits, Stripe, keeper | On | Off | Not planned | | |
| 321 | + | | sudo (staff console) | Access | Off | Phase 4: `G1T_ADMINS` | | |
| 322 | + | | Git over SSH | Not yet | Off | Phase 3 (`crates/sshd`, which is native already) | | |
| 323 | + | | REST API, MCP, CLI | On | Off | Phase 2 | | |
| 324 | + | ||
| 325 | + | ### Auth, Access and email | |
| 326 | + | ||
| 327 | + | - `apps/sudo` checks Cloudflare Access. Self-hosted, it should check a | |
| 328 | + | normal g1t session against `G1T_ADMINS`. That needs an `AdminAuth` port in | |
| 329 | + | `workers/app.ts` with two adapters. It is phase 4, because sudo is about | |
| 330 | + | billing. | |
| 331 | + | - Sessions are random tokens stored hashed in D1, with no signing key, so | |
| 332 | + | nothing to configure. The cookie is `Secure`: fine on `localhost`, but any | |
| 333 | + | other address needs HTTPS. The compose file should gain an optional Caddy | |
| 334 | + | service in phase 2. | |
| 335 | + | - **Email verification is required** before creating anything, and there is | |
| 336 | + | no bypass in code. That is why the proof ships a working mail path | |
| 337 | + | (Mailpit) rather than "email off". An admin "mark verified" or | |
| 338 | + | `G1T_SKIP_EMAIL_VERIFICATION` belongs with `G1T_ADMINS`. | |
| 339 | + | ||
| 340 | + | ### Configuration, upgrades and backups | |
| 341 | + | ||
| 342 | + | - **Today:** environment variables in the compose file (`PUBLIC_URL`, | |
| 343 | + | `G1T_PORT`, `MAIL_URL`, `MAIL_FROM`). Keys (`ACTIONS_KEY`, | |
| 344 | + | `INTEGRATIONS_KEY`, `WEBHOOKS_KEY`, the git store secret) are generated on | |
| 345 | + | first start and kept on volumes. | |
| 346 | + | - **Phase 2:** one `g1t.toml`, read by the launcher and turned into | |
| 347 | + | bindings and variables: | |
| 348 | + | ||
| 349 | + | ```toml | |
| 350 | + | public_url = "https://git.example.com" | |
| 351 | + | ||
| 352 | + | [mail] | |
| 353 | + | smtp = "smtp://user:pass@smtp.example.com:587" | |
| 354 | + | from = "g1t <git@example.com>" | |
| 355 | + | ||
| 356 | + | [agents] # off when absent | |
| 357 | + | docker = "unix:///var/run/docker.sock" | |
| 358 | + | egress = "enforce" | |
| 359 | + | ||
| 360 | + | [context] # off when absent | |
| 361 | + | embeddings = "http://ollama:11434/v1" | |
| 362 | + | model = "nomic-embed-text" | |
| 363 | + | ||
| 364 | + | [admins] | |
| 365 | + | usernames = ["alice"] | |
| 366 | + | ``` | |
| 367 | + | ||
| 368 | + | - **Upgrades.** The container applies every service's D1 migrations to its | |
| 369 | + | SQLite file on start (`wrangler d1 migrations apply --local`). Applied | |
| 370 | + | migrations are recorded in `d1_migrations`, so this is idempotent. Hosted | |
| 371 | + | and self-hosted run the same migration files, which keeps them | |
| 372 | + | forward-compatible. Rule to keep: migrations stay additive, or come with | |
| 373 | + | a backfill a self-hoster's start can run. | |
| 374 | + | - **Backups.** Phase 1: stop, then tar the `g1t-data` and `g1t-git` | |
| 375 | + | volumes (documented in the guide). Phase 2: online backups with | |
| 376 | + | `sqlite3 .backup` per database and `git bundle` or rsync of the bare | |
| 377 | + | repositories, or Litestream for continuous replication. | |
| 378 | + | ||
| 379 | + | ## 3. Phase 1: what works today | |
| 380 | + | ||
| 381 | + | Everything is in `deploy/self-host/`: | |
| 382 | + | ||
| 383 | + | | File | What it is | | |
| 384 | + | | --- | --- | | |
| 385 | + | | `docker-compose.yml` | Three services: `g1t` (every core Worker in one workerd), `gitstore` (bare repositories), and `mailpit` (mail). Volumes: `g1t-data`, `g1t-git`, `g1t-secrets`. | | |
| 386 | + | | `Dockerfile` | Compiles the ten Rust services to WebAssembly with `worker-build`, as hosted does. Builds the site with React Router. The runtime image has Node, Wrangler, workerd and the built Workers. | | |
| 387 | + | | `Dockerfile.dockerignore` | Build-context rules for this image only (the root `.dockerignore` leaves out the site). | | |
| 388 | + | | `start.sh` | Makes the sealing keys once, writes the configs, applies migrations, and runs `wrangler dev` with every config on `0.0.0.0:8787`, persisting to `/data/state`. | | |
| 389 | + | | `configs.mjs` | Derives each self-hosted Wrangler config from the hosted `wrangler.jsonc`. It drops routes, account and placement, rebinds `ARTIFACTS`/`EMAIL` and the off services, and rewrites hosted URLs. Derived, so it cannot drift. | | |
| 390 | + | | `gitstore/server.mjs`, `gitstore/Dockerfile` | The git store. | | |
| 391 | + | | `workers/artifacts/index.js` | The `ARTIFACTS` binding, implemented against the git store. | | |
| 392 | + | | `workers/mail/index.js` | The `EMAIL` binding: logs, then sends to Mailpit. | | |
| 393 | + | | `workers/off/index.js` | The runner and the context hub when they are off. | | |
| 394 | + | | `smoke.sh` | The end-to-end check. | | |
| 395 | + | ||
| 396 | + | Workers running: the site; identity, repos, work, events, projects, search, | |
| 397 | + | billing, security, actions, webhooks, integrations and deployments; and the | |
| 398 | + | artifacts, mail and two off stand-ins. | |
| 399 | + | ||
| 400 | + | ### Verified | |
| 401 | + | ||
| 402 | + | On this machine (Windows 11, Docker Desktop 29.8, engine on Linux): | |
| 403 | + | ||
| 404 | + | 1. **Without Docker, with local processes.** The git store ran under Node | |
| 405 | + | on Windows and every Worker ran under `wrangler dev`, using the configs | |
| 406 | + | from `configs.mjs` and the existing Rust builds. `smoke.sh` passed every | |
| 407 | + | step: sign-up, confirmation through the logged link, workspace, repo, | |
| 408 | + | `git push` and `git clone` over HTTP, issue, and the code, blob and | |
| 409 | + | commits pages. Seventeen other pages answered 200: home, workspace, | |
| 410 | + | repo overview, issues, pulls, settings, agents, actions, deployments, | |
| 411 | + | security, people, usage, explore, search, account settings and tree. | |
| 412 | + | The workspace context page answered 403 from the off stand-in, as | |
| 413 | + | intended. | |
| 414 | + | 2. **With Docker Compose.** See the [Docker run](#docker-run) section below. | |
| 415 | + | ||
| 416 | + | ### Not verified, or not working yet | |
| 417 | + | ||
| 418 | + | - Pull requests between branches and forks, the merge queue and catch-up. | |
| 419 | + | They use `fork` and smart-HTTP pushes, which the git store implements, | |
| 420 | + | but they have not been exercised end to end. | |
| 421 | + | - Cron work: webhook retries, Actions schedules, security sweeps. | |
| 422 | + | - The REST API, MCP, the CLI, and git over SSH. | |
| 423 | + | - Anything on an address other than `localhost` without HTTPS (the | |
| 424 | + | session cookie is `Secure`). | |
| 425 | + | - Restart durability beyond one restart, upgrades across schema changes, | |
| 426 | + | and backup and restore. | |
| 427 | + | ||
| 428 | + | ## 4. Phased plan | |
| 429 | + | ||
| 430 | + | Estimates are for one engineer who knows the codebase, working with | |
| 431 | + | agents. They include docs and tests. | |
| 432 | + | ||
| 433 | + | | Phase | Scope | Estimate | | |
| 434 | + | | --- | --- | --- | | |
| 435 | + | | **1. Core forge** | **Done in this change:** compose stack, git store, Artifacts/Email shims, off stand-ins, config generator, smoke test, guide. **Left:** run the API worker (REST, MCP, OAuth) on its own port with `PUBLIC_URL` issuer; make `PUBLIC_URL` a setting in identity mail, `meta.ts`, `clone-box.tsx` and the API; fire cron (a ticker calling each Worker's `scheduled`); exercise pull requests and the merge queue in `smoke.sh`; optional Caddy for HTTPS; CI job that builds the images and runs `smoke.sh`. | 1–1.5 weeks left | | |
| 436 | + | | **2. Agents with Docker sandboxes** | Test Wrangler's local Containers first. Otherwise: `g1t-sandboxd` supervisor, `DockerSandbox` adapter in the runner, egress allow-list proxy and internal network, runner addresses from `PUBLIC_URL`, models through a workspace's own provider, `g1t.toml` and a launcher that replaces `wrangler dev`. | 2–3 weeks | | |
| 437 | + | | **3. Search, context and deployments** | `Embedder` (OpenAI-compatible) and `VectorIndex` (sqlite-vec first) ports in context; app host on workerd with Worker Loader; Caddy on-demand TLS for app and custom domains; git over SSH through `crates/sshd` plus the missing `/_internal/ssh/*` endpoints. | 3–4 weeks | | |
| 438 | + | | **4. Parity and upgrade path** | Code-level ports in `g1t_kit` / `@g1t/platform` replacing the binding shims (`LocalGitStore` in Rust, `Mailer` with SMTP); `AdminAuth` for sudo; online backups (Litestream or `.backup`); versioned releases with published images; an upgrade test in CI that migrates a snapshot of the previous release; a self-host column in the docs for every feature. | 3–4 weeks | | |
| 439 | + | ||
| 440 | + | Total to parity: about 10–13 weeks. Phase 1 alone is already a credible | |
| 441 | + | "run it yourself" for the core forge. | |
| 442 | + | ||
| 443 | + | ## 5. Risks | |
| 444 | + | ||
| 445 | + | - **`wrangler dev` is a development tool.** It exposes Miniflare's dev | |
| 446 | + | endpoints (`/cdn-cgi/...`, including a local data explorer) on the same | |
| 447 | + | port as the site. Treat phase 1 as **localhost or a trusted private | |
| 448 | + | network only** until the launcher in phase 2 replaces it. Its flags and | |
| 449 | + | behaviour can also change between Wrangler releases. Pin the Wrangler | |
| 450 | + | version, as the lockfile already does. | |
| 451 | + | - **No cron yet.** Webhook retries, Actions schedules, security sweeps and | |
| 452 | + | deployments' sweeps do not run. Anything that relies on a sweep to | |
| 453 | + | recover from a missed event stays stuck until phase 2 adds a ticker. | |
| 454 | + | - **New Cloudflare-only bindings break self-hosting silently.** | |
| 455 | + | `configs.mjs` passes unknown keys through untouched. A new binding type | |
| 456 | + | could make `wrangler dev` reach for a remote resource (for example | |
| 457 | + | `remote: true`, AI or Vectorize) and prompt for a login. Mitigation: CI | |
| 458 | + | that builds the compose stack and runs `smoke.sh` on every change, and an | |
| 459 | + | allow-list in `configs.mjs` that fails on unknown binding types. | |
| 460 | + | - **Artifacts semantics drift.** The shim copies the Artifacts methods g1t | |
| 461 | + | uses today. If repos starts using another method (`import`, | |
| 462 | + | `listTokens`, `revokeToken`), self-hosted fails at runtime. Mitigation: a | |
| 463 | + | contract test that runs `services/repos` against the git store, and the | |
| 464 | + | long-term `LocalGitStore` port. | |
| 465 | + | - **Error codes over RPC.** `ArtifactsStore::create` and `fork` tolerate | |
| 466 | + | `ALREADY_EXISTS` by reading the thrown error's `code`. Workers RPC may not | |
| 467 | + | carry custom error properties across a service binding. If it does not, | |
| 468 | + | retrying a half-finished create fails self-hosted where it would succeed | |
| 469 | + | hosted. This was not seen in testing, because creates were never retried. | |
| 470 | + | - **Single node, single writer.** SQLite (D1 local) and one workerd | |
| 471 | + | process suit a team, not a large organisation. Scaling out means | |
| 472 | + | Postgres behind a `Database` port, which is a large change and is not | |
| 473 | + | planned. | |
| 474 | + | - **The `Secure` cookie.** It needs HTTPS anywhere but `localhost`. A LAN | |
| 475 | + | install over plain HTTP cannot sign in. | |
| 476 | + | - **Building from a working tree that is mid-change.** The image compiles | |
| 477 | + | every Rust service, including ones other work is changing. A service that | |
| 478 | + | does not compile breaks the whole image. Released images (phase 4) fix | |
| 479 | + | this. | |
| 480 | + | - **Image size and build time.** The first build compiles ten Rust crates | |
| 481 | + | to WebAssembly and installs the site's dependencies. Expect minutes and | |
| 482 | + | several GB. Published images remove this for users. | |
| 483 | + | - **Hard-coded hosted URLs.** About a dozen code paths name `g1t.sh` or | |
| 484 | + | `api.g1t.sh`. Until they read `PUBLIC_URL`, some links and redirects | |
| 485 | + | (OG images, the docs link, the clone box's MCP line) point at the hosted | |
| 486 | + | service. |