architecture and such
31 files+3125−680/31 viewed
| 1 | 1 | # Generated account passwords (services/identity/scripts/create-user.mjs) | |
| 2 | 2 | /.credentials/ | |
| 3 | + | /.secrets/ | |
| 3 | 4 | ||
| 4 | 5 | node_modules/ | |
| 5 | 6 | target/ | |
| ⋯ | |||
| 14 | 15 | wrangler.deploy.json | |
| 15 | 16 | ||
| 16 | 17 | # Personal notes for the next session | |
| 17 | − | /TOMORROW.md | |
| 18 | + | /SCRUTINY.md | |
| 18 | 19 | /TODO.md | |
| 17 | 17 | | How agents behave | `guides/working-with-g1t.md`, and `apps/web/public/llms.txt` | | |
| 18 | 18 | | Settings, limits or prices | The page that names them, and the table it is in | | |
| 19 | 19 | | A new feature | A section in the guide that owns it, linked from the docs home if it is a new task | | |
| 20 | + | | How g1t is put together: a service, a binding, a store, an event route, the path of a request | Its page in the docs' Architecture section (`apps/docs/src/content/docs/architecture/`), and its diagram in `apps/docs/src/diagrams/architecture.ts`. [How g1t ships](https://docs.g1t.sh/architecture/shipping/#keeping-this-section-true) says which page covers what. | | |
| 20 | 21 | ||
| 21 | 22 | The style, in short: plain sentences, second person, no marketing words, | |
| 22 | 23 | sentence-case headings, a numbered list for steps, a table for options, and | |
| ⋯ | |||
| 112 | 113 | Pushes to `main` deploy themselves: `.g1t/workflows/deploy.yml` runs | |
| 113 | 114 | `scripts/deploy.mjs`, which deploys only the parts that changed, migrations | |
| 114 | 115 | first. Every deployable part is listed in `deploy/stack.jsonc`; a new service | |
| 115 | − | or app goes there, and in the table on | |
| 116 | − | [How a self-hosted g1t runs](https://docs.g1t.sh/guides/self-hosting-architecture/#each-part) | |
| 116 | + | or app goes there, in the table on | |
| 117 | + | [How a self-hosted g1t runs](https://docs.g1t.sh/guides/self-hosting-architecture/#each-part), | |
| 118 | + | and in the one on [Services and contracts](https://docs.g1t.sh/architecture/services/#every-unit) | |
| 117 | 119 | (`npm run test:deploy` says what is missing). | |
| 118 | 120 | [Deploy g1t to Cloudflare](https://docs.g1t.sh/guides/deploy-to-cloudflare/) | |
| 119 | 121 | covers the tool, the workflow, rollbacks, adding a unit and first-time | |
| 210 | 210 | ], | |
| 211 | 211 | }, | |
| 212 | 212 | { | |
| 213 | + | label: 'Architecture', | |
| 214 | + | items: [ | |
| 215 | + | { label: 'Overview', slug: 'architecture/overview' }, | |
| 216 | + | { label: 'How a request is served', slug: 'architecture/requests' }, | |
| 217 | + | { label: 'Services and contracts', slug: 'architecture/services' }, | |
| 218 | + | { label: 'Git', slug: 'architecture/git' }, | |
| 219 | + | { label: 'Events', slug: 'architecture/events' }, | |
| 220 | + | { label: 'Work and the merge queue', slug: 'architecture/work' }, | |
| 221 | + | { label: 'Actions, runners and sandboxes', slug: 'architecture/compute' }, | |
| 222 | + | { label: 'Deployments and telemetry', slug: 'architecture/deployments' }, | |
| 223 | + | { label: 'Data and storage', slug: 'architecture/data' }, | |
| 224 | + | { label: 'Identity and access', slug: 'architecture/identity' }, | |
| 225 | + | { label: 'How g1t ships', slug: 'architecture/shipping' }, | |
| 226 | + | ], | |
| 227 | + | }, | |
| 228 | + | { | |
| 213 | 229 | label: 'Reference', | |
| 214 | 230 | items: [ | |
| 215 | 231 | { label: 'API overview', slug: 'reference/api' }, |
| 6 | 6 | "dev": "astro dev", | |
| 7 | 7 | "build": "astro build", | |
| 8 | 8 | "preview": "astro preview", | |
| 9 | − | "deploy": "astro build && wrangler deploy" | |
| 9 | + | "deploy": "astro build && wrangler deploy", | |
| 10 | + | "test": "node --test \"src/diagrams/*.test.ts\"" | |
| 10 | 11 | }, | |
| 11 | 12 | "dependencies": { | |
| 12 | 13 | "@astrojs/starlight": "^0.42.5", |
| 1 | + | --- | |
| 2 | + | // A diagram from src/diagrams, by name: drawn as an inline SVG in the g1t | |
| 3 | + | // tokens, with its caption, a key, and Expand, which opens it at full size | |
| 4 | + | // in a dialog. See src/diagrams/engine.ts. | |
| 5 | + | import { DIAGRAMS } from '../diagrams/index.ts'; | |
| 6 | + | import { render } from '../diagrams/engine.ts'; | |
| 7 | + | import LucideIcon from './LucideIcon.astro'; | |
| 8 | + | ||
| 9 | + | interface Props { | |
| 10 | + | name: string; | |
| 11 | + | } | |
| 12 | + | ||
| 13 | + | const { name } = Astro.props; | |
| 14 | + | const spec = DIAGRAMS[name]; | |
| 15 | + | if (!spec) throw new Error(`No diagram named "${name}". Add it to src/diagrams/index.ts.`); | |
| 16 | + | // The dialog's copy has its own ids, so its arrowheads and title never | |
| 17 | + | // point at the page's copy. | |
| 18 | + | const svg = render(`d-${name}`, spec); | |
| 19 | + | const large = render(`d-${name}-full`, spec); | |
| 20 | + | const asyncUsed = spec.kind === 'graph' ? spec.edges.some((e) => e.async) : spec.steps.some((s) => 'from' in s && (s.async || s.reply)); | |
| 21 | + | --- | |
| 22 | + | ||
| 23 | + | <figure class="g1t-diagram not-content" data-diagram> | |
| 24 | + | <div class="g1t-diagram__scroll" tabindex="0" role="group" aria-label={`${spec.caption}. Scrolls sideways on a narrow screen.`}> | |
| 25 | + | <Fragment set:html={svg} /> | |
| 26 | + | </div> | |
| 27 | + | <figcaption> | |
| 28 | + | <span>{spec.caption}</span> | |
| 29 | + | <span class="g1t-diagram__tools"> | |
| 30 | + | <span class="g1t-diagram__key" aria-hidden="true"> | |
| 31 | + | <span><svg width="22" height="8"><line x1="1" y1="4" x2="21" y2="4" class="d-key" /></svg> | |
| 32 | + | {spec.kind === 'sequence' ? 'call' : 'calls'}</span> | |
| 33 | + | { | |
| 34 | + | asyncUsed && ( | |
| 35 | + | <span><svg width="22" height="8"><line x1="1" y1="4" x2="21" y2="4" class="d-key d-async" /></svg> | |
| 36 | + | {spec.kind === 'sequence' ? 'answer or event' : 'event, queue or stream'}</span> | |
| 37 | + | ) | |
| 38 | + | } | |
| 39 | + | </span> | |
| 40 | + | <button type="button" class="g1t-diagram__expand" data-diagram-open aria-haspopup="dialog"> | |
| 41 | + | <LucideIcon name="maximize-2" size="0.875rem" /> | |
| 42 | + | Expand | |
| 43 | + | </button> | |
| 44 | + | </span> | |
| 45 | + | </figcaption> | |
| 46 | + | <dialog class="g1t-diagram__dialog" aria-label={spec.caption}> | |
| 47 | + | <div class="g1t-diagram__dialog-bar"> | |
| 48 | + | <span>{spec.caption}</span> | |
| 49 | + | <button type="button" class="g1t-diagram__expand" data-diagram-close> | |
| 50 | + | <LucideIcon name="x" size="0.875rem" /> | |
| 51 | + | Close | |
| 52 | + | </button> | |
| 53 | + | </div> | |
| 54 | + | <div class="g1t-diagram g1t-diagram__dialog-body"> | |
| 55 | + | <div class="g1t-diagram__scroll"><Fragment set:html={large} /></div> | |
| 56 | + | </div> | |
| 57 | + | </dialog> | |
| 58 | + | </figure> | |
| 59 | + | ||
| 60 | + | <script> | |
| 61 | + | // Expand opens the figure's dialog; Close, Escape or a click on the | |
| 62 | + | // backdrop closes it. | |
| 63 | + | for (const figure of document.querySelectorAll<HTMLElement>('[data-diagram]')) { | |
| 64 | + | const dialog = figure.querySelector('dialog'); | |
| 65 | + | if (!dialog) continue; | |
| 66 | + | figure.querySelector('[data-diagram-open]')?.addEventListener('click', () => dialog.showModal()); | |
| 67 | + | dialog.querySelector('[data-diagram-close]')?.addEventListener('click', () => dialog.close()); | |
| 68 | + | dialog.addEventListener('click', (event) => { | |
| 69 | + | if (event.target === dialog) dialog.close(); | |
| 70 | + | }); | |
| 71 | + | } | |
| 72 | + | </script> |
| 1 | + | --- | |
| 2 | + | title: Actions, runners and sandboxes | |
| 3 | + | description: How a workflow job is planned and run, on g1t's runners or yours, what walls each job in, how secrets reach a job without the job holding them, and what else runs in a sandbox. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | import Aside from '../../../components/Aside.astro'; | |
| 7 | + | import Diagram from '../../../components/Diagram.astro'; | |
| 8 | + | ||
| 9 | + | Everything that runs code you did not write runs in a sandbox: workflow | |
| 10 | + | jobs, merge queue builds, deployments, dependency updates, backups. Three | |
| 11 | + | pieces share the work. | |
| 12 | + | ||
| 13 | + | | Piece | Is | Does | | |
| 14 | + | | --- | --- | --- | | |
| 15 | + | | `services/actions` | A Rust Worker, D1 `g1t-actions` | Reads workflows, starts runs, plans jobs, keeps logs, secrets and self-hosted runners | | |
| 16 | + | | `services/runner` | A TypeScript Worker on Cloudflare Containers | Starts a sandbox per job, and owns what leaves it | | |
| 17 | + | | `crates/runner` | The `g1t-runner` binary | Runs inside every sandbox and on every self-hosted runner, does one job, and exits | | |
| 18 | + | ||
| 19 | + | `crates/actions` holds the workflow language itself, with no I/O: | |
| 20 | + | syntax, `${{ }}` expressions, filters, matrices and cron. The Worker plans | |
| 21 | + | with it and the sandbox runs with it, so the two cannot read a workflow | |
| 22 | + | differently. | |
| 23 | + | ||
| 24 | + | ## A workflow job | |
| 25 | + | ||
| 26 | + | <Diagram name="actions-job" /> | |
| 27 | + | ||
| 28 | + | 1. **Trigger.** `actions` reads its event queue, `g1t-events-actions`, ten | |
| 29 | + | at a time (step 1), and matches each event to the workflows read from | |
| 30 | + | the default branch's `.g1t/workflows` (or `.github/workflows`). The | |
| 31 | + | files are GitHub Actions syntax, unchanged; [GitHub | |
| 32 | + | Actions](/guides/actions/) lists what is supported. | |
| 33 | + | 2. **Plan.** It expands matrices, orders jobs by `needs`, and reports the | |
| 34 | + | run as pending on the commit (steps 2 and 3). | |
| 35 | + | 3. **Start.** A hosted job is taken by flipping its row from `queued` to | |
| 36 | + | `in_progress` in one statement, so two starters can never both take it. | |
| 37 | + | `actions` asks `runner` for a sandbox (step 4), and `runner` starts a | |
| 38 | + | Container for that job alone (step 5). | |
| 39 | + | 4. **Run.** The sandbox reports its steps and logs to the API, with the | |
| 40 | + | job's own token (step 6), which passes them to `actions` (step 7). The | |
| 41 | + | answer can tell the job to stop, which is how **Cancel** reaches a | |
| 42 | + | running job (step 8). | |
| 43 | + | 5. **Finish.** The run's conclusion goes onto the commit as a status | |
| 44 | + | (step 9), where [required checks](/architecture/work/#checks-and-statuses) | |
| 45 | + | find it, and `workflow.completed` is published (step 10). | |
| 46 | + | ||
| 47 | + | A job that says nothing for 10 minutes is taken as lost. A cron in | |
| 48 | + | `actions` runs every minute for schedules, jobs waiting for room, and | |
| 49 | + | jobs whose runner went quiet. | |
| 50 | + | ||
| 51 | + | ### Limits | |
| 52 | + | ||
| 53 | + | | | | | |
| 54 | + | | --- | --- | | |
| 55 | + | | Hosted jobs running at once | 4 per workspace | | |
| 56 | + | | A hosted job's timeout | 60 minutes at most | | |
| 57 | + | | A self-hosted job's timeout | 24 hours at most | | |
| 58 | + | | Logs | 4 MiB a job, kept 90 days | | |
| 59 | + | | Workflows per repository | 50 | | |
| 60 | + | | Schedules | Paused after 60 days without a push | | |
| 61 | + | ||
| 62 | + | ### Logs, caches and artifacts | |
| 63 | + | ||
| 64 | + | Logs are kept in D1, in chunks, deduplicated so a report sent twice is | |
| 65 | + | stored once. Cache entries and artifacts go to the R2 bucket | |
| 66 | + | `g1t-actions-cache`, uploaded through the API: cache entries under `c/` | |
| 67 | + | for 30 days, artifacts under `a/` for 91. `actions` lists them and decides | |
| 68 | + | what is kept. | |
| 69 | + | ||
| 70 | + | ## Sandboxes on g1t's runners | |
| 71 | + | ||
| 72 | + | `runner` defines three Container classes, picked by `runs-on`: | |
| 73 | + | ||
| 74 | + | | `runs-on` | Class | Size | | |
| 75 | + | | --- | --- | --- | | |
| 76 | + | | Any other hosted label, and every other kind of work | `AttemptSandbox` | Cloudflare's `standard-1` | | |
| 77 | + | | `g1t-2core` | `Sandbox2Core` | 2 vCPU, 8 GiB memory, 16 GB disk | | |
| 78 | + | | `g1t-4core` | `Sandbox4Core` | 4 vCPU, 12 GiB memory, 20 GB disk | | |
| 79 | + | ||
| 80 | + | Every sandbox starts from one image: `services/runner/base/Dockerfile`, | |
| 81 | + | which has Node, Python, Go, Rust, Java, .NET, Ruby and Docker, with the | |
| 82 | + | `g1t-runner` binary on top. Hosted runners are Linux only. | |
| 83 | + | ||
| 84 | + | <Diagram name="sandboxes" /> | |
| 85 | + | ||
| 86 | + | ### Egress | |
| 87 | + | ||
| 88 | + | A sandbox starts with no internet. Every HTTP and HTTPS request it makes | |
| 89 | + | goes through the runner Worker's outbound handler (`src/egress.ts`, | |
| 90 | + | `src/guard.ts`), which decides what may leave. Other protocols have no | |
| 91 | + | route out at all. | |
| 92 | + | ||
| 93 | + | ### Secrets a job never holds | |
| 94 | + | ||
| 95 | + | A secret can be **brokered**: the job sees a stand-in, `g1tsec_…`, and | |
| 96 | + | the egress handler swaps in the real value only on a request to a host | |
| 97 | + | the secret allows (`src/broker.ts`). On a self-hosted runner, the job sends | |
| 98 | + | the request to `POST /actions/jobs/{job}/broker` instead, and g1t makes | |
| 99 | + | it, so the value never reaches your machine at all. | |
| 100 | + | ||
| 101 | + | ## Runners of your own | |
| 102 | + | ||
| 103 | + | A [self-hosted runner](/guides/self-hosted-runners/) only ever calls out, | |
| 104 | + | over HTTPS, so it needs no open port: | |
| 105 | + | ||
| 106 | + | | Call | What it does | | |
| 107 | + | | --- | --- | | |
| 108 | + | | `POST /runners/register` | Trades a registration token, `g1trt_…`, valid for an hour, for a runner credential, `g1tr_…` | | |
| 109 | + | | `POST /runners/{id}/poll` | Waits up to 20 seconds for a job; doubles as the heartbeat | | |
| 110 | + | | `POST /runners/{id}/finished` | Says the job is over | | |
| 111 | + | | `POST /runners/{id}/remove` | Signs the runner out | | |
| 112 | + | ||
| 113 | + | The runner credential is stored only as a hash and rotates every day on a | |
| 114 | + | poll. A runner that has not polled for 90 seconds is offline; one offline | |
| 115 | + | for 14 days is removed. A job goes to a runner that has every one of its | |
| 116 | + | `runs-on` labels and whose group allows the repository, and waits up to 24 | |
| 117 | + | hours for one. | |
| 118 | + | ||
| 119 | + | ### Three walls | |
| 120 | + | ||
| 121 | + | On your machine, `g1t-runner` puts three walls between a job and the | |
| 122 | + | host: | |
| 123 | + | ||
| 124 | + | 1. **A VM.** WSL2 on Windows, with drives, interop and the Windows `PATH` | |
| 125 | + | turned off, vfkit on macOS, or QEMU on Linux (`deploy/sandbox`). | |
| 126 | + | 2. **A Sysbox container**, the runner itself, with its own dockerd. | |
| 127 | + | It refuses to start if the host's `docker.sock` is mounted | |
| 128 | + | (`deploy/runner`). | |
| 129 | + | 3. **A container per job**, new each time. | |
| 130 | + | ||
| 131 | + | Native Windows jobs run in Windows Sandbox instead. Each runner reports | |
| 132 | + | its isolation, `nested-container`, `vm`, `container` or `host`, and a | |
| 133 | + | `host` runner never gets a public repository's jobs. The [desktop | |
| 134 | + | app](/guides/desktop/) runs the sandboxed runner by default while it is | |
| 135 | + | open; Docker or direct execution happen only if you choose them. | |
| 136 | + | ||
| 137 | + | ## Other work in sandboxes | |
| 138 | + | ||
| 139 | + | `g1t-runner` takes a `MODE`, so the same image does more than workflows: | |
| 140 | + | ||
| 141 | + | | Mode | Started by | Does | | |
| 142 | + | | --- | --- | --- | | |
| 143 | + | | `actions` | `actions` | A workflow job | | |
| 144 | + | | `queue` | `runner`, when the queue changes | Builds a merge queue candidate | | |
| 145 | + | | `mergecheck` | `runner`, when a pull request changes | Checks whether a pull request merges cleanly | | |
| 146 | + | | `deploy` | `deployments` | Builds an app ([Deployments](/architecture/deployments/)) | | |
| 147 | + | | `bump` | `security` | Makes a dependency update | | |
| 148 | + | | `backup` | `runner`'s sweep, every 5 minutes | Cuts a repository backup ([Git](/architecture/git/#backups)) | | |
| 149 | + | ||
| 150 | + | Compute is admitted through billing: a sandbox starts only after | |
| 151 | + | `billing` holds its estimated cost (`reserve`), and the hold is settled | |
| 152 | + | with the real cost when it ends (`settle`). | |
| 153 | + | ||
| 154 | + | <Aside type="caution" title="Self-hosted installations"> | |
| 155 | + | A [self-hosted g1t](/guides/self-hosting-architecture/) binds `runner` | |
| 156 | + | to a Worker that answers that it is off, so workflows, the merge queue | |
| 157 | + | and deployments do not run there yet. | |
| 158 | + | </Aside> |
| 1 | + | --- | |
| 2 | + | title: Data and storage | |
| 3 | + | description: Which service owns which database, bucket and Durable Object, the few stores that are shared, how schema changes ship, and the ports that let the same code run on Cloudflare or in Docker. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | import Aside from '../../../components/Aside.astro'; | |
| 7 | + | import Diagram from '../../../components/Diagram.astro'; | |
| 8 | + | ||
| 9 | + | ## Who owns what | |
| 10 | + | ||
| 11 | + | Each D1 database is bound by exactly one Worker. If you need another | |
| 12 | + | service's data, call that service; never read its tables. | |
| 13 | + | ||
| 14 | + | | Service | D1 | Other storage | | |
| 15 | + | | --- | --- | --- | | |
| 16 | + | | `identity` | `g1t` | Writes the avatars KV | | |
| 17 | + | | `repos` | `g1t-repos` | The git store (Artifacts), the git cache KV, the `g1t-git-packs` and `g1t-backups` buckets | | |
| 18 | + | | `work` | `g1t-work` | | | |
| 19 | + | | `events` | `g1t-events` | The `g1t-events` queue, which it produces and consumes, and the subscriber queues it produces | | |
| 20 | + | | `actions` | `g1t-actions` | Lists and expires the `g1t-actions-cache` bucket | | |
| 21 | + | | `billing` | `g1t-billing` | | | |
| 22 | + | | `integrations` | `g1t-integrations` | | | |
| 23 | + | | `webhooks` | `g1t-webhooks` | | | |
| 24 | + | | `security` | `g1t-security` | | | |
| 25 | + | | `search` | `g1t-search` | The `g1t-search-jobs` queue | | |
| 26 | + | | `packages` | `g1t-packages` | The `g1t-packages` bucket | | |
| 27 | + | | `projects` | `g1t-projects` | Writes project images to the avatars KV | | |
| 28 | + | | `chat` | `g1t-chat` | A `ChannelRoom` Durable Object per channel; writes custom emoji to the avatars KV | | |
| 29 | + | | `artifacts` | `g1t-docs` | The `g1t-docs-files` bucket, the `g1t-docs` and `g1t-folios` Vectorize indexes, `PageRoom` and `FolioRoom` Durable Objects | | |
| 30 | + | | `deployments` | `g1t-deployments` | Writes the domains KV; the `g1t-deployments` dispatch namespace | | |
| 31 | + | | `apps/status` | `g1t-status` | | | |
| 32 | + | | `notify` | | `Feed`, `Room` and `Topic` Durable Objects | | |
| 33 | + | | `telemetry` | | A `TelemetryStore` Durable Object per project | | |
| 34 | + | | `runner` | | The sandbox Container classes | | |
| 35 | + | | `og` | | The `g1t-screenshots` bucket | | |
| 36 | + | | `apps/web` | | The `g1t-downloads` bucket | | |
| 37 | + | ||
| 38 | + | ### The shared stores | |
| 39 | + | ||
| 40 | + | Three stores are shared on purpose, each with a single writer per key | |
| 41 | + | prefix: | |
| 42 | + | ||
| 43 | + | | Store | Written by | Read by | | |
| 44 | + | | --- | --- | --- | | |
| 45 | + | | `g1t-avatars` KV | `identity` (people and workspaces), `projects` (`production/`), `chat` (`emoji/`) | The site, `og` | | |
| 46 | + | | `g1t-domains` KV | `deployments` | `pages`, `telemetry` | | |
| 47 | + | | `g1t-actions-cache` R2 | The API, as jobs upload | `actions`, which decides what is kept | | |
| 48 | + | ||
| 49 | + | ## Durable Objects | |
| 50 | + | ||
| 51 | + | Durable Objects are used where something has to be live, or where data | |
| 52 | + | belongs to one thing and grows on its own: a chat channel, an open doc, a | |
| 53 | + | person's open tabs, a project's telemetry. Each object keeps its own | |
| 54 | + | SQLite storage. A rule from CONTRIBUTING.md: never wake an object for a | |
| 55 | + | read that D1 can answer. | |
| 56 | + | ||
| 57 | + | ## Secrets at rest | |
| 58 | + | ||
| 59 | + | Secrets that g1t has to use later, such as webhook secrets, Actions | |
| 60 | + | secrets, GitHub tokens of linked accounts and two-factor seeds, are sealed | |
| 61 | + | with AES-256-GCM by `Sealer` in `crates/secrets`, with the row's id as | |
| 62 | + | associated data. A sealed value copied into another row will not open. | |
| 63 | + | Each service seals with its own key (`IDENTITY_KEY`, `ACTIONS_KEY`, | |
| 64 | + | `REPOS_KEY`, …), set as a Worker secret. | |
| 65 | + | ||
| 66 | + | Access tokens and session cookies are not stored at all: only their | |
| 67 | + | SHA-256 is. | |
| 68 | + | ||
| 69 | + | ## Schema changes | |
| 70 | + | ||
| 71 | + | Each database's migrations live beside its service, in | |
| 72 | + | `services/<name>/migrations`, numbered. A deploy applies every pending | |
| 73 | + | migration before any code ships | |
| 74 | + | ([How g1t ships](/architecture/shipping/)), so for a minute or more the old | |
| 75 | + | code runs against the new schema. Three rules follow: | |
| 76 | + | ||
| 77 | + | 1. Add tables and columns. Never rename or drop one that live code reads. | |
| 78 | + | 2. Change what a value means in two deploys: first code that reads both | |
| 79 | + | forms, then the change. | |
| 80 | + | 3. Remove what nothing reads anymore only after the code that read it is | |
| 81 | + | gone. | |
| 82 | + | ||
| 83 | + | ## Ports and adapters | |
| 84 | + | ||
| 85 | + | Each Cloudflare product a service depends on sits behind a port: a trait | |
| 86 | + | in Rust, or a binding whose shape a stand-in can copy. g1t.sh binds | |
| 87 | + | Cloudflare's adapters. A self-hosted installation, built from the same | |
| 88 | + | source, binds open ones, and the service code does not change. | |
| 89 | + | ||
| 90 | + | <Diagram name="ports-and-adapters" /> | |
| 91 | + | ||
| 92 | + | | Port | Defined in | g1t.sh | Self-hosted | | |
| 93 | + | | --- | --- | --- | --- | | |
| 94 | + | | `GitStore`, `GitRepo` | `services/repos/src/store.rs` | `ArtifactsStore` | A Worker with the same API as the Artifacts binding (`deploy/self-host/workers/gitstore`), in front of a server of bare repositories using `git http-backend` | | |
| 95 | + | | `BlobStore` | `crates/blobstore` | `R2Store` | `S3Store`, with AWS Signature V4, for RustFS, MinIO, Ceph, Garage or AWS S3 | | |
| 96 | + | | `PackStore` | `services/repos/src/pack_cache.rs` | R2 | S3, with `PACK_STORE=s3` | | |
| 97 | + | | `EMAIL` | A `send_email` binding | Cloudflare Email Sending | A mail Worker that hands each message to Mailpit | | |
| 98 | + | | `RUNNER` | A service binding | `services/runner` on Containers | A Worker that answers that the feature is off | | |
| 99 | + | | D1, KV, Queues, Durable Objects | Workers bindings | Cloudflare | workerd, with SQLite files on a volume | | |
| 100 | + | ||
| 101 | + | `deploy/self-host/configs.mjs` writes each Worker's self-hosted config from | |
| 102 | + | its `wrangler.jsonc`, so the two never drift apart: it drops routes and | |
| 103 | + | placement, binds the stand-ins above, points buckets at S3, and replaces | |
| 104 | + | g1t.sh's addresses with yours. What runs and what is off comes from each | |
| 105 | + | unit's `self_host` field in `deploy/stack.jsonc`. | |
| 106 | + | ||
| 107 | + | <Aside title="Adding a dependency on a Cloudflare product"> | |
| 108 | + | Put it behind a port first, and give the self-hosted build an adapter | |
| 109 | + | or a stand-in in the same pull request. [How a self-hosted g1t | |
| 110 | + | runs](/guides/self-hosting-architecture/) is the self-hosted side of | |
| 111 | + | this table. | |
| 112 | + | </Aside> |
| 1 | + | --- | |
| 2 | + | title: Deployments, packages and telemetry | |
| 3 | + | description: How apps are built and served on g1t.page, how the package registries are routed and stored, and where a project's logs, traces and metrics go. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | import Diagram from '../../../components/Diagram.astro'; | |
| 7 | + | ||
| 8 | + | ## Deployments | |
| 9 | + | ||
| 10 | + | `deployments` (`services/deployments`, TypeScript, D1 `g1t-deployments`) | |
| 11 | + | builds apps and puts them live. `pages` (`services/pages`) serves them. | |
| 12 | + | Neither runs your app as a server that waits: each app is a Worker, which | |
| 13 | + | runs only while it answers a request, so an app nobody visits costs | |
| 14 | + | nothing. | |
| 15 | + | ||
| 16 | + | <Diagram name="deployment-path" /> | |
| 17 | + | ||
| 18 | + | ### Building | |
| 19 | + | ||
| 20 | + | Every push to the default branch makes a production deployment, and every | |
| 21 | + | open pull request gets a preview. `deployments` reads pushes from its | |
| 22 | + | event queue and asks `runner` for a sandbox in `deploy` mode | |
| 23 | + | ([Actions, runners and sandboxes](/architecture/compute/#other-work-in-sandboxes)). | |
| 24 | + | The sandbox works out what the app is, in this order: | |
| 25 | + | ||
| 26 | + | 1. a Workers project, from `wrangler deploy --dry-run`; | |
| 27 | + | 2. a framework it knows, such as Vite, Astro, Next.js or SvelteKit; | |
| 28 | + | 3. a container, which g1t.sh does not run yet (`CONTAINERS` is `off`); | |
| 29 | + | 4. a static site. | |
| 30 | + | ||
| 31 | + | The sandbox is given a key that can upload the app's static files and | |
| 32 | + | nothing else. `deployments` uploads the built Worker itself to the Workers | |
| 33 | + | for Platforms dispatch namespace `g1t-deployments`. A site with only static | |
| 34 | + | files gets a Worker that serves just its assets. A workflow can build | |
| 35 | + | instead and hand over the result, with | |
| 36 | + | `POST /repos/{owner}/{repo}/deployments/builds` or the `g1t/deploy` action. | |
| 37 | + | ||
| 38 | + | ### Serving | |
| 39 | + | ||
| 40 | + | | | Hostname | | |
| 41 | + | | --- | --- | | |
| 42 | + | | Production | `<project>-<workspace>.g1t.page` | | |
| 43 | + | | A preview | `<project>-git-<branch>-<workspace>.g1t.page` | | |
| 44 | + | | Your domain | A CNAME to `domains.g1t.page`, a custom hostname on g1t.page's zone | | |
| 45 | + | ||
| 46 | + | `pages` answers every one of those hostnames. It reads the app's name from | |
| 47 | + | the hostname, or looks a custom domain up in the `g1t-domains` KV, which | |
| 48 | + | `deployments` writes, and hands the request to that app's Worker with | |
| 49 | + | `env.APPS.get(app)`. Each request to an app may use 50 ms of CPU and 50 | |
| 50 | + | subrequests. A preview comes down when its pull request closes, or after it | |
| 51 | + | has been idle for a number of days you choose (7 by default). | |
| 52 | + | ||
| 53 | + | ## Packages | |
| 54 | + | ||
| 55 | + | `packages` (`services/packages`, Rust, D1 `g1t-packages`) is every package | |
| 56 | + | registry g1t has. The site recognises a registry's path and passes the | |
| 57 | + | request on whole, as it does for git: | |
| 58 | + | ||
| 59 | + | | Registry | Path on `g1t.sh` | | |
| 60 | + | | --- | --- | | |
| 61 | + | | Container images (OCI Distribution 1.1) | `/v2/`, with tokens at `/v2/token` | | |
| 62 | + | | npm | `/-/npm/` | | |
| 63 | + | | Cargo, sparse index | `/-/cargo/<workspace>/` | | |
| 64 | + | | Composer | `/-/composer/` | | |
| 65 | + | | Maven | `/-/maven/` | | |
| 66 | + | | NuGet | `/-/nuget/<workspace>/v3/index.json` | | |
| 67 | + | | RubyGems | `/-/rubygems/` | | |
| 68 | + | ||
| 69 | + | Files are stored in the `g1t-packages` bucket by their digest, so a layer | |
| 70 | + | or a tarball pushed twice is kept once. The bucket sits behind the | |
| 71 | + | `BlobStore` port, so a self-hosted installation keeps them in any S3 | |
| 72 | + | store. Large downloads are answered with a signed URL to the bucket. | |
| 73 | + | [Packages](/guides/packages/) has each registry's setup. | |
| 74 | + | ||
| 75 | + | ## Telemetry | |
| 76 | + | ||
| 77 | + | `telemetry` (`services/telemetry`, TypeScript) keeps each project's logs, | |
| 78 | + | traces, metrics and errors in a Durable Object of its own, | |
| 79 | + | `TelemetryStore`, in that object's SQLite storage. A project's data never | |
| 80 | + | shares a database with another's. | |
| 81 | + | ||
| 82 | + | It has three ways in: | |
| 83 | + | ||
| 84 | + | | Door | From | | |
| 85 | + | | --- | --- | | |
| 86 | + | | `otel.g1t.sh/v1/traces`, `/v1/logs`, `/v1/metrics` | Your software, as OpenTelemetry over HTTP, JSON or protobuf, with a `g1to_…` token | | |
| 87 | + | | Tail events | Apps on `g1t.page`: `telemetry` is `pages`' tail consumer, and apps name it as their Tail Worker | | |
| 88 | + | | `/otlp/v1/*` on the API | The same as `otel.g1t.sh`, passed on as it is; the only door on a self-hosted installation | | |
| 89 | + | ||
| 90 | + | A store keeps up to 4 GiB and sheds its oldest day past that. You choose | |
| 91 | + | how long raw data is kept, from 1 to 30 days; rollups are kept 90 days. | |
| 92 | + | [Observability](/guides/observability/) shows what you can do with it. |
| 1 | + | --- | |
| 2 | + | title: Events | |
| 3 | + | description: How a state change in one service reaches every other that cares, through one log and a queue per subscriber, and how webhooks, live pages and followers read it. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | import Aside from '../../../components/Aside.astro'; | |
| 7 | + | import Diagram from '../../../components/Diagram.astro'; | |
| 8 | + | ||
| 9 | + | Every state change in g1t is published as an event: a push, an issue | |
| 10 | + | opened, a pull request merged, a workflow finished, a workspace renamed. | |
| 11 | + | Services do not call each other to announce what happened. They publish, | |
| 12 | + | and the `events` service (`services/events`, Rust) delivers. | |
| 13 | + | ||
| 14 | + | ## The envelope | |
| 15 | + | ||
| 16 | + | Every event has the same shape, which follows CloudEvents | |
| 17 | + | (`crates/contracts/src/events.rs`, `packages/contracts/src/events.ts`): | |
| 18 | + | ||
| 19 | + | | Field | What it is | | |
| 20 | + | | --- | --- | | |
| 21 | + | | `id` | `evt_` and a UUIDv7: sortable by the time it was made | | |
| 22 | + | | `type` | Such as `git.push`, `pull.merged`, `workflow.completed` | | |
| 23 | + | | `source` | The service that published it | | |
| 24 | + | | `time` | When, in RFC 3339 | | |
| 25 | + | | `repoId` | The repository it is about, or none | | |
| 26 | + | | `actor` | The account that caused it, or none | | |
| 27 | + | | `data` | The type's own fields, in its contract | | |
| 28 | + | ||
| 29 | + | Coordination has its own events, published by `work`: sessions starting | |
| 30 | + | and ending, claims, hand-offs, dependencies, overlaps, and the merge | |
| 31 | + | queue's verdicts (`integration.failed`, `integration.passed`, | |
| 32 | + | `decision.recorded`). They reach webhooks and followers like any other | |
| 33 | + | ([Work and the merge queue](/architecture/work/#coordination)). | |
| 34 | + | ||
| 35 | + | Two fields in `data` matter everywhere: | |
| 36 | + | ||
| 37 | + | - `causedByJob` marks an event caused by a workflow's own token, so it | |
| 38 | + | does not start more workflows. | |
| 39 | + | - `truncated: true` marks an event that was too big for a queue message | |
| 40 | + | (96 KB) and had its long strings cut, step by step, until it fit. | |
| 41 | + | ||
| 42 | + | ## From publish to every subscriber | |
| 43 | + | ||
| 44 | + | <Diagram name="event-fanout" /> | |
| 45 | + | ||
| 46 | + | 1. A service calls `events`' `publish` with a batch of events. `events` | |
| 47 | + | gives each its id and time and puts them on the `g1t-events` queue, then | |
| 48 | + | returns. Publishing never waits on a subscriber. | |
| 49 | + | 2. `events` reads its own queue, 100 messages at a time, and for each | |
| 50 | + | batch: | |
| 51 | + | - writes the events to the log with `INSERT OR IGNORE`, so a batch that | |
| 52 | + | comes round twice is written once; | |
| 53 | + | - sends each subscriber queue the events routed to it; | |
| 54 | + | - tells `notify`, so open pages of that repository refresh, and puts | |
| 55 | + | items in people's inboxes, at most once per person per event. | |
| 56 | + | 3. If a send to a subscriber fails, `events` notes which queues already | |
| 57 | + | have the batch (`fanout_sent`, kept a day) and throws. The retry skips | |
| 58 | + | those queues. | |
| 59 | + | 4. After three tries, a batch goes to `g1t-events-dlq`. | |
| 60 | + | ||
| 61 | + | Every subscriber's consumer is set up the same way: three tries, then the | |
| 62 | + | dead-letter queue. | |
| 63 | + | ||
| 64 | + | ### Subscriber queues | |
| 65 | + | ||
| 66 | + | Each service that reacts to events has its own queue, | |
| 67 | + | `g1t-events-<service>`. It gets only the types it needs, as `ROUTES` in | |
| 68 | + | `crates/contracts/src/subscribers.rs` says, plus every lifecycle event | |
| 69 | + | (renames, transfers and deletions of repositories, workspaces and users), | |
| 70 | + | which every subscriber gets. | |
| 71 | + | ||
| 72 | + | | Queue | Read by | Gets | | |
| 73 | + | | --- | --- | --- | | |
| 74 | + | | `g1t-events-work` | `work` | `git.push`, `branch.renamed`, `repo.created`, `repo.archived`, `repo.default_branch_changed`, `pull.merged`, `comment.created` | | |
| 75 | + | | `g1t-events-runner` | `runner` | Pull request changes, `checks.completed`, `queue.changed`, `comment.created`, issue changes, `repo.archived` | | |
| 76 | + | | `g1t-events-actions` | `actions` | `git.push`, `git.deleted`, `pull.*`, `issue.*`, `comment.*`, `release.*`, deployments, `workflow.completed`, `repo.archived` | | |
| 77 | + | | `g1t-events-webhooks` | `webhooks` | Everything | | |
| 78 | + | | `g1t-events-security` | `security` | `git.push`, pull request changes, `comment.created`, `repo.created`, `repo.visibility_changed` | | |
| 79 | + | | `g1t-events-search` | `search` | `git.push`, `repo.*`, `issue.*`, `pull.*`, `user.*`, `workspace.*` | | |
| 80 | + | | `g1t-events-deployments` | `deployments` | Pushes, the pull request lifecycle, workspace deletion and restore | | |
| 81 | + | | `g1t-events-projects` | `projects` | Repositories, packages and deployments | | |
| 82 | + | | `g1t-events-repos` | `repos` | `pull.merged`, `pull.closed`, `pull.reopened`, deletions and restores | | |
| 83 | + | | `g1t-events-packages` | `packages` | `git.push`, `repo.visibility_changed`, workspace and team changes | | |
| 84 | + | | `g1t-events-integrations` | `integrations` | `issue.closed`, `pull.opened` | | |
| 85 | + | | `g1t-events-billing` | `billing` | Lifecycle events only | | |
| 86 | + | | `g1t-events-docs` | `artifacts` | `git.push`, `pull.merged` | | |
| 87 | + | ||
| 88 | + | A subscriber should treat every event as possibly delivered twice, and | |
| 89 | + | make its handling idempotent: most do it with a unique key on the event's | |
| 90 | + | `id`. | |
| 91 | + | ||
| 92 | + | ## The log | |
| 93 | + | ||
| 94 | + | `g1t-events`' D1 database keeps the log. Each event gets a `seq` as it is | |
| 95 | + | written, one higher than the last; the log has one writer, the consumer | |
| 96 | + | above. Ids are made by publishers in many places, so they can arrive out | |
| 97 | + | of order. Reading "everything after this event" therefore follows `seq`, | |
| 98 | + | not `id`. | |
| 99 | + | ||
| 100 | + | You can read a repository's log with the API's `list_events`, newest | |
| 101 | + | first, 50 a page and up to 200, filtered by type. Status and check events | |
| 102 | + | are left out unless you ask for their type. The same database keeps the | |
| 103 | + | audit log, for 400 days. | |
| 104 | + | ||
| 105 | + | ### Following | |
| 106 | + | ||
| 107 | + | Pass `after`, an event id, and `list_events` returns what the log wrote | |
| 108 | + | after it, oldest first. Add `wait`, up to 25 seconds, and the call waits | |
| 109 | + | for something to arrive: | |
| 110 | + | ||
| 111 | + | <Diagram name="follow-events" /> | |
| 112 | + | ||
| 113 | + | The API asks the log about once a second while it waits (`apps/api/src/follow.rs`). | |
| 114 | + | `g1t work events --follow` runs this in a loop, starting each call from the | |
| 115 | + | last event it saw, so it misses nothing between calls. Coding agents get | |
| 116 | + | the same over MCP, with the `list_events` tool. | |
| 117 | + | ||
| 118 | + | ## Webhooks | |
| 119 | + | ||
| 120 | + | `webhooks` reads every event and delivers those each subscription asks | |
| 121 | + | for: | |
| 122 | + | ||
| 123 | + | | | | | |
| 124 | + | | --- | --- | | |
| 125 | + | | **Request** | `POST` with JSON, to public addresses only, with a 10 second timeout. Anything but a 2xx is a failure. | | |
| 126 | + | | **Headers** | `x-g1t-event`, `x-g1t-delivery`, `x-g1t-hook`, and `x-g1t-signature-256: sha256=<HMAC-SHA256 of the body>` | | |
| 127 | + | | **Body** | `{ id, type, time, workspace, repository: { id, full_name }, actor: { id, username }, data }`, in snake_case | | |
| 128 | + | | **Retries** | After 1 minute, 5 minutes, 30 minutes, 2 hours and 5 hours: six attempts in about seven and a half hours | | |
| 129 | + | | **At most once** | A delivery is unique on the subscription and the event's `id` | | |
| 130 | + | | **Kept** | 14 days | | |
| 131 | + | ||
| 132 | + | Each subscription's secret is sealed in D1. [Webhooks](/guides/webhooks/) | |
| 133 | + | shows how to check a signature. | |
| 134 | + | ||
| 135 | + | ## Live pages | |
| 136 | + | ||
| 137 | + | `notify` (TypeScript) keeps three kinds of Durable Object: | |
| 138 | + | ||
| 139 | + | | Object | One per | Holds | | |
| 140 | + | | --- | --- | --- | | |
| 141 | + | | `Feed` | Person | Their open tabs' sockets, notifications, counts and push subscriptions | | |
| 142 | + | | `Room` | Workspace | Who is here | | |
| 143 | + | | `Topic` | Repository | The pages open on it, told by `repo_changed` to refresh | | |
| 144 | + | ||
| 145 | + | Sockets use the WebSocket Hibernation API, so an idle tab costs nothing | |
| 146 | + | while it waits. Browser push uses VAPID, and phones get Expo push; both are | |
| 147 | + | sent to live 12 hours. | |
| 148 | + | ||
| 149 | + | `chat` and `artifacts` work the same way with rooms of their own: a | |
| 150 | + | `ChannelRoom` per channel, and a `PageRoom` per doc that syncs edits as a | |
| 151 | + | Yjs document and saves them to D1 a few seconds after a burst of typing. | |
| 152 | + | ||
| 153 | + | <Aside title="Scheduled jobs"> | |
| 154 | + | Some work runs on a clock rather than on events: webhook retries every | |
| 155 | + | minute, Actions schedules every minute, fork retirement hourly, backups | |
| 156 | + | nightly, the event log's purge daily. Each service's `wrangler.jsonc` | |
| 157 | + | lists its `crons`. | |
| 158 | + | </Aside> |
| 1 | + | --- | |
| 2 | + | title: Git | |
| 3 | + | description: Where g1t keeps repositories, what happens on a push and a clone, the caches in front of the git store, pull request forks, merges, and backups. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | import Aside from '../../../components/Aside.astro'; | |
| 7 | + | import Diagram from '../../../components/Diagram.astro'; | |
| 8 | + | ||
| 9 | + | Every repository on g1t.sh is an ordinary git repository in Cloudflare | |
| 10 | + | Artifacts, a git store. One service, `repos` (`services/repos`, Rust), | |
| 11 | + | holds the only binding to it. Everything else that reads code, the site | |
| 12 | + | and the API included, asks `repos`. | |
| 13 | + | ||
| 14 | + | ## Where repositories live | |
| 15 | + | ||
| 16 | + | `repos` talks to the store through a port: the `GitStore` and `GitRepo` | |
| 17 | + | traits in `services/repos/src/store.rs`, with `ArtifactsStore` as the | |
| 18 | + | adapter for Artifacts. Self-hosted installations bind a different adapter | |
| 19 | + | in its place ([Data and storage](/architecture/data/#ports-and-adapters)). | |
| 20 | + | Every call to the store goes through one function that meters it, retries | |
| 21 | + | reads, and trips a circuit breaker when the store struggles | |
| 22 | + | (`resilience.rs`). | |
| 23 | + | ||
| 24 | + | ### Namespaces and sharding | |
| 25 | + | ||
| 26 | + | An Artifacts namespace takes a fixed number of control requests per 10 | |
| 27 | + | seconds, and its jurisdiction, US or EU, is set when it is made. So g1t | |
| 28 | + | can spread repositories across several namespaces, each its own binding | |
| 29 | + | (`services/repos/src/shards.rs`): | |
| 30 | + | ||
| 31 | + | | Setting | What it decides | | |
| 32 | + | | --- | --- | | |
| 33 | + | | `GITSTORE_NAMESPACES` | The namespaces and their bindings, such as `GITSTORE_1` and `GITSTORE_EU` | | |
| 34 | + | | `GITSTORE_NEW_REPOS` | Where new repositories go | | |
| 35 | + | | `GITSTORE_EU_NAMESPACE` | Where an EU workspace's repositories go | | |
| 36 | + | ||
| 37 | + | A repository's row in D1 records its store key, `namespace/owner--name`. | |
| 38 | + | A key with no namespace is in the first, `g1t`. Moving a repository | |
| 39 | + | between namespaces is an operator task, in `moves.rs`. | |
| 40 | + | ||
| 41 | + | ### What D1 holds | |
| 42 | + | ||
| 43 | + | `g1t-repos` keeps what git does not: each repository's row (store key, | |
| 44 | + | size, default branch, visibility, `refs_version`), renames and their | |
| 45 | + | redirects, releases, stars, stats, backups, moves, the commits each push | |
| 46 | + | brought for the contribution calendar, and the meter of every git | |
| 47 | + | operation. | |
| 48 | + | ||
| 49 | + | ## A push | |
| 50 | + | ||
| 51 | + | <Diagram name="push-path" /> | |
| 52 | + | ||
| 53 | + | The site recognises a git path and hands the whole request to `repos` | |
| 54 | + | (steps 1 and 2), after the git rate limits. From there, `repos` does the | |
| 55 | + | work in `answer_git` (`services/repos/src/lib.rs`): | |
| 56 | + | ||
| 57 | + | 1. **Who is pushing.** HTTP Basic credentials go to `identity` | |
| 58 | + | (steps 3 and 4). A secret with the token prefix is an access token, and | |
| 59 | + | the username is ignored. Anything else is a password, which `identity` | |
| 60 | + | refuses for anyone with two-factor authentication on, so they push | |
| 61 | + | with a token. The repository's row is read at the same time. | |
| 62 | + | 2. **Which repository.** Renames redirect, transfers resolve, and a pull | |
| 63 | + | request's fork that was retired is made again. | |
| 64 | + | 3. **Whether they may.** The access check, then the workspace's limits. | |
| 65 | + | 4. **The rules.** `repos` asks `work` for the rulesets that apply | |
| 66 | + | (step 5). | |
| 67 | + | 5. **The checks** (step 6). The pack is read once, and four checks run on | |
| 68 | + | it side by side; their verdicts are taken in this order: | |
| 69 | + | ||
| 70 | + | | Check | Where | Refuses | | |
| 71 | + | | --- | --- | --- | | |
| 72 | + | | Workflow gate | `workflow_gate.rs` | A token without the workflow permission changing `.g1t/workflows` | | |
| 73 | + | | Rules | `rules.rs`, judged by `crates/rules` | What a ruleset forbids, with a `remote:` line naming the rule | | |
| 74 | + | | Sizes | `pack_limits.rs` | A file over 32 MB, a repository over 950 MB, a push over 100 MB | | |
| 75 | + | | Push protection | `secret_scan.rs`, using `crates/scan` | Secrets, private email addresses, and the workspace's own patterns | | |
| 76 | + | ||
| 77 | + | 6. **The store** (steps 7 and 8). The pack goes to Artifacts with a | |
| 78 | + | short-lived store credential. | |
| 79 | + | 7. **The answer** (steps 9 and 10). Once the store reports, `refs_version` | |
| 80 | + | goes up, which invalidates every cached ref listing, and only then | |
| 81 | + | does git get its answer. | |
| 82 | + | ||
| 83 | + | After the answer, in `waitUntil`, `repos` checks with the store's log that | |
| 84 | + | each ref really moved, adds the repository's new size, counts commits, and | |
| 85 | + | publishes `git.push` for each branch (step 11 and 12). Everything that | |
| 86 | + | reacts to a push, from workflows to search to deployments, starts from | |
| 87 | + | that event ([Events](/architecture/events/)). | |
| 88 | + | ||
| 89 | + | <Aside title="Large pushes"> | |
| 90 | + | A push up to 24 MiB is scanned before it is stored. A larger one is | |
| 91 | + | streamed straight to the store, its `git.push` events say | |
| 92 | + | `unscanned: true`, and `security` scans it after it lands. | |
| 93 | + | </Aside> | |
| 94 | + | ||
| 95 | + | A few details that keep pushes fast and correct: | |
| 96 | + | ||
| 97 | + | - **`no-thin`.** g1t advertises that it wants whole packs, so the checks | |
| 98 | + | never have to read objects from the store. A client that sends a thin | |
| 99 | + | pack anyway still works: up to 200 bases are fetched. | |
| 100 | + | - **Hidden refs.** `refs/pull/*` belong to g1t. A push to one is refused | |
| 101 | + | for that ref alone. | |
| 102 | + | - **Announcements are capped.** A push with more than three tags announces | |
| 103 | + | none of them, and one with more than 1,000 branches announces only the | |
| 104 | + | default branch. | |
| 105 | + | ||
| 106 | + | ### Timing | |
| 107 | + | ||
| 108 | + | Every git answer carries `Server-Timing`, with a step for each stage: | |
| 109 | + | `repo`, `access`, `mint` (a store credential), `store`, `checks` (with | |
| 110 | + | `read`, `rules`, `scan` and `gate`), `refs`, `total` and more. Run | |
| 111 | + | `GIT_CURL_VERBOSE=1 git push` to see where a slow push spent its time. | |
| 112 | + | ||
| 113 | + | ## A clone or fetch | |
| 114 | + | ||
| 115 | + | <Diagram name="clone-path" /> | |
| 116 | + | ||
| 117 | + | A fetch is answered from the first place that can: | |
| 118 | + | ||
| 119 | + | | Order | From | Holds | Kept | | |
| 120 | + | | --- | --- | --- | --- | | |
| 121 | + | | 1 | R2, `g1t-git-packs` | Whole packs for fresh clones: a request with wants and no haves, shallow ones included | 7 days, or until a ref moves | | |
| 122 | + | | 2 | The colo's cache | Ref listings, and trees and blobs up to 1 MiB, which never change | Refs 60 s; objects forever | | |
| 123 | + | | 3 | KV, `g1t-repos-git-cache` | Ref listings and store credentials, shared by every isolate | 60 s | | |
| 124 | + | | 4 | Artifacts | Everything | | | |
| 125 | + | ||
| 126 | + | Every cached ref listing is keyed by the repository's `refs_version`, so a | |
| 127 | + | push makes the old one unreachable at once. A change can take up to 5 | |
| 128 | + | seconds to reach every fetch, while a repository's row is reused. | |
| 129 | + | ||
| 130 | + | **Packs are cached as they stream.** On a miss, the store's answer goes | |
| 131 | + | to git and to R2 at the same time, never held whole: in one `put` when | |
| 132 | + | small, in 5 MiB parts when large, completed only after a check. A pack | |
| 133 | + | over 200 MiB is not cached, at most two fills run at once, and a | |
| 134 | + | repository fills at most 30 times a minute. A hit never touches the store. | |
| 135 | + | ||
| 136 | + | **What is in KV is sealed.** Each value is encrypted with AES-256-GCM under | |
| 137 | + | the `REPOS_KEY` secret, with its own key as associated data, so a value | |
| 138 | + | copied to another key will not open (`services/repos/src/shared.rs`). | |
| 139 | + | ||
| 140 | + | When the store is busy, git gets 429 or 503 with `Retry-After`. Pushes are | |
| 141 | + | never retried on your behalf. | |
| 142 | + | ||
| 143 | + | ## Pull request forks | |
| 144 | + | ||
| 145 | + | A pull request made for an issue gets a fork of its own | |
| 146 | + | ([Forks and branches](/concepts/forks/)). `fork_for_pull` makes it in the | |
| 147 | + | same namespace as its source, as `pulls--<pull request id>`, with | |
| 148 | + | Artifacts' own fork of the default branch, and it is served at | |
| 149 | + | `g1t.sh/pulls/<id>.git`. Anyone who can read the source can read the fork. | |
| 150 | + | ||
| 151 | + | A day after the pull request merges or closes, an hourly job retires the | |
| 152 | + | fork: its head is kept in the source as `refs/pull/<id>/head`, and its data | |
| 153 | + | is removed. Reads of a retired fork are answered from the source, and any | |
| 154 | + | write makes the fork again first. | |
| 155 | + | ||
| 156 | + | ## Merging | |
| 157 | + | ||
| 158 | + | The Artifacts binding reads but does not write, so `repos` lands a merge | |
| 159 | + | over git itself (`services/repos/src/land.rs`). It asks the fork for a pack | |
| 160 | + | with exactly the objects the target lacks, and streams it into a push to | |
| 161 | + | the target, without parsing it or holding it whole. | |
| 162 | + | ||
| 163 | + | `repos` also builds the test merge for every pull request, | |
| 164 | + | `refs/pull/<n>/merge`, without a sandbox, using a line-level merge on | |
| 165 | + | files up to 1 MiB (`test_merge.rs`). When a pull request is behind and the | |
| 166 | + | two sides changed different files, **Catch up with main** is done the same | |
| 167 | + | way (`catch_up.rs`). Testing several pull requests together is the merge | |
| 168 | + | queue's job ([Work and the merge queue](/architecture/work/#the-merge-queue)). | |
| 169 | + | ||
| 170 | + | ## Backups | |
| 171 | + | ||
| 172 | + | At 02:53 UTC, `repos` queues up to 200 repositories whose refs moved since | |
| 173 | + | their last backup, oldest first. The runner cuts them: each job gets a | |
| 174 | + | sandbox that clones with a read-only store credential valid for a few | |
| 175 | + | minutes, writes a `git bundle`, and uploads it in 32 MiB parts to the | |
| 176 | + | `g1t-backups` bucket. Every thirtieth backup is full and the rest are | |
| 177 | + | incremental. | |
| 178 | + | ||
| 179 | + | | Path | Holds | | |
| 180 | + | | --- | --- | | |
| 181 | + | | `backups/<repo id>/manifest.json` | The repository's bundles, in order | | |
| 182 | + | | `backups/<repo id>/<time>-full.bundle` | Every ref and object | | |
| 183 | + | | `backups/<repo id>/<time>-incr.bundle` | What changed since the bundle before | | |
| 184 | + | ||
| 185 | + | `scripts/ops/backup-restore-drill.mjs` restores from them, and | |
| 186 | + | `scripts/ops/restore-to-gitstore.mjs` can rebuild repositories into a | |
| 187 | + | self-hosted git store, which g1t.sh can then read from if Artifacts is | |
| 188 | + | unavailable (`fallback.rs`). | |
| 189 | + | ||
| 190 | + | ## SSH | |
| 191 | + | ||
| 192 | + | `crates/sshd` is an SSH front end for git that authenticates your key with | |
| 193 | + | `identity` and bridges to the store. It is not deployed: git on g1t is | |
| 194 | + | HTTPS only for now. You can already add deploy keys. |
| 1 | + | --- | |
| 2 | + | title: Identity and access | |
| 3 | + | description: How every kind of credential becomes one viewer, what the viewer carries, and how each service decides access with the same compiled-in rules instead of asking identity. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | import Aside from '../../../components/Aside.astro'; | |
| 7 | + | import Diagram from '../../../components/Diagram.astro'; | |
| 8 | + | ||
| 9 | + | `identity` (`services/identity`, Rust, D1 `g1t`) is the only service that | |
| 10 | + | knows about credentials. It holds accounts, sessions, workspaces and their | |
| 11 | + | members, teams, repository roles and invitations, access tokens, OAuth, | |
| 12 | + | device sign-in and two-factor authentication. Every other service learns | |
| 13 | + | who is asking from the viewer it is handed. | |
| 14 | + | ||
| 15 | + | <Diagram name="identity-flow" /> | |
| 16 | + | ||
| 17 | + | ## Credentials | |
| 18 | + | ||
| 19 | + | | Credential | Looks like | Used by | Resolved with | | |
| 20 | + | | --- | --- | --- | --- | | |
| 21 | + | | Session cookie | `g1t_session`, 64 hex characters, 30 days | The site | `user_for_session` | | |
| 22 | + | | Access token | `g1t_…`, personal or a workspace's, with scopes | The API, MCP, git, the CLI | `user_for_access_token` | | |
| 23 | + | | OAuth access token | Issued by `POST /oauth/token`, refreshed with `g1r_…` | MCP clients, the phone app, apps you authorise | `user_for_access_token` | | |
| 24 | + | | Job token | `G1T_TOKEN` in a workflow job | Actions, for that job alone | `user_for_access_token` | | |
| 25 | + | | Run credential | Made per sandbox run, up to 6 hours | Sandboxes | `user_for_access_token` | | |
| 26 | + | | git password | Your account's password | git, only without two-factor authentication | `user_for_git_credentials` | | |
| 27 | + | ||
| 28 | + | The CLI signs in with the device flow of RFC 8628 (`/device/code`, then | |
| 29 | + | `/device/token`) and keeps its token in the system keychain. Session | |
| 30 | + | cookies and access tokens are stored as their SHA-256, so neither can be | |
| 31 | + | read back. | |
| 32 | + | ||
| 33 | + | ## The viewer | |
| 34 | + | ||
| 35 | + | `identity` answers with a `User` (`crates/contracts/src/lib.rs`), and the | |
| 36 | + | edge hands it to every service it calls, as `viewer` in the call's | |
| 37 | + | arguments. It is `null` for someone signed out. | |
| 38 | + | ||
| 39 | + | | Field | Holds | | |
| 40 | + | | --- | --- | | |
| 41 | + | | `kind` | `User`, `Workspace`, `Agent` or `System`: what kind of principal this is | | |
| 42 | + | | `workspaces` | Each membership: the role, the workspace's base permission, organisation roles such as billing or security manager, and privileges | | |
| 43 | + | | `grants` | Roles on repositories, directly or through teams | | |
| 44 | + | | `acting` | For a job or run token: the narrower scope it acts within | | |
| 45 | + | ||
| 46 | + | A service never asks `identity` whether someone may do something. Each | |
| 47 | + | calls `can` in `g1t_contracts::access`, the same rules compiled into every | |
| 48 | + | Rust service, with `packages/contracts/src/access.ts` as the TypeScript | |
| 49 | + | mirror, which a test keeps identical. That takes `identity` out of every | |
| 50 | + | request after the first call, and means access is decided by the service | |
| 51 | + | that owns the data, next to it. | |
| 52 | + | ||
| 53 | + | <Aside title="Why the viewer can be trusted"> | |
| 54 | + | The viewer is passed as plain JSON, not signed. That is safe because | |
| 55 | + | services can be reached only through service bindings: none is on | |
| 56 | + | `workers.dev`, and the only routes in the core are `telemetry`'s and | |
| 57 | + | `og`'s public endpoints, which take no viewer. A Worker can only be | |
| 58 | + | called by the Workers bound to it, and those are the edge Workers and | |
| 59 | + | other services, which resolved the viewer themselves. | |
| 60 | + | </Aside> | |
| 61 | + | ||
| 62 | + | ## Roles | |
| 63 | + | ||
| 64 | + | | Level | Roles | | |
| 65 | + | | --- | --- | | |
| 66 | + | | Workspace | Owner and member, plus organisation roles (billing manager, security manager) and member privileges | | |
| 67 | + | | Repository | Read, Triage, Write, Maintain and Admin, given to a person, a team, or everyone in the workspace as its base permission | | |
| 68 | + | | Outside collaborators | A repository role for someone who is not a member | | |
| 69 | + | ||
| 70 | + | The rules live in `crates/contracts/src/access.rs`; the grants live in | |
| 71 | + | `identity`. [Access and roles](/guides/access-and-roles/) explains each | |
| 72 | + | role from a person's side. | |
| 73 | + | ||
| 74 | + | ## Scopes | |
| 75 | + | ||
| 76 | + | A token can do no more than its owner, and often less. Scopes are written | |
| 77 | + | `resource:level`, where `admin` includes `write` and `write` includes | |
| 78 | + | `read` (`crates/contracts/src/scopes.rs`). A token's reach is every | |
| 79 | + | workspace its owner belongs to, one of them, or none. The API checks the | |
| 80 | + | scopes before it calls any service, and the MCP server lists only the | |
| 81 | + | tools they allow ([How a request is served](/architecture/requests/#scopes)). | |
| 82 | + | ||
| 83 | + | ## Sign-in hardening | |
| 84 | + | ||
| 85 | + | | | | | |
| 86 | + | | --- | --- | | |
| 87 | + | | Two-factor | TOTP (RFC 6238), with ten recovery codes stored as SHA-256. A workspace can require it. | | |
| 88 | + | | Sudo mode | Sensitive changes ask you to prove it is you again (`Identity::proof`) | | |
| 89 | + | | Lockouts | Wrong passwords are throttled per account and per address | | |
| 90 | + | | Unconfirmed email | Can sign in, but cannot push | | |
| 91 | + | | GitHub sign-in | A GitHub App's OAuth web flow with PKCE, keyed on your GitHub account's numeric id | | |
| 92 | + | | Staff | `sudo.g1t.sh` checks the Cloudflare Access token against a staff list, and is the only caller of `identity`'s admin methods | | |
| 93 | + | ||
| 94 | + | ## Events | |
| 95 | + | ||
| 96 | + | `identity` publishes `user.*`, `workspace.*`, `team.*` and `invite.*` | |
| 97 | + | events. Lifecycle events, such as a workspace being | |
| 98 | + | deleted or renamed, go to every subscriber, so each service cleans up or | |
| 99 | + | renames its own data ([Events](/architecture/events/#subscriber-queues)). |
| 1 | + | --- | |
| 2 | + | title: Architecture overview | |
| 3 | + | description: How g1t is put together, from the four public Workers at the edge to the nineteen services behind them, the event bus that connects them, and the platform each one runs on. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | import { CardGrid } from '@astrojs/starlight/components'; | |
| 7 | + | import Aside from '../../../components/Aside.astro'; | |
| 8 | + | import Diagram from '../../../components/Diagram.astro'; | |
| 9 | + | import LinkCard from '../../../components/LinkCard.astro'; | |
| 10 | + | ||
| 11 | + | g1t is a set of small programs, each with one job, that run on Cloudflare | |
| 12 | + | Workers. A few face the internet. Almost all the rest can be reached only by the | |
| 13 | + | Workers that are bound to them. Every service that keeps data keeps it in its own | |
| 14 | + | database, and services learn what happened elsewhere from events instead of | |
| 15 | + | calling each other to find out. | |
| 16 | + | ||
| 17 | + | This section is for developers: people building on g1t, contributing to it, | |
| 18 | + | or running it themselves. It names the files, bindings and limits that | |
| 19 | + | decide how g1t behaves, so you can read the code next to it. | |
| 20 | + | ||
| 21 | + | <Diagram name="platform-map" /> | |
| 22 | + | ||
| 23 | + | ## The layers | |
| 24 | + | ||
| 25 | + | | Layer | What is in it | Reached by | | |
| 26 | + | | --- | --- | --- | | |
| 27 | + | | **Clients** | Browsers, the [desktop](/guides/desktop/) and [phone](/guides/mobile/) apps, git, the [CLI](/guides/cli/), coding agents, [self-hosted runners](/guides/self-hosted-runners/), and visitors to apps deployed on g1t. | The internet | | |
| 28 | + | | **Edge** | Four Workers with public routes: the site (`apps/web`), the API and MCP server (`apps/api`), the status page (`apps/status`) and the app dispatcher (`services/pages`). | Their hostnames | | |
| 29 | + | | **Core** | Nineteen services, each a Worker with `workers_dev` off. Seventeen have no route at all; `telemetry` takes OTLP at `otel.g1t.sh` and `og` draws social cards at `og.g1t.sh`. | Service bindings from the Workers that name them | | |
| 30 | + | | **Platform** | D1, Durable Objects, Queues, R2, KV, Artifacts for git, and Containers for sandboxes. | The one service that owns each, through its bindings | | |
| 31 | + | ||
| 32 | + | The edge does three things: it works out who is asking, enforces rate | |
| 33 | + | limits, and turns a request into calls to the services that own the | |
| 34 | + | answer. It keeps no state of its own. The services do the work and own the | |
| 35 | + | data. | |
| 36 | + | ||
| 37 | + | ## Where each hostname goes | |
| 38 | + | ||
| 39 | + | | Hostname | Worker | Serves | | |
| 40 | + | | --- | --- | --- | | |
| 41 | + | | `g1t.sh` | `g1t` (`apps/web`) | The site, git over HTTPS, and the package registries, which it passes whole to `repos` and `packages` | | |
| 42 | + | | `g1tusercontent.com` | `g1t` (`apps/web`) | Raw files, avatars and attachments, on a domain that never sees the session cookie | | |
| 43 | + | | `api.g1t.sh`, `mcp.g1t.sh` | `g1t-api` (`apps/api`) | The REST API, the MCP server, OAuth, the runner protocol, and OTLP | | |
| 44 | + | | `*.g1t.page` and custom domains | `g1t-pages` (`services/pages`) | Apps deployed on g1t | | |
| 45 | + | | `otel.g1t.sh` | `g1t-telemetry` | OpenTelemetry from your software | | |
| 46 | + | | `status.g1t.sh` | `g1t-status` (`apps/status`) | The status page, with a database of its own so it answers when the rest does not | | |
| 47 | + | | `docs.g1t.sh` | `g1t-docs` (`apps/docs`) | These pages | | |
| 48 | + | ||
| 49 | + | `sudo.g1t.sh`, g1t's staff console, sits behind Cloudflare Access and is | |
| 50 | + | not part of what you can call. | |
| 51 | + | ||
| 52 | + | ## Calls, events and ownership | |
| 53 | + | ||
| 54 | + | Three rules shape the code: | |
| 55 | + | ||
| 56 | + | - **A service owns its data.** Each of the sixteen D1 databases is bound | |
| 57 | + | by exactly one Worker. To read another service's data, you call that | |
| 58 | + | service. Three stores are deliberately shared, each with one writer: | |
| 59 | + | the avatars KV, the custom domains KV, and the Actions cache bucket | |
| 60 | + | ([Data and storage](/architecture/data/)). | |
| 61 | + | - **Calls are for answers; events are for news.** When the site needs a pull | |
| 62 | + | request, it calls `work`. When a push lands, `repos` does not call the | |
| 63 | + | dozen services that care: it publishes `git.push`, and each of them gets | |
| 64 | + | its own copy on its own queue ([Events](/architecture/events/)). | |
| 65 | + | - **The viewer travels with the call.** The edge resolves the person or | |
| 66 | + | token once, and passes the result to every service in the call's | |
| 67 | + | arguments. Each service decides access itself with the same rules, | |
| 68 | + | compiled in, rather than asking `identity` again | |
| 69 | + | ([Identity and access](/architecture/identity/)). | |
| 70 | + | ||
| 71 | + | A call is an HTTP request over a service binding: `POST /rpc/<method>` | |
| 72 | + | with a JSON body. An expected failure, such as "not found" or "not | |
| 73 | + | allowed", comes back as a value, `{ "ok": false, "error": { "code", | |
| 74 | + | "message" } }`, with status 200. Only something unexpected, a thrown error | |
| 75 | + | or a busy store, becomes a non-200 status | |
| 76 | + | ([Services and contracts](/architecture/services/)). | |
| 77 | + | ||
| 78 | + | ## Rust and TypeScript | |
| 79 | + | ||
| 80 | + | The services that hold accounts, repositories, work, events, money and the | |
| 81 | + | API are written in Rust, compiled to WebAssembly with `worker-build`. The | |
| 82 | + | web apps and the services built around Durable Objects and live rooms are | |
| 83 | + | TypeScript. | |
| 84 | + | ||
| 85 | + | | Rust | TypeScript | | |
| 86 | + | | --- | --- | | |
| 87 | + | | `identity`, `repos`, `work`, `events`, `actions`, `packages`, `security`, `search`, `billing`, `integrations`, `webhooks`, and `apps/api` | `runner`, `deployments`, `projects`, `chat`, `artifacts`, `notify`, `telemetry`, `og`, `pages`, and `apps/web`, `apps/sudo`, `apps/status` | | |
| 88 | + | ||
| 89 | + | Both sides share their types through contracts: `crates/contracts` in Rust | |
| 90 | + | and `packages/contracts` in TypeScript. Where the two must agree exactly, | |
| 91 | + | such as the access rules and the scopes, a Rust test reads the TypeScript | |
| 92 | + | file and fails if they differ. | |
| 93 | + | ||
| 94 | + | ## One codebase, two homes | |
| 95 | + | ||
| 96 | + | The same source runs on g1t.sh and in a [self-hosted | |
| 97 | + | installation](/guides/self-hosting/). Each Cloudflare product a service | |
| 98 | + | uses sits behind a port, and the self-hosted build binds an open | |
| 99 | + | adapter in its place: bare repositories served by `git http-backend` for | |
| 100 | + | Artifacts, any S3 store for R2, Mailpit for email. The service code does | |
| 101 | + | not change. [Data and storage](/architecture/data/#ports-and-adapters) | |
| 102 | + | shows each one. | |
| 103 | + | ||
| 104 | + | <Aside type="tip" title="Read the code next to this"> | |
| 105 | + | Every unit g1t deploys is listed in `deploy/stack.jsonc`, with its | |
| 106 | + | language, its stage and what a self-hosted installation does with it. | |
| 107 | + | Each service's `wrangler.jsonc` lists its bindings: its database, the | |
| 108 | + | services it calls, and the queues it reads. | |
| 109 | + | </Aside> | |
| 110 | + | ||
| 111 | + | ## In this section | |
| 112 | + | ||
| 113 | + | <CardGrid> | |
| 114 | + | <LinkCard title="How a request is served" href="/architecture/requests/" icon="route" description="The site and the API, from the first byte to Server-Timing." /> | |
| 115 | + | <LinkCard title="Services and contracts" href="/architecture/services/" icon="boxes" description="Every unit, what it owns, what it calls, and how calls work." /> | |
| 116 | + | <LinkCard title="Git" href="/architecture/git/" icon="git-branch" description="Where repositories live, the push path, caches and forks." /> | |
| 117 | + | <LinkCard title="Events" href="/architecture/events/" icon="radio-tower" description="The bus, the log, fan-out, webhooks and following live." /> | |
| 118 | + | <LinkCard title="Work and the merge queue" href="/architecture/work/" icon="git-merge" description="Issues, pull requests, checks, merging and the queue." /> | |
| 119 | + | <LinkCard title="Actions, runners and sandboxes" href="/architecture/compute/" icon="cpu" description="Workflow jobs, hosted and self-hosted, and what walls them in." /> | |
| 120 | + | <LinkCard title="Deployments and telemetry" href="/architecture/deployments/" icon="rocket" description="Apps on g1t.page, packages, and observability." /> | |
| 121 | + | <LinkCard title="Data and storage" href="/architecture/data/" icon="database" description="Who owns what, migrations, and the ports that make self-hosting work." /> | |
| 122 | + | <LinkCard title="Identity and access" href="/architecture/identity/" icon="key-round" description="Credentials, the viewer, scopes, and where access is decided." /> | |
| 123 | + | <LinkCard title="How g1t ships" href="/architecture/shipping/" icon="ship" description="The monorepo, builds, staged deploys, and keeping these pages true." /> | |
| 124 | + | </CardGrid> |
| 1 | + | --- | |
| 2 | + | title: How a request is served | |
| 3 | + | description: What happens between a request reaching g1t.sh or api.g1t.sh and its answer leaving, step by step, with the deadlines, caches, rate limits and timings along the way. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | import Aside from '../../../components/Aside.astro'; | |
| 7 | + | import Diagram from '../../../components/Diagram.astro'; | |
| 8 | + | ||
| 9 | + | Two Workers answer almost everything you send g1t: the site, `apps/web` | |
| 10 | + | on `g1t.sh`, and the API, `apps/api` on `api.g1t.sh` and `mcp.g1t.sh`. | |
| 11 | + | Neither keeps data. Each works out who is asking, then calls the services | |
| 12 | + | that own the answer. | |
| 13 | + | ||
| 14 | + | ## The site | |
| 15 | + | ||
| 16 | + | `apps/web/workers/app.ts` is the site's entry point. It sorts each request | |
| 17 | + | before any page code runs: | |
| 18 | + | ||
| 19 | + | 1. A request for `g1tusercontent.com` goes to `serveUsercontent`, which | |
| 20 | + | never reads or sets the session cookie. | |
| 21 | + | 2. `go get` lookups get their meta page. | |
| 22 | + | 3. **git over HTTPS** (paths ending in `info/refs`, `git-upload-pack` or | |
| 23 | + | `git-receive-pack`) passes the git rate limits, then goes to `repos` | |
| 24 | + | whole, through `env.REPOS.fetch`. The site never parses git | |
| 25 | + | ([Git](/architecture/git/)). | |
| 26 | + | 4. **Package registries** (`/v2/`, `/-/npm/`, `/-/cargo/` and the others) | |
| 27 | + | go to `packages` whole. A 404 is tried once more under the workspace's | |
| 28 | + | current name, so a renamed workspace's old URLs keep working. | |
| 29 | + | 5. Everything else passes the page rate limit and reaches React Router. | |
| 30 | + | ||
| 31 | + | ### A page | |
| 32 | + | ||
| 33 | + | <Diagram name="page-request" /> | |
| 34 | + | ||
| 35 | + | The root route's middleware, `viewerMiddleware` in | |
| 36 | + | `apps/web/app/lib/session.server.ts`, resolves the viewer once (steps 2 | |
| 37 | + | and 3). The `g1t_session` cookie is 64 hex characters, `HttpOnly`, | |
| 38 | + | `Secure` and `SameSite=Lax`, and lasts 30 days; `identity` stores only its | |
| 39 | + | SHA-256. A request can send `Authorization: Bearer` with an access token | |
| 40 | + | that has the **website** permission instead of the cookie. The same | |
| 41 | + | middleware enforces the email confirmation, workspace and Code access | |
| 42 | + | gates. | |
| 43 | + | ||
| 44 | + | Loaders then call their services together (steps 4 to 7). They use the | |
| 45 | + | typed clients in `@g1t/contracts`, built in `app/lib/services.server.ts`, | |
| 46 | + | and pass the viewer in each call's arguments. Nothing waits on a call it | |
| 47 | + | does not need. | |
| 48 | + | ||
| 49 | + | ### Deadlines | |
| 50 | + | ||
| 51 | + | Each binding is wrapped by `instrumented()` in `app/lib/perf.server.ts`, | |
| 52 | + | which gives every call a deadline (`app/lib/deadline.ts`). A call that | |
| 53 | + | outlives it is aborted and throws `DeadlineExceeded`, so one slow service | |
| 54 | + | cannot hold a page forever. | |
| 55 | + | ||
| 56 | + | | Call | Deadline | | |
| 57 | + | | --- | --- | | |
| 58 | + | | A read | 15 s | | |
| 59 | + | | A write, or a read named in `SLOW_READS` | 60 s | | |
| 60 | + | | A long job, named in `LONG_CALLS` | 600 s | | |
| 61 | + | ||
| 62 | + | ### Reading your own writes | |
| 63 | + | ||
| 64 | + | D1 has read replicas. The site uses D1's Sessions API with the services | |
| 65 | + | that support it (`identity`, `repos`, `work`, `search`, `billing`, | |
| 66 | + | `projects`, `deployments`) by sending an `x-d1-bookmark` header on each | |
| 67 | + | call. The latest bookmark is kept in the `g1t_d1` cookie for 5 minutes, | |
| 68 | + | and for 30 seconds after a write, reads go to the primary. You never see a | |
| 69 | + | page older than a change you just made. The service side is | |
| 70 | + | `crates/kit/src/d1.rs`; TypeScript services use | |
| 71 | + | `packages/contracts/src/d1.ts`. | |
| 72 | + | ||
| 73 | + | ### The public page cache | |
| 74 | + | ||
| 75 | + | A signed-out `GET` with no cookie and no `Authorization` header, for a | |
| 76 | + | public page, can be answered from the colo's cache. It is fresh for 30 | |
| 77 | + | seconds, then served stale while it refreshes in the background, for up | |
| 78 | + | to 5 minutes. Before a cached repository page is served, the site asks | |
| 79 | + | `repos` whether the repository is still public, so making it private | |
| 80 | + | takes effect at once. | |
| 81 | + | ||
| 82 | + | ### Server-Timing | |
| 83 | + | ||
| 84 | + | Every page carries a `Server-Timing` header: the total, each loader, each | |
| 85 | + | service's calls and time, and the database time each service reports | |
| 86 | + | about itself (`svc`, `db` and `rpc`, from `Served` in | |
| 87 | + | `crates/kit/src/d1.rs`). A cached page says | |
| 88 | + | `cache;desc="hit, Ns old"`. Open your browser's network panel on any page | |
| 89 | + | to see where its time went. | |
| 90 | + | ||
| 91 | + | <Aside title="Where it runs"> | |
| 92 | + | Every Worker runs with placement off, in the data centre nearest the | |
| 93 | + | visitor. `scripts/perf/placement-probe.mjs` measures the alternative, | |
| 94 | + | and CONTRIBUTING.md's **Speed** section has the rules this design | |
| 95 | + | follows: start calls together, add no round trips, and never wake a | |
| 96 | + | Durable Object for a read D1 can answer. | |
| 97 | + | </Aside> | |
| 98 | + | ||
| 99 | + | ## The API and MCP | |
| 100 | + | ||
| 101 | + | `apps/api` serves both `api.g1t.sh` and `mcp.g1t.sh` from one Worker; a | |
| 102 | + | host starting `mcp.` is the MCP server. It sorts each request in this | |
| 103 | + | order (`apps/api/src/lib.rs`): | |
| 104 | + | ||
| 105 | + | 1. OTLP at `/otlp/v1/*` goes to `telemetry` as it is. | |
| 106 | + | 2. Requests that bring their own credentials are handled before any g1t | |
| 107 | + | token is read: the Stripe webhook, inbound integration hooks at | |
| 108 | + | `/hooks/<id>`, deployment build jobs, the Actions toolkit and OIDC | |
| 109 | + | issuer, a job's artifacts, cache and brokered requests, and the runner | |
| 110 | + | protocol at `/runners/*`. | |
| 111 | + | 3. Rate limits. | |
| 112 | + | 4. The token. | |
| 113 | + | 5. OAuth. | |
| 114 | + | 6. MCP, when the host says so. | |
| 115 | + | 7. The REST routes. | |
| 116 | + | ||
| 117 | + | <Diagram name="api-request" /> | |
| 118 | + | ||
| 119 | + | A token is sent as `Authorization: Bearer g1t_…` (or `token g1t_…`) and | |
| 120 | + | resolved by `identity` (step 3). A wrong or expired one gets a 401 with a | |
| 121 | + | `www-authenticate` header that points at the OAuth metadata, as RFC 9728 | |
| 122 | + | describes, so an MCP client can start signing in by itself. | |
| 123 | + | ||
| 124 | + | ### Scopes | |
| 125 | + | ||
| 126 | + | Scopes are written `resource:level`, and `admin` includes `write`, which | |
| 127 | + | includes `read`. They are defined in `crates/contracts/src/scopes.rs`, and | |
| 128 | + | `packages/contracts/src/scopes.ts` mirrors them. The token's scopes decide | |
| 129 | + | which operations it may call (step 5), and the MCP server lists only the | |
| 130 | + | tools those scopes allow. A workflow job's token carries the job's own, | |
| 131 | + | narrower scope. | |
| 132 | + | ||
| 133 | + | ### One table for every operation | |
| 134 | + | ||
| 135 | + | `apps/api/src/operations.rs` lists every operation once. The REST routes, | |
| 136 | + | the MCP tools and the OpenAPI document at | |
| 137 | + | [`/openapi.json`](https://api.g1t.sh/openapi.json) are all generated from | |
| 138 | + | it; an operation without a description does not compile. The docs' | |
| 139 | + | [API reference](/reference/api/) is built from that document, and the | |
| 140 | + | CLI's command table from the reference, so the four cannot disagree. | |
| 141 | + | ||
| 142 | + | ### snake_case at the edge | |
| 143 | + | ||
| 144 | + | Inside g1t, contracts are camelCase. The API's `reply()` turns every | |
| 145 | + | answer to snake_case on the way out (`g1t_kit::wire::snake_case`), leaving | |
| 146 | + | user-named keys such as inputs, env, secrets and headers as they were | |
| 147 | + | written. Errors keep one shape, `{"error": {"code", "message"}}`, with the | |
| 148 | + | status that the failure's code maps to. | |
| 149 | + | ||
| 150 | + | ### OAuth 2.1 | |
| 151 | + | ||
| 152 | + | The API is the OAuth issuer. It supports authorization code with PKCE for | |
| 153 | + | public clients only, with no client secrets, and dynamic registration at | |
| 154 | + | `POST /oauth/register`, whose client id encodes the client's name and | |
| 155 | + | redirect addresses. The person approves on the site, at | |
| 156 | + | `g1t.sh/oauth/authorize`; the token endpoint, `POST /oauth/token`, asks | |
| 157 | + | `identity` to exchange and refresh. Refresh tokens rotate. | |
| 158 | + | ||
| 159 | + | ### MCP | |
| 160 | + | ||
| 161 | + | The MCP server speaks streamable HTTP with no session state: each `POST` | |
| 162 | + | gets one JSON-RPC answer. It supports protocol versions 2025-06-18, | |
| 163 | + | 2025-03-26 and 2024-11-05. Its tools are resources with an `action`, as | |
| 164 | + | [MCP tools](/reference/mcp/) lists. | |
| 165 | + | ||
| 166 | + | ## Rate limits | |
| 167 | + | ||
| 168 | + | Every limit is a Workers rate-limit binding, and | |
| 169 | + | `packages/contracts/src/rate-limits.ts` is the table of record: a test | |
| 170 | + | checks every `wrangler.jsonc` against it. | |
| 171 | + | ||
| 172 | + | | Where | Limit | Counted by | | |
| 173 | + | | --- | --- | --- | | |
| 174 | + | | The API, without a token | 60 a minute | The client's address | | |
| 175 | + | | The API, with a token | 1,000 a minute | The token, hashed | | |
| 176 | + | | REST and MCP | Counted separately | | | |
| 177 | + | | Sandbox and runner reports | Not counted | | | |
| 178 | + | ||
| 179 | + | When the limiter itself cannot answer, the request goes through: the | |
| 180 | + | limits fail open. The [rate limits page](/reference/rate-limits/) lists | |
| 181 | + | every limit you can hit. |
| 1 | + | --- | |
| 2 | + | title: Services and contracts | |
| 3 | + | description: Every unit g1t deploys, what each one does, what it owns and what it calls, and how a call between two services works, down to the wire. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | g1t deploys twenty-five units, listed once in `deploy/stack.jsonc`. Each is | |
| 7 | + | a Worker with its own `wrangler.jsonc`, which names its database, its | |
| 8 | + | queues and the services it calls. This page puts all of them in one | |
| 9 | + | place. | |
| 10 | + | ||
| 11 | + | ## Every unit | |
| 12 | + | ||
| 13 | + | **Calls** lists the services each unit binds to, which are the only ones | |
| 14 | + | it can reach. A test, `npm run test:deploy`, reads every `wrangler.jsonc` | |
| 15 | + | and fails if this table and the bindings disagree. | |
| 16 | + | ||
| 17 | + | | Unit | Language | Stage | What it does | Owns | Calls | | |
| 18 | + | | --- | --- | --- | --- | --- | --- | | |
| 19 | + | | `apps/web` | TypeScript | front | The site on `g1t.sh`: pages, git over HTTPS and the package registries passed to their services, and `g1tusercontent.com` | The downloads bucket | `actions`, `artifacts`, `billing`, `chat`, `deployments`, `events`, `identity`, `integrations`, `notify`, `og`, `packages`, `projects`, `repos`, `search`, `security`, `telemetry`, `webhooks`, `work` | | |
| 20 | + | | `apps/api` | Rust | edge | REST, MCP, OAuth, the runner protocol, OTLP ingest, Actions toolkit and OIDC | Nothing; writes Actions cache and artifact files | `actions`, `artifacts`, `billing`, `deployments`, `events`, `identity`, `integrations`, `packages`, `projects`, `repos`, `runner`, `search`, `security`, `telemetry`, `webhooks`, `work` | | |
| 21 | + | | `apps/status` | TypeScript | edge | `status.g1t.sh`: probes every minute and keeps incidents | D1 `g1t-status` | `billing`, `repos` | | |
| 22 | + | | `apps/sudo` | TypeScript | front | g1t.sh's staff console, behind Cloudflare Access | Nothing | `actions`, `billing`, `events`, `identity`, `projects`, `status`, `telemetry` | | |
| 23 | + | | `apps/docs` | Astro | front | `docs.g1t.sh`, these pages | Nothing | none | | |
| 24 | + | | `services/identity` | Rust | core | Accounts, sessions, workspaces, members, teams, repository roles, tokens, OAuth, device sign-in, 2FA | D1 `g1t`, the avatars KV | `billing`, `events`, `projects`, `repos` | | |
| 25 | + | | `services/repos` | Rust | core | Repositories and git: smart HTTP, pre-receive checks, caches, forks, landing merges, releases, backups | D1 `g1t-repos`, the git store, the git cache KV, the packs and backups buckets | `billing`, `events`, `identity`, `security`, `work` | | |
| 26 | + | | `services/work` | Rust | core | Issues, pull requests, comments, reviews, labels, milestones, rulesets, checks, statuses, CODEOWNERS, the merge queue | D1 `g1t-work` | `actions`, `events`, `identity`, `repos` | | |
| 27 | + | | `services/events` | Rust | core | The event bus and log, the audit log, the inbox | D1 `g1t-events`, the `g1t-events` queue and its subscriber queues | `billing`, `identity`, `notify`, `repos`, `work` | | |
| 28 | + | | `services/actions` | Rust | core | Workflows: reading them, triggering runs, planning jobs, secrets, self-hosted runners, logs | D1 `g1t-actions`; lists and expires the Actions cache bucket | `billing`, `deployments`, `events`, `identity`, `notify`, `projects`, `repos`, `runner`, `work` | | |
| 29 | + | | `services/runner` | TypeScript | core | Sandboxes on Cloudflare Containers, one per job, with their egress | The Container classes | `actions`, `billing`, `deployments`, `events`, `identity`, `repos`, `work` | | |
| 30 | + | | `services/deployments` | TypeScript | core | Builds and serves apps on `g1t.page`, previews, custom domains | D1 `g1t-deployments`, the domains KV, the dispatch namespace | `actions`, `billing`, `events`, `identity`, `og`, `projects`, `repos`, `runner`, `telemetry`, `work` | | |
| 31 | + | | `services/pages` | TypeScript | edge | The dispatcher for `*.g1t.page` and custom domains | Nothing; reads the domains KV | none | | |
| 32 | + | | `services/packages` | Rust | core | npm, Cargo, Composer, Maven, NuGet, RubyGems, Go and container images | D1 `g1t-packages`, the packages bucket | `billing`, `events`, `identity`, `repos` | | |
| 33 | + | | `services/security` | Rust | core | Secret scanning, dependency alerts and updates, code scanning, the supply chain | D1 `g1t-security` | `actions`, `billing`, `events`, `identity`, `repos`, `runner`, `work` | | |
| 34 | + | | `services/search` | Rust | core | Site search over repositories, issues, pull requests, people and code | D1 `g1t-search`, its jobs queue | `billing`, `identity`, `repos`, `work` | | |
| 35 | + | | `services/projects` | TypeScript | core | Projects and what they depend on | D1 `g1t-projects` | `deployments`, `identity`, `og`, `packages`, `repos` | | |
| 36 | + | | `services/chat` | TypeScript | core | Channels, direct messages and threads, live | D1 `g1t-chat`, a Durable Object per channel | `identity`, `notify` | | |
| 37 | + | | `services/artifacts` | TypeScript | core | Docs written together, live, with their spaces, sharing and search | D1 `g1t-docs`, its files bucket, two Vectorize indexes, a Durable Object per open doc | `billing`, `events`, `identity`, `notify`, `repos`, `work` | | |
| 38 | + | | `services/notify` | TypeScript | core | Notifications, live pages, presence, browser and phone push | A Durable Object per person, per workspace and per repository | `identity` | | |
| 39 | + | | `services/telemetry` | TypeScript | core | Logs, traces, metrics and errors for projects, at `otel.g1t.sh` | A Durable Object per project | `billing` | | |
| 40 | + | | `services/billing` | Rust | core | Usage, holds and settlements, limits, Stripe | D1 `g1t-billing` | `identity`, `packages`, `repos` | | |
| 41 | + | | `services/integrations` | Rust | core | Model providers, alert sources, issue trackers, the GitHub App, Ask g1t | D1 `g1t-integrations` | `identity`, `repos`, `work` | | |
| 42 | + | | `services/webhooks` | Rust | core | Webhook subscriptions and signed deliveries with retries | D1 `g1t-webhooks` | `identity`, `repos` | | |
| 43 | + | | `services/og` | TypeScript | core | Social cards and page screenshots, at `og.g1t.sh` | The screenshots bucket | `billing`, `identity`, `projects`, `repos`, `work` | | |
| 44 | + | ||
| 45 | + | Calls go both ways in places, `repos` and `work` for one. That is allowed: | |
| 46 | + | bindings only have to respect the deploy stages, and every service is in | |
| 47 | + | the same one ([How g1t ships](/architecture/shipping/)). | |
| 48 | + | ||
| 49 | + | ## A call, on the wire | |
| 50 | + | ||
| 51 | + | A service binding is a direct connection from one Worker to another in the | |
| 52 | + | same data centre, with no trip over the public internet. g1t uses it as | |
| 53 | + | plain HTTP: | |
| 54 | + | ||
| 55 | + | ```http | |
| 56 | + | POST https://work/rpc/get_pull | |
| 57 | + | content-type: application/json | |
| 58 | + | ||
| 59 | + | {"repo": {"namespace": "acme", "name": "app"}, "number": 12, "viewer": {"id": "usr_…", "workspaces": […], …}} | |
| 60 | + | ``` | |
| 61 | + | ||
| 62 | + | The hostname is ignored; the method is the path. The arguments are that | |
| 63 | + | method's struct in the contracts, here `ViewArgs` in | |
| 64 | + | `crates/contracts/src/work.rs`. `viewer` is the person or token asking, or | |
| 65 | + | `null` for someone signed out. The answer is always JSON: | |
| 66 | + | ||
| 67 | + | ```json | |
| 68 | + | { "ok": true, "value": { … } } | |
| 69 | + | ``` | |
| 70 | + | ||
| 71 | + | or, for anything the caller should handle, still with status 200: | |
| 72 | + | ||
| 73 | + | ```json | |
| 74 | + | { "ok": false, "error": { "code": "not_found", "message": "No pull request #12." } } | |
| 75 | + | ``` | |
| 76 | + | ||
| 77 | + | | Status | Means | The caller gets | | |
| 78 | + | | --- | --- | --- | | |
| 79 | + | | 200 with `ok: true` | It worked | The value | | |
| 80 | + | | 200 with `ok: false` | An expected failure, with a code from `FailureCode` | A value to handle, such as `Result` in TypeScript or `Outcome` in Rust | | |
| 81 | + | | 404 | No such method | An error | | |
| 82 | + | | 503 | A store is busy, with `Retry-After` | An error | | |
| 83 | + | | 500 | Something threw | An error | | |
| 84 | + | ||
| 85 | + | On the Rust side, `g1t_kit::call(&binding, "method", &args)` sends a call | |
| 86 | + | and `rpc_method`, `args` and `reply` serve one (`crates/kit/src/lib.rs`). | |
| 87 | + | On the TypeScript side, the clients in `packages/contracts/src/clients.ts` | |
| 88 | + | do both. Two Workers expose Worker RPC entrypoints instead: `og`'s | |
| 89 | + | `Screenshots` and `status`'s `StatusAdmin`. | |
| 90 | + | ||
| 91 | + | There is no secret between services. A service can be called only | |
| 92 | + | through a binding, because none of them is published on `workers.dev` | |
| 93 | + | and only `og` and `telemetry` have routes, for their own public endpoints. | |
| 94 | + | That makes the binding list in each `wrangler.jsonc` the complete list of | |
| 95 | + | who can call it. | |
| 96 | + | ||
| 97 | + | ## Contracts | |
| 98 | + | ||
| 99 | + | Services share types, never code. Each one depends on the contracts, and | |
| 100 | + | none on another service's crate or package. | |
| 101 | + | ||
| 102 | + | | Package | Holds | | |
| 103 | + | | --- | --- | | |
| 104 | + | | `crates/contracts` (`g1t-contracts`) | A module per service with its data and its methods' arguments; `Outcome` and `Failure`; the event envelope and every event's data; `ROUTES`, which says which events each subscriber queue gets; `access`, the access rules every service compiles in; `scopes`; ids | | |
| 105 | + | | `packages/contracts` (`@g1t/contracts`) | The same for TypeScript: each service's API type (`IdentityApi`, `ReposApi`, …), typed clients, `Result` and `FailureCode`, events, scopes, rate limits, the D1 session helper. TypeScript source, with no build step. | | |
| 106 | + | | `crates/kit` (`g1t-kit`) | Plumbing for Rust Workers: the call protocol, D1 sessions and `Server-Timing` (`d1.rs`), rate limits that fail open, billing's platform pause, the snake_case converter, and handlers every service shares for renames, transfers and deletions | | |
| 107 | + | ||
| 108 | + | Where the two languages must agree exactly, a Rust test reads the | |
| 109 | + | TypeScript file and compares: the access table (`access.rs`), scopes, and | |
| 110 | + | the shapes that have shared fixtures in JSON. Keep both in the same pull | |
| 111 | + | request. | |
| 112 | + | ||
| 113 | + | ### Ids | |
| 114 | + | ||
| 115 | + | Ids are TypeIDs: a prefix that says what the thing is, then a UUIDv7 in | |
| 116 | + | Crockford base32, such as `evt_01jb…` for an event or `usr_…` for a person. | |
| 117 | + | They sort by the time they were made, and are monotonic within one process | |
| 118 | + | and millisecond (`crates/contracts/src/ids.rs`). | |
| 119 | + | ||
| 120 | + | ## Shared crates | |
| 121 | + | ||
| 122 | + | | Crate | What it is | | |
| 123 | + | | --- | --- | | |
| 124 | + | | `crates/rules` | The rulesets engine, with no I/O: which rules apply to a ref, and whether a push or a merge passes them | | |
| 125 | + | | `crates/scan` | Detectors for secrets, a git pack reader, lockfiles, OSV, the dependency graph, SPDX and SARIF | | |
| 126 | + | | `crates/secrets` | `Sealer`, AES-256-GCM with the row's id as associated data, and HMAC signatures | | |
| 127 | + | | `crates/blobstore` | The `BlobStore` port, with R2 and S3 adapters | | |
| 128 | + | | `crates/actions` | Workflow syntax, expressions, filters, matrices and cron, with no I/O, used by both the `actions` service and the sandbox | | |
| 129 | + | | `crates/runner` | `g1t-runner`, the binary every sandbox and self-hosted runner runs | | |
| 130 | + | | `crates/g1t` | [The g1t CLI](/guides/cli/) | | |
| 131 | + | | `crates/sshd` | An SSH front end for git, not deployed yet | | |
| 132 | + | ||
| 133 | + | ## Adding a service | |
| 134 | + | ||
| 135 | + | 1. Make it under `services/<name>`, with its `wrangler.jsonc`: its D1 | |
| 136 | + | database if it keeps data, and `"workers_dev": false`. | |
| 137 | + | 2. Put its types in `crates/contracts` or `packages/contracts`, and its | |
| 138 | + | client in `packages/contracts/src/clients.ts` if the site calls it. | |
| 139 | + | 3. Add it to `deploy/stack.jsonc` with its kind, stage, secrets and | |
| 140 | + | `self_host`, then to the table above and to [How a self-hosted g1t | |
| 141 | + | runs](/guides/self-hosting-architecture/#each-part). | |
| 142 | + | 4. If it reacts to events, give it a `g1t-events-<name>` queue and a | |
| 143 | + | `SUBSCRIBER_*` binding on `events`, and add its event types to `ROUTES` | |
| 144 | + | ([Events](/architecture/events/#subscriber-queues)). | |
| 145 | + | 5. Run `npm run test:deploy`. It says what is missing. | |
| 146 | + | ||
| 147 | + | [Deploy g1t to Cloudflare](/guides/deploy-to-cloudflare/) covers the | |
| 148 | + | manifest's fields and a unit's first deploy. |
| 1 | + | --- | |
| 2 | + | title: How g1t ships | |
| 3 | + | description: The monorepo's layout, how each kind of unit is built, how a push to main becomes a staged deploy on g1t's own Actions, the tests that guard it, and how this section stays true. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | import Aside from '../../../components/Aside.astro'; | |
| 7 | + | import Diagram from '../../../components/Diagram.astro'; | |
| 8 | + | ||
| 9 | + | g1t is built on g1t. Its source is at | |
| 10 | + | [g1t.sh/flagon-io/g1t](https://g1t.sh/flagon-io/g1t), every change is a | |
| 11 | + | pull request with required checks, and its own workflows deploy it. | |
| 12 | + | ||
| 13 | + | ## The repository | |
| 14 | + | ||
| 15 | + | | Folder | Holds | | |
| 16 | + | | --- | --- | | |
| 17 | + | | `apps/` | What people open: `web` (the site), `api`, `status`, `sudo`, `docs`, and the `desktop` (Tauri) and `mobile` (Expo) apps | | |
| 18 | + | | `services/` | The services, one folder each, with their `wrangler.jsonc` and `migrations` | | |
| 19 | + | | `crates/` | Shared Rust: `contracts`, `kit`, `rules`, `scan`, `secrets`, `blobstore`, `actions`, the `runner` binary and the `g1t` CLI | | |
| 20 | + | | `packages/` | Shared TypeScript: `contracts` and `theme` (the design tokens) | | |
| 21 | + | | `deploy/` | `stack.jsonc`, the inventory; `self-host`, the Docker Compose installation; `runner` and `sandbox`, the self-hosted runner's images | | |
| 22 | + | | `scripts/` | `deploy.mjs` and its helpers, releases, operations | | |
| 23 | + | | `plugins/` | The [Claude Code plugin](/guides/claude-code-plugin/), and setup for other coding agents | | |
| 24 | + | | `.g1t/workflows/` | CI, deploys and releases, run by g1t Actions | | |
| 25 | + | | `conformance/` | Checks that a job's environment matches what workflows expect | | |
| 26 | + | ||
| 27 | + | ## Building | |
| 28 | + | ||
| 29 | + | `deploy/stack.jsonc` gives each unit a kind, and each kind builds its own | |
| 30 | + | way: | |
| 31 | + | ||
| 32 | + | | Kind | Built by | Units | | |
| 33 | + | | --- | --- | --- | | |
| 34 | + | | `rust-worker` | `scripts/build-rust-worker.mjs`: `worker-build` to WebAssembly, then `wasm-opt` | The Rust services and `apps/api` | | |
| 35 | + | | `ts-worker` | Wrangler's bundler | The TypeScript services | | |
| 36 | + | | `react-router` | `vite build`, then Wrangler | `apps/web`, `apps/sudo` | | |
| 37 | + | | `astro` | `astro build` | `apps/docs` | | |
| 38 | + | ||
| 39 | + | The runner's image is built in two layers: a large base with every | |
| 40 | + | toolchain, rebuilt only when its folder changes and recorded in | |
| 41 | + | `services/runner/base.json`, and a thin layer on top with the static | |
| 42 | + | `g1t-runner` binary. | |
| 43 | + | ||
| 44 | + | ## Deploying | |
| 45 | + | ||
| 46 | + | <Diagram name="deploy-stages" /> | |
| 47 | + | ||
| 48 | + | A push to `main` starts `.g1t/workflows/deploy.yml`, one run at a time, | |
| 49 | + | never cancelled: | |
| 50 | + | ||
| 51 | + | 1. **Check and plan.** The manifest is checked against every config and | |
| 52 | + | `npm run test:deploy` runs. Beside it, `scripts/deploy.mjs plan` works | |
| 53 | + | out which units changed since the commit each one is running. | |
| 54 | + | 2. **Migrations.** Every pending D1 migration, before any code. | |
| 55 | + | 3. **Core, edge, front.** The units in each stage deploy in parallel; a | |
| 56 | + | stage starts only when the one before succeeded. A unit binds only to | |
| 57 | + | units in its own stage or an earlier one, which a test checks, so new | |
| 58 | + | code never calls a service that has not shipped yet. | |
| 59 | + | 4. **Smoke.** `scripts/ops/smoke.mjs` checks that sign-in and sign-up still | |
| 60 | + | work, without making an account. | |
| 61 | + | ||
| 62 | + | Each run is one `production` deployment, shown on the commit as the | |
| 63 | + | `deploy / production` status. Only units whose inputs changed are | |
| 64 | + | deployed: their folder, or a crate or package they are built from, which | |
| 65 | + | `deploy.mjs` reads from Cargo's and npm's workspace metadata. | |
| 66 | + | [Deploy g1t to Cloudflare](/guides/deploy-to-cloudflare/) covers the | |
| 67 | + | tool, rollbacks and a first deploy. | |
| 68 | + | ||
| 69 | + | ## Tests that guard the architecture | |
| 70 | + | ||
| 71 | + | Some of what this section describes is enforced by tests, so a change that | |
| 72 | + | breaks it fails CI: | |
| 73 | + | ||
| 74 | + | | Test | Fails when | | |
| 75 | + | | --- | --- | | |
| 76 | + | | `npm run test:deploy` | A unit is missing from `stack.jsonc`, a binding crosses stages the wrong way, a unit is missing from [How a self-hosted g1t runs](/guides/self-hosting-architecture/), or [Services and contracts](/architecture/services/) disagrees with a `wrangler.jsonc` | | |
| 77 | + | | `cargo test -p g1t-contracts` | The Rust and TypeScript access rules or scopes differ | | |
| 78 | + | | `cargo test -p g1t-api openapi` | The docs' copy of the OpenAPI document is out of date | | |
| 79 | + | | `app/lib/front-door-limits.test.ts` in `apps/web` | A `wrangler.jsonc` and `packages/contracts/src/rate-limits.ts` disagree | | |
| 80 | + | | `npm test` in `apps/docs` | A diagram's arrow names a box that does not exist, two boxes overlap, or a label does not fit its box | | |
| 81 | + | ||
| 82 | + | CI (`.g1t/workflows/ci.yml`) runs the Rust tests, the TypeScript checks and | |
| 83 | + | tests, and builds the site, sudo and these docs on every pull request. | |
| 84 | + | ||
| 85 | + | ## Keeping this section true | |
| 86 | + | ||
| 87 | + | These pages describe the code on `main`. When you change how g1t is put | |
| 88 | + | together, change them in the same pull request: | |
| 89 | + | ||
| 90 | + | | If you change | Update | | |
| 91 | + | | --- | --- | | |
| 92 | + | | A service's bindings, storage or job | Its row in [Services and contracts](/architecture/services/#every-unit) and [Data and storage](/architecture/data/) | | |
| 93 | + | | A request's path, a limit or a deadline | [How a request is served](/architecture/requests/) | | |
| 94 | + | | An event type or a route to a subscriber | [Events](/architecture/events/) | | |
| 95 | + | | The push, fetch or merge path | [Git](/architecture/git/) and [Work and the merge queue](/architecture/work/) | | |
| 96 | + | | Sandboxes, runners or egress | [Actions, runners and sandboxes](/architecture/compute/) | | |
| 97 | + | | A new dependency on a Cloudflare product | Its port in [Data and storage](/architecture/data/#ports-and-adapters) | | |
| 98 | + | ||
| 99 | + | Diagrams are data, in `apps/docs/src/diagrams/architecture.ts`. Change the | |
| 100 | + | spec, and both the picture and the page's Markdown copy change with it. | |
| 101 | + | ||
| 102 | + | <Aside title="Run it yourself"> | |
| 103 | + | `docker compose -f deploy/self-host/docker-compose.yml up --build` | |
| 104 | + | builds every unit from source and starts the site on port 8787 and the | |
| 105 | + | API on 8789. [Run g1t yourself](/guides/self-hosting/) has the rest. | |
| 106 | + | </Aside> |
| 1 | + | --- | |
| 2 | + | title: Work and the merge queue | |
| 3 | + | description: How the work service keeps issues, pull requests, checks and rulesets, coordinates sessions and claims, decides and lands a merge, and tests many changes at once in the merge queue. | |
| 4 | + | --- | |
| 5 | + | ||
| 6 | + | import Aside from '../../../components/Aside.astro'; | |
| 7 | + | import Diagram from '../../../components/Diagram.astro'; | |
| 8 | + | ||
| 9 | + | `work` (`services/work`, Rust) keeps everything about a change that is not | |
| 10 | + | the code itself: issues, pull requests, comments, reviews, labels, | |
| 11 | + | milestones, rulesets, checks and commit statuses, CODEOWNERS, and the | |
| 12 | + | merge queue. Its database is `g1t-work`. The code lives in `repos`, and | |
| 13 | + | `work` asks `repos` whenever it needs it. | |
| 14 | + | ||
| 15 | + | ## Issues and pull requests | |
| 16 | + | ||
| 17 | + | Issues and pull requests share one sequence of numbers per repository | |
| 18 | + | (the `counters` table), so `#12` names exactly one of them. A pull request | |
| 19 | + | records where its head is: a fork of its own, `pulls--<id>` in the git | |
| 20 | + | store, or a branch in the repository ([Git](/architecture/git/#pull-request-forks)). | |
| 21 | + | ||
| 22 | + | `work` reads every push from its event queue. A push to a pull request's | |
| 23 | + | head updates it: the new head, the files it changes, whether it is behind | |
| 24 | + | the base, and which other open pull requests change the same files, the | |
| 25 | + | overlap shown on each. | |
| 26 | + | ||
| 27 | + | When a pull request merges, its issue closes and records which pull | |
| 28 | + | request resolved it, and the issue's other open pull requests close as | |
| 29 | + | superseded ([How g1t works](/concepts/overview/#several-pull-requests-for-one-issue)). | |
| 30 | + | ||
| 31 | + | ## Coordination | |
| 32 | + | ||
| 33 | + | Many agents and people can work on one repository at once. `work` keeps | |
| 34 | + | what lets them see each other, around the issue, which is the unit of | |
| 35 | + | work (`services/work/src/coordination/`): | |
| 36 | + | ||
| 37 | + | | Table | Holds | Module | | |
| 38 | + | | --- | --- | --- | | |
| 39 | + | | `sessions` | One run of someone's own harness (`claude_code`, `codex`, `cursor`, `copilot` or `other`), attached to an issue, a pull request and a branch | `sessions.rs` | | |
| 40 | + | | `session_entries` | What a session recorded: prompts, reasoning, tool calls, notes, each with an idempotency `key` | `sessions.rs` | | |
| 41 | + | | `claims` | An advisory, expiring "I'm working on #12" | `claims.rs` | | |
| 42 | + | | Issues' `blocked_by` | The issues that have to land first, refused if it would make a cycle | `dependencies.rs` | | |
| 43 | + | | `overlap_pairs` | Pairs of open pull requests changing the same files, announced once per pair | `overlaps.rs` | | |
| 44 | + | ||
| 45 | + | <Diagram name="coordination" /> | |
| 46 | + | ||
| 47 | + | Three choices shape it: | |
| 48 | + | ||
| 49 | + | - **Nothing is a lock.** A claim is advice: anyone may claim an issue | |
| 50 | + | someone else holds, is told about them, and decides. Nothing is ever | |
| 51 | + | refused because of a claim (step 9 in the diagram). | |
| 52 | + | - **Nothing waits on a harness.** No cron sweeps for quiet sessions or | |
| 53 | + | stale claims. The next read or change of a repository's coordination | |
| 54 | + | finds them (`sweep`) and announces them then: a session quiet for 15 | |
| 55 | + | minutes is idle, one quiet for 2 hours has ended, and a claim not renewed | |
| 56 | + | within its time (30 minutes unless asked, at most 240) has expired. | |
| 57 | + | - **Everything can be sent twice.** Starting a session is idempotent on the | |
| 58 | + | harness's own session id, and each entry on its `key`, so a hook can call | |
| 59 | + | every time without checking first, and a retry after a lost answer does | |
| 60 | + | no harm. | |
| 61 | + | ||
| 62 | + | `get_work` gathers an issue's whole context in one call (`context.rs`): the | |
| 63 | + | issue, its definition of done, the required checks, claims, sessions, | |
| 64 | + | pull requests with their checks, overlaps and place in the queue, what it | |
| 65 | + | is blocked by and blocks, its decision records, and its latest | |
| 66 | + | coordination events. A session's entries on a pull request are what | |
| 67 | + | **Blame** reads to show the agent's account of a line ([Sessions and | |
| 68 | + | why-blame](/guides/why-blame/)). [Sessions, claims and | |
| 69 | + | dependencies](/guides/sessions/) has the calls. | |
| 70 | + | ||
| 71 | + | ## Checks and statuses | |
| 72 | + | ||
| 73 | + | Anything can report on a commit: a workflow, an integration, g1t itself. | |
| 74 | + | Each report is a status with a `context` (its name) and a state, kept per | |
| 75 | + | commit in `commit_statuses`; richer reports are check runs and suites. | |
| 76 | + | ||
| 77 | + | Actions reports one status per run, named `<workflow> / <event>`, such as | |
| 78 | + | `CI / pull_request` (`services/actions/src/plan.rs`). A required check | |
| 79 | + | named `CI` is met by a status of that name for any event: | |
| 80 | + | `check_name` in `crates/contracts/src/work.rs` reads the name before the | |
| 81 | + | last ` / `. One that failed fails the check, even if another passed. | |
| 82 | + | ||
| 83 | + | ## Deciding a merge | |
| 84 | + | ||
| 85 | + | Rulesets are stored in `work`, and judged by `crates/rules`, which does no | |
| 86 | + | I/O and so can be tested in full. The same rules run in two places: | |
| 87 | + | ||
| 88 | + | | When | Who asks | What is judged | | |
| 89 | + | | --- | --- | --- | | |
| 90 | + | | A push | `repos`, before anything is stored | Which refs may move, and how: deletion, force-push, signed commits, file paths | | |
| 91 | + | | A merge | `work` | Required reviews, required checks, CODEOWNERS, up to date, the merge queue | | |
| 92 | + | ||
| 93 | + | `requirements` gathers every rule that applies to the target branch, and | |
| 94 | + | `judge` returns, for each, whether it passes, fails or is still waiting, | |
| 95 | + | and whether the person asking may bypass it. A merge happens only when | |
| 96 | + | none fails or waits, for people and coding agents alike. | |
| 97 | + | ||
| 98 | + | When everything passes, `work` asks `repos` to land the pull request's | |
| 99 | + | head on the base, and publishes `pull.merged`. | |
| 100 | + | ||
| 101 | + | ## The merge queue | |
| 102 | + | ||
| 103 | + | Two changes can merge without a conflict and still break each other. A | |
| 104 | + | repository whose rules require the merge queue tests each pull request | |
| 105 | + | together with the ones ahead of it, and moves `main` only to a state whose | |
| 106 | + | required checks passed (`services/work/src/queue.rs`). | |
| 107 | + | ||
| 108 | + | <Diagram name="merge-queue" /> | |
| 109 | + | ||
| 110 | + | 1. **Build.** Up to `max_entries_to_build` entries (4 by default) are | |
| 111 | + | tested at once. For each, a sandbox merges the default branch with that | |
| 112 | + | entry and every entry ahead of it, and pushes the result to | |
| 113 | + | `g1t-queue/<entry>` in the repository. | |
| 114 | + | 2. **Test.** Workflows that run on `merge_group` run on each of those | |
| 115 | + | commits, and the required checks have to pass there. | |
| 116 | + | 3. **Land.** Entries land in order. Once everything ahead of it has landed, | |
| 117 | + | an entry moves the default branch to the very commit that was tested. | |
| 118 | + | 4. **Eject.** An entry whose checks fail leaves the queue with a failed | |
| 119 | + | check run on its pull request, and the entries behind it are tested | |
| 120 | + | again without it. | |
| 121 | + | ||
| 122 | + | | State | Means | | |
| 123 | + | | --- | --- | | |
| 124 | + | | `waiting` | In the queue, not yet being built | | |
| 125 | + | | `testing` | Built, its checks running | | |
| 126 | + | | `passed` | Its checks passed; it lands when those ahead have | | |
| 127 | + | | `failed` | Its checks failed; it leaves the queue | | |
| 128 | + | | `landed` | On the default branch | | |
| 129 | + | | `removed` | Taken out by hand | | |
| 130 | + | ||
| 131 | + | ### Who answers for a failure | |
| 132 | + | ||
| 133 | + | A combined state can fail because of the entry itself, or because of an | |
| 134 | + | entry ahead of it. `integration.rs` works out which: | |
| 135 | + | ||
| 136 | + | - **Held, not blamed.** A state that fails while an entry ahead is still | |
| 137 | + | being tested is held. If that entry fails too, the held one is tested | |
| 138 | + | again without it and is not blamed. Only once everything ahead has | |
| 139 | + | passed does the held entry answer for the failure, so a broken change is | |
| 140 | + | blamed on itself, never on what was queued behind it. | |
| 141 | + | - **Narrowed down.** An entry whose own checks pass but whose combination | |
| 142 | + | fails breaks *with* some of the entries ahead. `work` builds the default | |
| 143 | + | branch with half of them and the entry, and halves again, within three | |
| 144 | + | extra builds (`PROBE_BUDGET`), recorded in `queue_probes`. The builds run | |
| 145 | + | on `merge_group` like any queued state and never land. What is left is | |
| 146 | + | reported as the `suspects`. | |
| 147 | + | - **Told everywhere.** The answer goes on the pull request's and the | |
| 148 | + | issue's timelines, into the pull request's session, as a decision record | |
| 149 | + | (`integration_failure`, confidence `tested`, in `decisions`), and as the | |
| 150 | + | `integration.failed` event. A passing combination is recorded as | |
| 151 | + | `integration.passed`. | |
| 152 | + | - **Rejoining.** A pull request the queue took out keeps its merge | |
| 153 | + | request. Once a new commit's checks pass, it is merged again, into the | |
| 154 | + | queue, as whoever queued it, by the rules as they are then. | |
| 155 | + | ||
| 156 | + | [The merge queue](/guides/merge-queue/#who-answers-for-it) has the table of | |
| 157 | + | outcomes from a person's side. | |
| 158 | + | ||
| 159 | + | <Aside title="What a merge queue needs"> | |
| 160 | + | Building candidates takes sandboxes, so the merge queue needs a paid or | |
| 161 | + | trial workspace, or the open-source pool for a public repository. | |
| 162 | + | [The merge queue](/guides/merge-queue/) has the settings. | |
| 163 | + | </Aside> | |
| 164 | + | ||
| 165 | + | ## Events it publishes and reads | |
| 166 | + | ||
| 167 | + | `work` publishes what happens to issues, pull requests, comments, checks | |
| 168 | + | and the queue (`issue.*`, `pull.*`, `comment.*`, `status.created`, | |
| 169 | + | `queue.changed` and the like), and the coordination events: | |
| 170 | + | `session.started`, `session.ended`, `work.claimed`, `work.released`, | |
| 171 | + | `work.claim_expired`, `work.handoff_requested`, `dependency.changed`, | |
| 172 | + | `overlap.detected`, `integration.failed`, `integration.passed` and | |
| 173 | + | `decision.recorded`. It reads | |
| 174 | + | `git.push` and a few repository events from `g1t-events-work` | |
| 175 | + | ([Events](/architecture/events/#subscriber-queues)). |
| 218 | 218 | opened, a pull request being merged, a session growing. Events are delivered | |
| 219 | 219 | to the services that react to them and are kept as a timeline per | |
| 220 | 220 | repository, which you can read through the [API](/reference/api/). | |
| 221 | + | [Events](/architecture/events/) in the Architecture section shows how. | |
| 221 | 222 | ||
| 222 | 223 | ## What is not built yet | |
| 223 | 224 |
| 6 | 6 | [Run g1t yourself](/guides/self-hosting/) starts g1t with Docker Compose. | |
| 7 | 7 | This page is for the person who keeps that installation running: what is | |
| 8 | 8 | inside it, what stands in for each Cloudflare service g1t.sh uses, and | |
| 9 | − | where the limits are. | |
| 9 | + | where the limits are. How the services themselves work, wherever they | |
| 10 | + | run, is in the [Architecture](/architecture/overview/) section. | |
| 10 | 11 | ||
| 11 | 12 | ## The shape | |
| 12 | 13 |
| 117 | 117 | | Storage | Databases and buckets your teams can create, at a level each team is given, with copies of production data for anything not yet reviewed. | <Status is="coming" /> | | |
| 118 | 118 | | [Audit log](/guides/audit-log/) | Every change to the workspace, with export. | <Status is="live" /> | | |
| 119 | 119 | | [Run g1t yourself](/guides/self-hosting/) | The core forge in Docker. Deployments are off in this version. | <Status is="preview" /> | | |
| 120 | + | | [Architecture](/architecture/overview/) | How g1t is built, service by service, for developers building on it, contributing to it or running it. | <Status is="live" /> | | |
| 120 | 121 | ||
| 121 | 122 | ## Bring a repository | |
| 122 | 123 |
Loading the diff…
Loading the diff…
Loading the diff…
Loading the diff…
Loading the diff…
Loading the diff…
Loading the diff…
Loading the diff…
Loading the diff…
Loading the diff…
Loading the diff…
Loading the diff…