| 1 | --- |
| 2 | title: Data and storage |
| 3 | description: Which service owns which database, bucket and Durable Object, the few stores that are shared, how schema changes ship, and the ports that let the same code run on Cloudflare or in Docker. |
| 4 | --- |
| 5 | |
| 6 | import Aside from "../../../components/Aside.astro"; |
| 7 | import Diagram from "../../../components/Diagram.astro"; |
| 8 | |
| 9 | ## Who owns what |
| 10 | |
| 11 | Each D1 database is bound by exactly one Worker. If you need another |
| 12 | service's data, call that service; never read its tables. |
| 13 | |
| 14 | | Service | D1 | Other storage | |
| 15 | | -------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------- | |
| 16 | | `identity` | `g1t` | Writes the avatars KV | |
| 17 | | `repos` | `g1t-repos` | The git store (Artifacts), the git cache KV, the `g1t-git-packs` and `g1t-backups` buckets | |
| 18 | | `work` | `g1t-work` | | |
| 19 | | `events` | `g1t-events` | The `g1t-events` queue, which it produces and consumes, and the subscriber queues it produces | |
| 20 | | `actions` | `g1t-actions` | Lists and expires the `g1t-actions-cache` bucket | |
| 21 | | `billing` | `g1t-billing` | | |
| 22 | | `integrations` | `g1t-integrations` | | |
| 23 | | `webhooks` | `g1t-webhooks` | | |
| 24 | | `security` | `g1t-security` | | |
| 25 | | `search` | `g1t-search` | The `g1t-search-jobs` queue | |
| 26 | | `packages` | `g1t-packages` | The `g1t-packages` bucket | |
| 27 | | `projects` | `g1t-projects` | Writes project images to the avatars KV | |
| 28 | | `chat` | `g1t-chat` | A `ChannelRoom` Durable Object per channel; writes custom emoji to the avatars KV | |
| 29 | | `artifacts` | `g1t-docs` | The `g1t-docs-files` bucket, the `g1t-docs` and `g1t-folios` Vectorize indexes, `PageRoom` and `FolioRoom` Durable Objects | |
| 30 | | `deployments` | `g1t-deployments` | Writes the domains KV; the `g1t-deployments` dispatch namespace | |
| 31 | | `apps/status` | `g1t-status` | | |
| 32 | | `notify` | | `Feed`, `Room` and `Topic` Durable Objects | |
| 33 | | `telemetry` | | A `TelemetryStore` Durable Object per project | |
| 34 | | `runner` | | The sandbox Container classes | |
| 35 | | `og` | | The `g1t-screenshots` bucket | |
| 36 | | `apps/web` | | The `g1t-downloads` bucket | |
| 37 | |
| 38 | ### The shared stores |
| 39 | |
| 40 | Three stores are shared on purpose, each with a single writer per key |
| 41 | prefix: |
| 42 | |
| 43 | | Store | Written by | Read by | |
| 44 | | ---------------------- | --------------------------------------------------------------------------------- | ------------------------------------- | |
| 45 | | `g1t-avatars` KV | `identity` (people and workspaces), `projects` (`production/`), `chat` (`emoji/`) | The site, `og` | |
| 46 | | `g1t-domains` KV | `deployments` | `pages`, `telemetry` | |
| 47 | | `g1t-actions-cache` R2 | The API, as jobs upload | `actions`, which decides what is kept | |
| 48 | |
| 49 | ## Durable Objects |
| 50 | |
| 51 | Durable Objects are used where something has to be live, or where data |
| 52 | belongs to one thing and grows on its own: a chat channel, an open doc, a |
| 53 | person's open tabs, a project's telemetry. Each object keeps its own |
| 54 | SQLite storage. A rule from CONTRIBUTING.md: never wake an object for a |
| 55 | read that D1 can answer. |
| 56 | |
| 57 | ## Secrets at rest |
| 58 | |
| 59 | Secrets that g1t has to use later, such as webhook secrets, Actions |
| 60 | secrets, GitHub tokens of linked accounts and two-factor seeds, are sealed |
| 61 | with AES-256-GCM by `Sealer` in `crates/secrets`, with the row's id as |
| 62 | associated data. A sealed value copied into another row will not open. |
| 63 | Each service seals with its own key (`IDENTITY_KEY`, `ACTIONS_KEY`, |
| 64 | `REPOS_KEY`, …), set as a Worker secret. |
| 65 | |
| 66 | Access tokens and session cookies are not stored at all: only their |
| 67 | SHA-256 is. |
| 68 | |
| 69 | ## Schema changes |
| 70 | |
| 71 | Each database's migrations live beside its service, in |
| 72 | `services/<name>/migrations`, numbered. A deploy applies every pending |
| 73 | migration before any code ships |
| 74 | ([How g1t ships](/architecture/shipping/)), so for a minute or more the old |
| 75 | code runs against the new schema. Three rules follow: |
| 76 | |
| 77 | 1. Add tables and columns. Never rename or drop one that live code reads. |
| 78 | 2. Change what a value means in two deploys: first code that reads both |
| 79 | forms, then the change. |
| 80 | 3. Remove what nothing reads anymore only after the code that read it is |
| 81 | gone. |
| 82 | |
| 83 | ## Ports and adapters |
| 84 | |
| 85 | Each Cloudflare product a service depends on sits behind a port: a trait |
| 86 | in Rust, or a binding whose shape a stand-in can copy. g1t.sh binds |
| 87 | Cloudflare's adapters. A self-hosted installation, built from the same |
| 88 | source, binds open ones, and the service code does not change. |
| 89 | |
| 90 | <Diagram name="ports-and-adapters" /> |
| 91 | |
| 92 | | Port | Defined in | g1t.sh | Self-hosted | |
| 93 | | ------------------------------- | ---------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | |
| 94 | | `GitStore`, `GitRepo` | `services/repos/src/store.rs` | `ArtifactsStore` | A Worker with the same API as the Artifacts binding (`deploy/self-host/workers/gitstore`), in front of a server of bare repositories using `git http-backend` | |
| 95 | | `BlobStore` | `crates/blobstore` | `R2Store` | `S3Store`, with AWS Signature V4, for RustFS, MinIO, Ceph, Garage or AWS S3 | |
| 96 | | `PackStore` | `services/repos/src/pack_cache.rs` | R2 | S3, with `PACK_STORE=s3` | |
| 97 | | `EMAIL` | A `send_email` binding | Cloudflare Email Sending | A mail Worker that hands each message to Mailpit | |
| 98 | | `RUNNER` | A service binding | `services/runner` on Containers | A Worker that answers that the feature is off | |
| 99 | | D1, KV, Queues, Durable Objects | Workers bindings | Cloudflare | workerd, with SQLite files on a volume | |
| 100 | |
| 101 | `deploy/self-host/configs.mjs` writes each Worker's self-hosted config from |
| 102 | its `wrangler.jsonc`, so the two never drift apart: it drops routes and |
| 103 | placement, binds the stand-ins above, points buckets at S3, and replaces |
| 104 | g1t.sh's addresses with yours. What runs and what is off comes from each |
| 105 | unit's `self_host` field in `deploy/stack.jsonc`. |
| 106 | |
| 107 | <Aside title="Adding a dependency on a Cloudflare product"> |
| 108 | Put it behind a port first, and give the self-hosted build an adapter or a stand-in in the same pull request. [How a self-hosted g1t |
| 109 | runs](/guides/self-hosting-architecture/) is the self-hosted side of this table. |
| 110 | </Aside> |