| 1 | --- |
| 2 | title: Git |
| 3 | description: Where g1t keeps repositories, what happens on a push and a clone, the caches in front of the git store, pull request forks, merges, and backups. |
| 4 | --- |
| 5 | |
| 6 | import Aside from '../../../components/Aside.astro'; |
| 7 | import Diagram from '../../../components/Diagram.astro'; |
| 8 | |
| 9 | Every repository on g1t.sh is an ordinary git repository in Cloudflare |
| 10 | Artifacts, a git store. One service, `repos` (`services/repos`, Rust), |
| 11 | holds the only binding to it. Everything else that reads code, the site |
| 12 | and the API included, asks `repos`. |
| 13 | |
| 14 | ## Where repositories live |
| 15 | |
| 16 | `repos` talks to the store through a port: the `GitStore` and `GitRepo` |
| 17 | traits in `services/repos/src/store.rs`, with `ArtifactsStore` as the |
| 18 | adapter for Artifacts. Self-hosted installations bind a different adapter |
| 19 | in its place ([Data and storage](/architecture/data/#ports-and-adapters)). |
| 20 | Every call to the store goes through one function that meters it, retries |
| 21 | reads, and trips a circuit breaker when the store struggles |
| 22 | (`resilience.rs`). |
| 23 | |
| 24 | ### Namespaces and sharding |
| 25 | |
| 26 | An Artifacts namespace takes a fixed number of control requests per 10 |
| 27 | seconds, and its jurisdiction, US or EU, is set when it is made. So g1t |
| 28 | can spread repositories across several namespaces, each its own binding |
| 29 | (`services/repos/src/shards.rs`): |
| 30 | |
| 31 | | Setting | What it decides | |
| 32 | | --- | --- | |
| 33 | | `GITSTORE_NAMESPACES` | The namespaces and their bindings, such as `GITSTORE_1` and `GITSTORE_EU` | |
| 34 | | `GITSTORE_NEW_REPOS` | Where new repositories go | |
| 35 | | `GITSTORE_EU_NAMESPACE` | Where an EU workspace's repositories go | |
| 36 | |
| 37 | A repository's row in D1 records its store key, `namespace/owner--name`. |
| 38 | A key with no namespace is in the first, `g1t`. Moving a repository |
| 39 | between namespaces is an operator task, in `moves.rs`. |
| 40 | |
| 41 | ### What D1 holds |
| 42 | |
| 43 | `g1t-repos` keeps what git does not: each repository's row (store key, |
| 44 | size, default branch, visibility, `refs_version`), renames and their |
| 45 | redirects, releases, stars, stats, backups, moves, the commits each push |
| 46 | brought for the contribution calendar, and the meter of every git |
| 47 | operation. |
| 48 | |
| 49 | ## A push |
| 50 | |
| 51 | <Diagram name="push-path" /> |
| 52 | |
| 53 | The site recognises a git path and hands the whole request to `repos` |
| 54 | (steps 1 and 2), after the git rate limits. From there, `repos` does the |
| 55 | work in `answer_git` (`services/repos/src/lib.rs`): |
| 56 | |
| 57 | 1. **Who is pushing.** HTTP Basic credentials go to `identity` |
| 58 | (steps 3 and 4). A secret with the token prefix is an access token, and |
| 59 | the username is ignored. Anything else is a password, which `identity` |
| 60 | refuses for anyone with two-factor authentication on, so they push |
| 61 | with a token. The repository's row is read at the same time. |
| 62 | 2. **Which repository.** Renames redirect, transfers resolve, and a pull |
| 63 | request's fork that was retired is made again. |
| 64 | 3. **Whether they may.** The access check, then the workspace's limits. |
| 65 | 4. **The rules.** `repos` asks `work` for the rulesets that apply |
| 66 | (step 5). |
| 67 | 5. **The checks** (step 6). The pack is read once, and four checks run on |
| 68 | it side by side; their verdicts are taken in this order: |
| 69 | |
| 70 | | Check | Where | Refuses | |
| 71 | | --- | --- | --- | |
| 72 | | Workflow gate | `workflow_gate.rs` | A token without the workflow permission changing `.g1t/workflows` | |
| 73 | | Rules | `rules.rs`, judged by `crates/rules` | What a ruleset forbids, with a `remote:` line naming the rule | |
| 74 | | Sizes | `pack_limits.rs` | A file over 32 MB, a repository over 950 MB, a push over 100 MB | |
| 75 | | Push protection | `secret_scan.rs`, using `crates/scan` | Secrets, private email addresses, and the workspace's own patterns | |
| 76 | |
| 77 | 6. **The store** (steps 7 and 8). The pack goes to Artifacts with a |
| 78 | short-lived store credential. |
| 79 | 7. **The answer** (steps 9 and 10). Once the store reports, `refs_version` |
| 80 | goes up, which invalidates every cached ref listing, and only then |
| 81 | does git get its answer. |
| 82 | |
| 83 | After the answer, in `waitUntil`, `repos` checks with the store's log that |
| 84 | each ref really moved, adds the repository's new size, counts commits, and |
| 85 | publishes `git.push` for each branch (step 11 and 12). Everything that |
| 86 | reacts to a push, from workflows to search to deployments, starts from |
| 87 | that event ([Events](/architecture/events/)). |
| 88 | |
| 89 | <Aside title="Large pushes"> |
| 90 | A push up to 24 MiB is scanned before it is stored. A larger one is |
| 91 | streamed straight to the store, its `git.push` events say |
| 92 | `unscanned: true`, and `security` scans it after it lands. |
| 93 | </Aside> |
| 94 | |
| 95 | A few details that keep pushes fast and correct: |
| 96 | |
| 97 | - **`no-thin`.** g1t advertises that it wants whole packs, so the checks |
| 98 | never have to read objects from the store. A client that sends a thin |
| 99 | pack anyway still works: up to 200 bases are fetched. |
| 100 | - **Hidden refs.** `refs/pull/*` belong to g1t. A push to one is refused |
| 101 | for that ref alone. |
| 102 | - **Announcements are capped.** A push with more than three tags announces |
| 103 | none of them, and one with more than 1,000 branches announces only the |
| 104 | default branch. |
| 105 | |
| 106 | ### Timing |
| 107 | |
| 108 | Every git answer carries `Server-Timing`, with a step for each stage: |
| 109 | `repo`, `access`, `mint` (a store credential), `store`, `checks` (with |
| 110 | `read`, `rules`, `scan` and `gate`), `refs`, `total` and more. Run |
| 111 | `GIT_CURL_VERBOSE=1 git push` to see where a slow push spent its time. |
| 112 | |
| 113 | ## A clone or fetch |
| 114 | |
| 115 | <Diagram name="clone-path" /> |
| 116 | |
| 117 | A fetch is answered from the first place that can: |
| 118 | |
| 119 | | Order | From | Holds | Kept | |
| 120 | | --- | --- | --- | --- | |
| 121 | | 1 | R2, `g1t-git-packs` | Whole packs for fresh clones: a request with wants and no haves, shallow ones included | 7 days, or until a ref moves | |
| 122 | | 2 | The colo's cache | Ref listings, and trees and blobs up to 1 MiB, which never change | Refs 60 s; objects forever | |
| 123 | | 3 | KV, `g1t-repos-git-cache` | Ref listings and store credentials, shared by every isolate | 60 s | |
| 124 | | 4 | Artifacts | Everything | | |
| 125 | |
| 126 | Every cached ref listing is keyed by the repository's `refs_version`, so a |
| 127 | push makes the old one unreachable at once. A change can take up to 5 |
| 128 | seconds to reach every fetch, while a repository's row is reused. |
| 129 | |
| 130 | **Packs are cached as they stream.** On a miss, the store's answer goes |
| 131 | to git and to R2 at the same time, never held whole: in one `put` when |
| 132 | small, in 5 MiB parts when large, completed only after a check. A pack |
| 133 | over 200 MiB is not cached, at most two fills run at once, and a |
| 134 | repository fills at most 30 times a minute. A hit never touches the store. |
| 135 | |
| 136 | **What is in KV is sealed.** Each value is encrypted with AES-256-GCM under |
| 137 | the `REPOS_KEY` secret, with its own key as associated data, so a value |
| 138 | copied to another key will not open (`services/repos/src/shared.rs`). |
| 139 | |
| 140 | When the store is busy, git gets 429 or 503 with `Retry-After`. Pushes are |
| 141 | never retried on your behalf. |
| 142 | |
| 143 | ## Pull request forks |
| 144 | |
| 145 | A pull request made for an issue gets a fork of its own |
| 146 | ([Forks and working copies](/concepts/forks/)). `fork_for_pull` makes it in the |
| 147 | same namespace as its source, as `pulls--<pull request id>`, with |
| 148 | Artifacts' own fork of the default branch, and it is served at |
| 149 | `g1t.sh/pulls/<id>.git`. Anyone who can read the source can read the fork. |
| 150 | |
| 151 | A day after the pull request merges or closes, an hourly job retires the |
| 152 | fork: its head is kept in the source as `refs/pull/<id>/head`, and its data |
| 153 | is removed. Reads of a retired fork are answered from the source, and any |
| 154 | write makes the fork again first. |
| 155 | |
| 156 | ## Merging |
| 157 | |
| 158 | The Artifacts binding reads but does not write, so `repos` lands a merge |
| 159 | over git itself (`services/repos/src/land.rs`). It asks the fork for a pack |
| 160 | with exactly the objects the target lacks, and streams it into a push to |
| 161 | the target, without parsing it or holding it whole. |
| 162 | |
| 163 | `repos` also builds the test merge for every pull request, |
| 164 | `refs/pull/<n>/merge`, without a sandbox, using a line-level merge on |
| 165 | files up to 1 MiB (`test_merge.rs`). When a pull request is behind and the |
| 166 | two sides' changes do not clash, updating its branch is done the same way |
| 167 | (`catch_up.rs`), by a merge or a rebase; so are merge, squash and rebase |
| 168 | merges, up to 1,000 changed files, 100 files merged line by line and 250 |
| 169 | commits in a rebase. Testing several pull requests together is the merge |
| 170 | queue's job ([Work and the merge queue](/architecture/work/#the-merge-queue)). |
| 171 | |
| 172 | ## Backups |
| 173 | |
| 174 | At 02:53 UTC, `repos` queues up to 200 repositories whose refs moved since |
| 175 | their last backup, oldest first. The runner cuts them: each job gets a |
| 176 | sandbox that clones with a read-only store credential valid for a few |
| 177 | minutes, writes a `git bundle`, and uploads it in 32 MiB parts to the |
| 178 | `g1t-backups` bucket. Every thirtieth backup is full and the rest are |
| 179 | incremental. |
| 180 | |
| 181 | | Path | Holds | |
| 182 | | --- | --- | |
| 183 | | `backups/<repo id>/manifest.json` | The repository's bundles, in order | |
| 184 | | `backups/<repo id>/<time>-full.bundle` | Every ref and object | |
| 185 | | `backups/<repo id>/<time>-incr.bundle` | What changed since the bundle before | |
| 186 | |
| 187 | `scripts/ops/backup-restore-drill.mjs` restores from them, and |
| 188 | `scripts/ops/restore-to-gitstore.mjs` can rebuild repositories into a |
| 189 | self-hosted git store, which g1t.sh can then read from if Artifacts is |
| 190 | unavailable (`fallback.rs`). |
| 191 | |
| 192 | ## SSH |
| 193 | |
| 194 | `crates/sshd` is an SSH front end for git that authenticates your key with |
| 195 | `identity` and bridges to the store. It is not deployed: git on g1t is |
| 196 | HTTPS only for now. You can already add deploy keys. |