| 1 | # Cloudflare Artifacts: due diligence for g1t at launch scale |
| 2 | |
| 3 | Status: research document, 2026-10-06; R1–R5, R9, R10, R13 and R7 groundwork were built the same day (section 9). |
| 4 | R7 (sharding, moves, EU residency) and R12 (the fallback store) were built 2026-10-07, off until their |
| 5 | infrastructure exists ("What you must create", section 9). |
| 6 | Scope: everything g1t stores in Cloudflare Artifacts (open beta since 2026-10-01; billing from 2026-10-14), |
| 7 | measured against what Cloudflare documents, and what we must build so that a few thousand workspaces |
| 8 | can run on it. |
| 9 | |
| 10 | ## Summary |
| 11 | |
| 12 | Artifacts fits what g1t does: one Durable Object per repository, smart HTTP for clones and pushes, a |
| 13 | read-only binding for commits, trees and blobs, forks and short-lived tokens. Nothing we rely on is |
| 14 | missing outright. Five things are not yet safe at a few thousand workspaces. |
| 15 | |
| 16 | 1. **We do not know what an "operation" is.** Pricing says "repo operations, such as `create`, `push`, |
| 17 | `pull`, and `clone`". Metrics list only `create`, `fork`, `push`, `pull`, `delete`. If binding reads |
| 18 | (`get`, `info`, `createToken`, `log`, `readTree`, `readBlob`, `readCommit`, `readFile`) and every git |
| 19 | HTTP request also count, our bill at launch scale is roughly **15× larger** (about $31k a month instead |
| 20 | of about $1.8k). We must settle this before 2026-10-14. |
| 21 | 2. **Pull request forks are never deleted.** Every pull request is an Artifacts fork (`pulls--<id>`). They |
| 22 | are purged only with their parent repository. Cloudflare does not say whether a fork shares objects |
| 23 | with its source. If forks copy objects, an agent-heavy workload reaches the **1 TB account storage |
| 24 | limit in days**, and every push then fails. If they share, storage still grows without bound. |
| 25 | 3. **One namespace carries everything.** All repositories and forks live in the `g1t` namespace. The |
| 26 | control-plane limit is **2,000 requests per 10 seconds per namespace** (200 per second). If binding |
| 27 | calls count against it, page views, token mints and mergeability checks together exceed it at launch |
| 28 | peaks. Sharding is now built (R7, section 9); the extra namespaces are not made yet. |
| 29 | 4. **Hard limits are not enforced in front of Artifacts.** 1 GB per repository, 32 MB per file, and a |
| 30 | 128 MB Worker isolate that buffers each push body twice. Large pushes and imports fail late, without a |
| 31 | message git can show. |
| 32 | 5. **No backup, no exit drill.** Cloudflare replicates data, but there is no SLA, no documented export |
| 33 | besides git itself, and the self-host git store is not a production fallback yet. Nightly bundles |
| 34 | to R2 and a restore drill are now built (R11, section 9), and so is the fallback path (R12): a |
| 35 | restore into the git store and a switch by configuration. The host it runs on is not made yet. |
| 36 | |
| 37 | None of these blocks an invite-only launch. Items 1 and 2 must be answered before billing starts on |
| 38 | 2026-10-14, and the fork cleanup must ship before agent pull requests reach thousands a day. |
| 39 | |
| 40 | ## 1. What Cloudflare documents |
| 41 | |
| 42 | All quotes are from developers.cloudflare.com, retrieved 2026-10-06. |
| 43 | |
| 44 | ### Limits ([Artifacts limits](https://developers.cloudflare.com/artifacts/platform/limits/), updated 2026-10-01) |
| 45 | |
| 46 | | Limit | Value | |
| 47 | | --- | --- | |
| 48 | | Control-plane request rate | 2,000 requests per 10 seconds **per namespace** | |
| 49 | | Git request rate | 2,000 requests per 10 seconds **per repository** | |
| 50 | | Storage per repository | **1 GB** | |
| 51 | | Largest file or blob | **32 MB** | |
| 52 | | Storage per account | **1 TB** (can be raised on request) | |
| 53 | | Repositories, namespaces | Unlimited | |
| 54 | | Names | 2 to 63 characters; letters, digits, `.`, `_`, `-`; start with a letter or digit | |
| 55 | |
| 56 | ### Pricing ([Artifacts pricing](https://developers.cloudflare.com/artifacts/platform/pricing/)) |
| 57 | |
| 58 | - Workers Paid only. "Cloudflare will begin billing for Artifacts operations and storage on October 14, 2026." |
| 59 | - Operations: first 10,000 a month, then **$0.15 per 1,000**. Defined only as "the number of repo |
| 60 | operations, such as `create`, `push`, `pull`, and `clone`." |
| 61 | - Storage: first 1 GB, then **$0.50 per GB-month**, "calculated by averaging peak storage per day over a |
| 62 | 30-day billing period". "Replicas do not add storage charges." "Repos remain stored until you |
| 63 | explicitly delete them." |
| 64 | |
| 65 | ### Architecture ([announcement blog](https://blog.cloudflare.com/artifacts-git-for-agents-beta/), 2026-04-16, Matt Carey and Matt Silverlock) |
| 66 | |
| 67 | - Each repository is a Durable Object; the git server is Zig compiled to WebAssembly (about 100 KB). |
| 68 | - "Files are stored in the underlying Durable Object's SQLite database." "Durable Object storage has a |
| 69 | 2MB max row size, so large Git objects are chunked and stored across multiple rows." |
| 70 | - "DOs have ~128MB memory limits." Fetch and push stream (`ReadableStream<Uint8Array>`). |
| 71 | - "Artifacts also uses R2 (for snapshots) and KV (for tracking auth tokens)." |
| 72 | - Deltas are stored as received; "if the requesting client already has the base object, Zig emits the |
| 73 | delta instead." |
| 74 | - Durability ([How Artifacts works](https://developers.cloudflare.com/artifacts/concepts/how-artifacts-works/)): |
| 75 | "Cloudflare replicates repo data synchronously across multiple data centers and copies it |
| 76 | asynchronously to object storage and snapshots." |
| 77 | - A repository is "a single logical instance that Cloudflare can route to from any region", like a |
| 78 | Durable Object. Durable Objects have a soft limit of 1,000 requests per second each |
| 79 | ([DO limits](https://developers.cloudflare.com/durable-objects/platform/limits/)). |
| 80 | |
| 81 | ### Git protocol ([Git protocol](https://developers.cloudflare.com/artifacts/api/git-protocol/)) |
| 82 | |
| 83 | | Feature | Documented support | |
| 84 | | --- | --- | |
| 85 | | Clone and fetch | Protocol v1 and v2 (`ls-refs`, `fetch`); v1 shallow and deepen | |
| 86 | | Push | v1 receive-pack only; v2 receive-pack not supported | |
| 87 | | `filter` (partial clone), `include-tag` | "not supported" for v1 | |
| 88 | | Tokens | `art_v1_<40 hex>?expires=<unix seconds>`; scopes `read`, `write`; TTL 60 s to 1 year, default 24 h | |
| 89 | |
| 90 | **Measured differently:** a protocol v2 `git clone --filter=blob:none` of `flagon-io/g1t` through g1t |
| 91 | returned a real partial clone (2,789 objects, 903 KB) instead of the full 6,187 objects and 5.5 MB. So |
| 92 | v2 filtering works today, despite the table. Ask Cloudflare whether this is supported or accidental. |
| 93 | |
| 94 | ### Binding API ([Workers binding](https://developers.cloudflare.com/artifacts/api/workers-binding/); generated types in `services/*/worker-configuration.d.ts`) |
| 95 | |
| 96 | - Namespace: `create`, `get`, `list`, `import`, `delete`. `get()` is a lookup that throws `NOT_FOUND` or |
| 97 | `*_IN_PROGRESS`, so it costs a round trip. |
| 98 | - Repository handle: `info()` ("Each call performs a fresh lookup"), `createToken`, `listTokens`, |
| 99 | `revokeToken`, `log` (first-parent only, at most 1,000), `readCommit`, `readTree` (one level), |
| 100 | `readBlob`, `readFile`, `fork` (`defaultBranchOnly` defaults to true). |
| 101 | - Not available: listing refs, updating refs, writing objects, repository size, gc or repack, hooks. |
| 102 | g1t works around the first three over smart HTTP (`refs.rs`, `land.rs`, `catch_up.rs`). |
| 103 | - Errors: `MEMORY_LIMIT` (10402) "if the object cannot be buffered safely", `INTERNAL_ERROR` (10400), |
| 104 | `UPSTREAM_UNAVAILABLE`, `IMPORT_IN_PROGRESS`, `FORK_IN_PROGRESS`, `CREATE_IN_PROGRESS` |
| 105 | ([Errors](https://developers.cloudflare.com/artifacts/api/errors/)). |
| 106 | - Doc drift: the [isomorphic-git example](https://developers.cloudflare.com/artifacts/examples/isomorphic-git/) |
| 107 | still says the binding "cannot read or write files inside them". It can read; it cannot write. |
| 108 | |
| 109 | ### Observability, events, data location |
| 110 | |
| 111 | - Metrics: GraphQL dataset `artifactsEventsAdaptiveGroups`, 31 days, with `eventType` in `create`, |
| 112 | `fork`, `push`, `pull`, `delete`, and errors `storageLimitReached`, `serverError`, `clientError`, |
| 113 | `rateLimited` ([Metrics](https://developers.cloudflare.com/artifacts/observability/metrics/)). |
| 114 | **This is the fastest way to learn what Cloudflare counts.** |
| 115 | - Events: account level `repo.created|deleted|forked|imported`; per-repository `pushed`, `cloned`, |
| 116 | `fetched`, `token.created|revoked`, which need one subscription per repository |
| 117 | ([Event subscriptions](https://developers.cloudflare.com/artifacts/guides/event-subscriptions/)). |
| 118 | g1t reports pushes itself for that reason (`record_push`). |
| 119 | - Jurisdictions `eu` and `us`, chosen when a namespace is created and never changeable |
| 120 | ([Data localization](https://developers.cloudflare.com/artifacts/guides/data-localization/)). A binding |
| 121 | names one namespace, so EU residency needs a second binding. |
| 122 | - Best practices: "Do not keep every repo in one default namespace once usage grows" and "When one |
| 123 | namespace becomes hot, shard new repos into additional namespaces" |
| 124 | ([Best practices](https://developers.cloudflare.com/artifacts/concepts/best-practices/)). |
| 125 | - Beta: the docs state no SLA, no support tier, and no backup or export feature beyond git itself. |
| 126 | |
| 127 | ### Platform limits that bound us |
| 128 | |
| 129 | | Limit | Value | Source | |
| 130 | | --- | --- | --- | |
| 131 | | Worker memory | 128 MB per isolate, shared by concurrent requests | [Workers limits](https://developers.cloudflare.com/workers/platform/limits/) | |
| 132 | | Request body | 100 MB on Free and Pro zones, 200 MB Business, up to 5 GB Enterprise | same | |
| 133 | | Response body | No limit (CDN cache 512 MB) | same | |
| 134 | | CPU | 30 s default, up to 5 min | same | |
| 135 | | Subrequests | 10,000 per invocation (Paid); 6 connections waiting for headers at once | same | |
| 136 | | Cache API | Per data center only; calls share the subrequest quota | [Cache](https://developers.cloudflare.com/workers/runtime-apis/cache/) | |
| 137 | | KV | 1 write per second per key; 25 MiB values; 60 s minimum TTL | [KV limits](https://developers.cloudflare.com/kv/platform/limits/) | |
| 138 | | D1 | 10 GB per database; 30 s per query; one writer | [D1 limits](https://developers.cloudflare.com/d1/platform/limits/) | |
| 139 | | Cloudflare REST API | 1,200 requests per 5 minutes per user | [API rate limits](https://developers.cloudflare.com/fundamentals/api/reference/limits/) | |
| 140 | | Containers | Up to 4 vCPU, 12 GiB, 20 GB disk (ephemeral) | [Containers limits](https://developers.cloudflare.com/containers/platform-details/limits/) | |
| 141 | | Smart Placement | Applies to fetch handlers, not RPC entrypoints | [Placement](https://developers.cloudflare.com/workers/configuration/placement/) | |
| 142 | |
| 143 | The REST API's 1,200 requests per 5 minutes means the hot path must use the binding, never the REST API. |
| 144 | |
| 145 | ## 2. How g1t uses Artifacts |
| 146 | |
| 147 | All Artifacts traffic goes through `services/repos` (`g1t-repos`), binding `ARTIFACTS`, namespace `g1t`, |
| 148 | placement off. Store keys are `<workspace>--<repo>`; pull request forks are `pulls--<pull id>`. |
| 149 | |
| 150 | ### Calls per user action |
| 151 | |
| 152 | "Binding" counts calls on `env.ARTIFACTS`; "git" counts HTTP requests to the repository's remote. |
| 153 | |
| 154 | | Action | Path in code | Binding calls | Git requests to Artifacts | |
| 155 | | --- | --- | --- | --- | |
| 156 | | Credential (mint) | `store.rs` `mint_access` | 3: `get`, then `info` and `createToken` at once | 0 | |
| 157 | | Clone, protocol v2 | `lib.rs` `answer_git` → `git_http::forward` | mint if none kept (3) | `info/refs` and `ls-refs` (0 on a `refs_cache` hit), `fetch` 1 | |
| 158 | | Clone, protocol v0 | same | mint if none kept | `info/refs` (0 on hit), `upload-pack` 1 | |
| 159 | | Fetch, nothing new | same | as above | `ls-refs` (0 on hit), `fetch` 1 | |
| 160 | | Push | same, then `record_push` | mint; then `get` + `log(branch, 1)` per pushed branch | `info/refs` 1, `receive-pack` 1 | |
| 161 | | Tree page | `Repos::tree` | `get`, `log(ref, 1)`, `readTree` per path segment (Cache API), `readBlob` README (cached) | 0 | |
| 162 | | File page | `Repos::blob` | `get`, `readFile` (not cached) | 0 | |
| 163 | | Branches list | `GitRepo::branches` (`refs.rs`) | mint (3) | `info/refs` 1 (does not use `refs_cache`) | |
| 164 | | Blame | `blame.rs` | `log(ref, 400)`, then trees and blobs per commit | 0 (one view took 6.6 s wall in the tail) | |
| 165 | | Open a pull request | `fork_for_pull` | `get`, `fork` | 0 | |
| 166 | | Mergeability (each time the target branch moves, for up to 100 open pull requests) | `divergence` | `get` ×2, `log(…, 1000)` ×2, `readCommit` per commit off the first-parent chain, `readTree` per changed directory | 0 | |
| 167 | | Catch-up (no conflicts) | `catch_up.rs` | `get` ×2, `log` ×2, trees | `upload-pack` 1, `receive-pack` 1 | |
| 168 | | Land a pull request | `land.rs` | 2 mints (6) | `upload-pack` 1 from the fork, `receive-pack` 1 to the target | |
| 169 | | Commit one file | `commit_file.rs` | mint, `log` | `info/refs` 1, `receive-pack` 1 | |
| 170 | | Mirror sync or import | `mirror.rs`, `import.rs` | mint | `info/refs` 1–2, `upload-pack` and `receive-pack` 1 each; packs capped at 40 MB | |
| 171 | | Search indexing | `listing.rs` | trees by level, blobs in groups | 0 | |
| 172 | | Push protection | `secret_scan.rs` | trees and blobs for bases, up to 24 MB scanned | 0 | |
| 173 | | Delete (purge) | `lifecycle.rs` | `delete` per key, including forks | 0 | |
| 174 | |
| 175 | Every sandbox job clones in full (`crates/runner/src/{main,checks,review,plan,queue,reply,update,deploy,mergecheck}.rs`: |
| 176 | `git clone --quiet`, no depth, no filter), and most then fetch an upstream branch. Only Actions' |
| 177 | checkout fetches with `--depth=1` (`crates/runner/src/actions/uses.rs`). |
| 178 | |
| 179 | ### What already saves calls |
| 180 | |
| 181 | | Mitigation | Where | What it saves | |
| 182 | | --- | --- | --- | |
| 183 | | Credential cache | `store.rs` `Credentials` (per isolate) and `shared.rs` (KV, sealed), TTL 300 s, reused 180 s | 3 binding calls and about 800 ms per git request on a hit | |
| 184 | | Ref listing cache | `refs_cache.rs`, keyed by `refs_version`, 60 s TTL, colo Cache API then KV | the `info/refs` and `ls-refs` round trip (330–400 ms) on a hit; measured hits answer in 3–6 ms | |
| 185 | | Object cache | `store.rs` `cached`/`keep`, Cache API, immutable | `readTree`, `readBlob` (≤ 1 MB), and `log` by commit hash | |
| 186 | | Recent row cache | `registry.by_path_recent` | D1, not Artifacts | |
| 187 | | Soft delete | `lifecycle.rs` | an accidental `delete` cannot lose data for `RESTORE_DAYS` | |
| 188 | |
| 189 | Gaps in those caches: the Cache API is per data center, so each colo pays its own misses; `log` by branch |
| 190 | name, `readFile`, `branches()` and `readCommit` are never cached; the credential reuse window is 180 s, |
| 191 | so a busy repository mints about 20 times an hour per scope. |
| 192 | |
| 193 | ## 3. Measured in production |
| 194 | |
| 195 | Read-only, from a client in Denver (`CF-Ray …-DEN`), 2026-10-06 07:40 UTC, git 2.45 user agent. |
| 196 | `flagon-io/hello`: 239 objects, 44 KB pack. `flagon-io/g1t`: 6,187 objects, 5.5 MB pack. |
| 197 | |
| 198 | | Measurement | hello | g1t | |
| 199 | | --- | --- | --- | |
| 200 | | `info/refs`, cold: credential minted | n/a | 1,306–1,425 ms total: `mint` 807–856 ms, `store` 334–399 ms | |
| 201 | | `info/refs`, `refs_cache` hit | 3–6 ms server, 80–150 ms client | 3–6 ms server | |
| 202 | | `ls-refs` / v2 advertisement miss with kept credential | 436 ms (`store` 349) | 458 ms (`store` 332) | |
| 203 | | Full upload-pack (replayed v0 want, kept credential) | 674–1,104 ms, `store` 607–997 | 1,960–2,223 ms to first byte, `store` 1,855–2,156; 3.2 s total | |
| 204 | | `git clone --bare`, first / repeats | 5.1 s / 2.0–2.4 s | 5.4 s / 4.2 s (4.8–5.8 s during the tail) | |
| 205 | | `git fetch`, nothing new | 0.99 s first, then 0.39–0.42 s | 0.95 s first, then 0.35–0.41 s | |
| 206 | | `clone --depth=1` | 1.6 s | 3.8 s | |
| 207 | | `clone --filter=blob:none` | 1.4 s | 3.4 s, 903 KB | |
| 208 | | `kept` step on every POST | 63–98 ms | same | |
| 209 | |
| 210 | Observations: |
| 211 | |
| 212 | - Artifacts builds the whole pack before the first byte: about 2 s for 5.5 MB. Every full clone pays |
| 213 | this, every time, because nothing caches packs. |
| 214 | - Minting a credential costs about 0.8 s, almost all in `get` then `info` and `createToken`. |
| 215 | - The 63–98 ms `kept` step on every POST is not Artifacts. It is `git_limits` doing a D1 upsert into |
| 216 | `git_operations` before the request is forwarded. |
| 217 | - `wrangler tail g1t-repos` for 75 s: 87 events, all `ok`, no exceptions or error logs. The slowest were a |
| 218 | blame RPC (6.6 s wall, 178 ms CPU) and an `info/refs` miss with a mint (2.6 s). |
| 219 | |
| 220 | ## 4. Where we and the docs disagree |
| 221 | |
| 222 | | # | Topic | Documented | What g1t does or assumes | Risk | Fix | |
| 223 | | --- | --- | --- | --- | --- | --- | |
| 224 | | M1 | What an operation is | "create, push, pull, clone"; metrics events `create`, `fork`, `push`, `pull`, `delete` | `git_ops.rs` bills workspaces per upload-pack or receive-pack POST, including `ls-refs` answered from our cache; binding reads assumed free | Cost could be about 15× the plan; customers billed for operations that never reach Artifacts | R1 | |
| 225 | | M2 | Fork storage | Not documented; fork returns `objects: N` | `store.rs` calls forks "copy-on-write"; forks never deleted until the parent is purged | 1 TB account limit; storage grows forever | R2 | |
| 226 | | M3 | Namespace rate | 2,000 per 10 s per namespace | One namespace for all repositories and forks | 429s across all of g1t at peak | R7 | |
| 227 | | M4 | Repository size | 1 GB | Free private storage is 1 GB per workspace; no per-repository check; imports capped at 40 MB | Late `storageLimitReached` failures with no explanation | R4 | |
| 228 | | M5 | File size | 32 MB | No check before forwarding | Push fails inside Artifacts | R4 | |
| 229 | | M6 | Push bodies | Workers 128 MB per isolate; 100 MB body cap on our zone plan | `git_http::forward` reads the body (`request.bytes()`) and copies it into a `Uint8Array`; push protection also reads it | Pushes above roughly 40–50 MB can exceed isolate memory, taking concurrent requests with them | R4 | |
| 230 | | M7 | Landing and mirrors | Fetch streams | `land::fetch_pack` buffers the pack, `unpack_sideband` copies it, `push_pack` copies again | Large pull requests can fail to land | R4 | |
| 231 | | M8 | Token lifetime | 60 s to 1 year | 300 s, reused 180 s | Mint load and 0.8 s latency on each miss | R3 | |
| 232 | | M9 | `info()` | "Each call performs a fresh lookup" | Called on every mint only to read `remote` | One wasted call per mint; `remote` is fixed per key | R3 | |
| 233 | | M10 | Partial clone | v1 `filter` unsupported | v2 `blob:none` works in production | Building on undocumented behaviour | Q3 | |
| 234 | | M11 | Read-after-write | Not documented | `record_push` reads `log(branch, 1)` right after the push response; `refs_moved` assumes refs are final when the response ends | Missed push events if a read lags | Q5 | |
| 235 | | M12 | Errors | Typed `ArtifactsError`, `rateLimited` events | `ALREADY_EXISTS`/`NOT_FOUND` tolerated; everything else returns 500; no retry, no breaker | Brief Artifacts errors become user-visible failures | R5 | |
| 236 | | M13 | Durability | Synchronous replication, asynchronous snapshots; no SLA; no export | No copy outside Artifacts | Beta incident or account issue with no recovery path | R11 | |
| 237 | | M14 | Data location | Jurisdiction per namespace, fixed | `PLAN.md` promises residency per workspace; only `g1t` exists | EU customers cannot be offered residency | R7 | |
| 238 | | M15 | Direct credentials | Tokens are bearer, repo-scoped | `git_access` hands out raw write tokens; pushes with them skip branch protection and push protection (`refs_open` only stops caching) | Policy bypass if a token leaks out of a sandbox | Keep TTL minimal on this path | |
| 239 | | M16 | Hot repository | 2,000 git requests per 10 s per repository; DO soft limit 1,000 req/s | Agents clone the same repository many times per pull request | Not near the limit today; a monorepo with many agents could be | R6 | |
| 240 | |
| 241 | ## 5. Capacity model: 3,000 workspaces |
| 242 | |
| 243 | ### Assumptions |
| 244 | |
| 245 | - 3,000 workspaces, 10,000 repositories, 6,000 active people, average repository 25 MB stored. |
| 246 | - People: 6 fetches and 3 pushes per person per day. |
| 247 | - Agent pull requests: 15,000 a day (5 per workspace). Human pull requests: 3,000 a day. |
| 248 | - Per agent pull request: 1 fork; 1.5 agent runs × (clone + upstream fetch + push); 1.5 check clones; |
| 249 | review clone + fetch; mergecheck in 30% of cases (clone + fetch); 0.5 catch-up push; merge queue clone + |
| 250 | fetch; landing upload-pack + receive-pack; 0.3 deploy clones. About **14 git operations**. |
| 251 | - Actions: 1.5 checkouts per push. Mirrors: 1,000 repositories × 24 syncs × 2. |
| 252 | - Web: 60 page views per person per day, about 5 uncached binding calls each. |
| 253 | - Mergeability: 18,000 default-branch moves a day × 8 open pull requests × about 20 binding calls. |
| 254 | |
| 255 | ### Operations per day |
| 256 | |
| 257 | | Source | Scenario A: git data operations only | Scenario B: every Artifacts call | |
| 258 | | --- | ---: | ---: | |
| 259 | | People: fetch and push | 54,000 | 54,000 | |
| 260 | | Agent pull requests (15,000 × 14) | 210,000 | 210,000 | |
| 261 | | Human pull requests (3,000 × 8) | 24,000 | 24,000 | |
| 262 | | Actions checkouts | 60,000 | 60,000 | |
| 263 | | Mirrors | 48,000 | 48,000 | |
| 264 | | Credential mints (≈ 200,000 × 3) | 0 | 600,000 | |
| 265 | | Ref listing misses (`info/refs`, `ls-refs`) | 0 | 300,000 | |
| 266 | | Web pages (360,000 × 5) | 0 | 1,800,000 | |
| 267 | | Mergeability (18,000 × 8 × 20) | 0 | 2,900,000 | |
| 268 | | Search indexing, push protection, `record_push` | 0 | 900,000 | |
| 269 | | **Total per day** | **≈ 400,000** | **≈ 6,900,000** | |
| 270 | | **Per month** | **≈ 12 M** | **≈ 207 M** | |
| 271 | | **Cost per month at $0.15 / 1,000** | **≈ $1,800** | **≈ $31,000** | |
| 272 | |
| 273 | Rates: scenario A averages 5 operations per second. Scenario B averages 80 calls per second and peaks |
| 274 | around 250–400, above the 200 per second a single namespace allows, if binding calls count toward it. |
| 275 | |
| 276 | Hot spots, largest first in scenario B: mergeability fan-out (42%), web reads (26%), credential mints |
| 277 | (9%), indexing and scanning (13%), sandbox clones (6%, but they carry almost all the bytes and the |
| 278 | latency). |
| 279 | |
| 280 | ### Storage |
| 281 | |
| 282 | | Item | Estimate | Cost per month | |
| 283 | | --- | --- | ---: | |
| 284 | | 10,000 repositories × 25 MB | 250 GB | ≈ $125 | |
| 285 | | Pull request forks if shared (about 0.5 MB of new objects each) | +270 GB every month, never deleted | +$135, growing each month | |
| 286 | | Pull request forks if copied (25 MB each, 540,000 a month) | +13.5 TB a month | ≈ $6,750 the first month, and the 1 TB account limit is reached in about 2 days | |
| 287 | |
| 288 | ### Free tier exposure |
| 289 | |
| 290 | `GIT_OPERATIONS_FREE_CAP` is 50,000 operations per workspace per month, never charged. At $0.15 per 1,000 |
| 291 | that is at most $7.50 per free workspace, or $22,500 a month if all 3,000 used it. Realistic use is far |
| 292 | lower, but the cap should be sized to what Cloudflare actually counts (R1). |
| 293 | |
| 294 | ## 6. Recommendations, ranked by risk × effort |
| 295 | |
| 296 | P0 means before 2026-10-14 or before agent volume ramps. Effort assumes one engineer. |
| 297 | |
| 298 | | # | Priority | What to build | Where | Expected effect | Effort | |
| 299 | | --- | --- | --- | --- | --- | --- | |
| 300 | | R1 | P0 | Find out what counts. Query `artifactsEventsAdaptiveGroups` grouped by `eventType` for the last 31 days and compare with `git_operations`; ask Cloudflare in writing. Then make `git_ops::count` count what Cloudflare counts (skip `ls-refs` and cache hits if they are free). | `services/repos/src/git_ops.rs`, a script under `scripts/` | Removes the 15× uncertainty; billing matches cost | 0.5 day plus Cloudflare's answer | |
| 301 | | R2 | P0 | Delete pull request forks after merge or close plus a grace period (for example 14 days), and verify fork storage semantics with a 100 MB test repository and the storage metric. | `lifecycle.rs` sweep (`23 * * * *`), `work` events `pull.merged`/`pull.closed` | Storage proportional to open work instead of all history; removes the 1 TB cliff | 1–2 days | |
| 302 | | R3 | P0 | Mint less and faster: TTL 3,600 s with a 50 min reuse window for credentials that never leave the service; keep 300 s for `git_access`. Derive `remote` from the key (`https://<account>.artifacts.cloudflare.net/git/g1t/<key>.git`) or keep it in the registry `store` column instead of calling `info()`. Retry a 401 once with a fresh token (already done for GET). | `store.rs` `TOKEN_TTL_SECONDS`, `TOKEN_REUSE_MS`, `ArtifactsRepo::access` | Mints about 15× fewer; cold git requests about 0.5 s faster | 0.5 day | |
| 303 | | R4 | P0 | Enforce limits in front of Artifacts: refuse objects over 32 MB and pushes that would take a repository over about 950 MB, with a git `ng` line (same framing as `declined`); stream the push body to Artifacts instead of buffering it twice, keeping only the command section and at most `MAX_SCANNED_PUSH` for scanning; stream `land` packs straight from upload-pack into receive-pack. | `git_http.rs` `forward`, `secret_scan.rs`, `land.rs` | Clear errors instead of late failures; no isolate OOM on 50–100 MB pushes | 2–3 days | |
| 304 | | R5 | P0 | Resilience: map `ArtifactsError.code` and HTTP 429/5xx to retry with jittered backoff (reads, mints, ref listings only; never a receive-pack), a per-isolate circuit breaker, `Retry-After` to git, and an "Artifacts" component on status.g1t.sh fed by the GraphQL error metrics (`rateLimited`, `serverError`, `storageLimitReached`). | `store.rs`, `git_http.rs`, `apps/status` | Brief Artifacts errors stop reaching users; incidents are visible | 2 days | |
| 305 | | R6 | P1 | Pack cache for full clones: when an upload-pack request has wants and no haves, key it by (repo id, `refs_version`, hash of the request) and serve the pack from R2; fill it on a miss with a tee of the response. Sandboxes clone the same commit repeatedly per pull request. | `git_http.rs`, `refs_cache.rs` pattern, new R2 bucket | Removes about 2 s of pack building per repeat clone and a large share of `pull` operations | 3–4 days | |
| 306 | | R7 | P1 | Shard namespaces: bindings `ARTIFACTS_0…N` plus `ARTIFACTS_EU`; record each repository's namespace in the registry `store` column (already read by `remember_store`); new repositories and forks go to the least-loaded shard; forks in their own namespaces. | `services/repos/wrangler.jsonc`, `store.rs`, `registry.rs` | Multiplies the control-plane ceiling; enables EU residency | 3 days | |
| 307 | | R8 | P1 | Sandboxes clone less: `--depth=1` for checks and deploy; `--filter=blob:none` only where lazy blob fetches are few (review of a small diff); keep full clones for agent runs and merges. Or restore a cached sandbox snapshot and `git fetch` (Sandbox SDK backups). | `crates/runner/src/*.rs` clone calls | Measured: 3.8 s (depth 1) and 3.4 s (blobless) against 5.4 s for g1t; fewer bytes from Artifacts | 1–2 days | |
| 308 | | R9 | P1 | Cache mutable reads by `refs_version`: `log(branch, n)`, `branches()` (parse the kept `refs_cache` advertisement instead of a new `info/refs`), and `readFile` (resolve the path through cached trees, then a cached `readBlob`). | `store.rs`, `refs.rs`, `lib.rs` `tree`/`blob`/`branches` | Most web page views stop reaching Artifacts | 2 days | |
| 309 | | R10 | P1 | Calm mergeability: coalesce re-checks per repository (one pass per target head, at most once a minute), resolve the target head once and `log` by hash (already cached), and cache `readCommit` like other objects. | `services/work/src/mergeability.rs`, `lib.rs` `divergence`, `descends_from` | The largest scenario-B hot spot drops by an order of magnitude | 1–2 days | |
| 310 | | R11 | P2 | Back up every repository to R2: a nightly incremental `git bundle` per repository whose `refs_version` changed, made in a Container (Workers cannot run git), with a restore drill. | new job in `services/runner` or a Container | Durability independent of the beta; also the export path | 3–5 days | |
| 311 | | R12 | P2 | Make the git store a real fallback: run `deploy/self-host/gitstore` on a host with persistent disk (Containers' disk is ephemeral and at most 20 GB), restore from the R2 bundles, and switch `GitStore` per namespace shard. | `deploy/self-host/gitstore`, `store.rs` | A tested exit path; not a hot standby | 1–2 weeks | |
| 312 | | R13 | P2 | Take the operation counter off the request path: record the count in `waitUntil` (or Analytics Engine), and check limits against a value cached for a minute. | `lib.rs` `git_limits`, `git_ops.rs` | Saves 63–98 ms on every fetch and push; removes a hot D1 row per workspace-hour | 1 day | |
| 313 | | R14 | P2 | Try Smart Placement or a placement hint for `g1t-repos`, which makes several sequential calls to Artifacts per request; compare `Server-Timing` before and after. | `services/repos/wrangler.jsonc` | Possibly lower latency for distant users | 0.5 day | |
| 314 | | R15 | P3 | Large files: an LFS endpoint backed by R2 for files over 32 MB. | new route in `services/repos` | Repositories with binaries can move to g1t | 1–2 weeks | |
| 315 | |
| 316 | ## 7. Questions for Cloudflare |
| 317 | |
| 318 | Ask in the Artifacts beta channel (the [beta request form](https://forms.gle/DwBoPRa3CWQ8ajFp7) is the |
| 319 | listed contact), through our account team, and in the Cloudflare Developers Discord. As a git |
| 320 | competition entrant (due 2026-10-14) we can also ask the competition organisers. Get answers in writing. |
| 321 | |
| 322 | 1. Exactly what is billed as an operation? Is each `upload-pack` and `receive-pack` request one, or each |
| 323 | clone or fetch? Do `info/refs`, `ls-refs`, binding calls (`get`, `info`, `createToken`, `log`, |
| 324 | `readTree`, `readBlob`, `readCommit`, `readFile`) or token creation count? |
| 325 | 2. Does `fork` share objects with its source or copy them? How is a fork's storage billed? Does |
| 326 | `objects` in the fork response mean objects copied? |
| 327 | 3. Is protocol v2 `filter` (partial clone) supported, given it works today but the docs say it does not? |
| 328 | 4. Do binding calls count toward "2,000 requests per 10 seconds per namespace"? What happens past it |
| 329 | (429, `rateLimited`, queueing)? Can it be raised per namespace? |
| 330 | 5. Read-after-write: once a receive-pack response has ended, are refs visible to `log`, `info/refs` and |
| 331 | other locations at once? |
| 332 | 6. Is `receive-pack` with `atomic` supported for multi-ref pushes? |
| 333 | 7. Where does a repository's Durable Object live, can it move, and can we hint a region per repository? |
| 334 | 8. What SLA, support path and incident communication apply during open beta? When is GA? |
| 335 | 9. Is there an export or snapshot restore for a deleted or corrupted repository? How long are R2 |
| 336 | snapshots kept? |
| 337 | 10. Can the 1 GB per repository and 32 MB per file limits be raised, and what does a client see when a |
| 338 | push crosses them? |
| 339 | 11. Does Artifacts garbage-collect or repack? Do deleted branches and force pushes free storage, and when? |
| 340 | 12. Can the 1 TB account limit be raised now, ahead of launch? |
| 341 | 13. Are per-repository event subscriptions practical at tens of thousands of repositories, or is an |
| 342 | account-level `pushed` event planned? |
| 343 | |
| 344 | ## 8. Launch blockers |
| 345 | |
| 346 | - **Before 2026-10-14:** R1 (know what is billed and count the same thing) and R2's verification (fork |
| 347 | storage semantics). If forks copy objects, R2's deletion is a launch blocker for agent-heavy |
| 348 | workspaces, because the 1 TB account limit would stop every push. |
| 349 | - **Before agent volume ramps (about 1,000 pull requests a day):** R2 deletion, R3, R4, R5. |
| 350 | - **Before a few thousand active workspaces:** R6, R7, R9, R10. |
| 351 | - Everything else is resilience and exit planning, ideally done while the product is still invite-only. |
| 352 | |
| 353 | ## 9. What was built (2026-10-06) |
| 354 | |
| 355 | Code in `services/repos` unless named; one migration, |
| 356 | `migrations/0011_artifacts_meters_forks_health.sql` (new columns on `repos`, new tables |
| 357 | `artifacts_meters`, `operation_mapping`, `store_health`; additive, no backfill). R11 added |
| 358 | `migrations/0013_backups.sql` (a new table, `repo_backups`, and one `operation_mapping` row; |
| 359 | additive). R7 added `migrations/0014_namespace_moves.sql` (two columns on `repos`, |
| 360 | `writes_paused_until` and `writes_paused_for`; new tables `repo_moves` and `repo_move_copies`; |
| 361 | additive) and, in identity, `services/identity/migrations/0026_workspace_residency.sql` (one column, |
| 362 | `workspaces.data_residency`; additive). R12 needs no migration. |
| 363 | |
| 364 | | # | Status | What | |
| 365 | | --- | --- | --- | |
| 366 | | R1 | Built; Cloudflare's answer still needed | Every interaction with the store is metered raw (`meters.rs` → `artifacts_meters`, per day, namespace, repository, workspace and meter, with bytes where known): client git (`git.info_refs`, `git.ls_refs`, `git.fetch`, `git.receive_pack`), g1t's own git (`internal.git.*`: landing, catch-up, mirrors, branch listings, fork retirement), every binding call (`binding.get`, `binding.create_token`, `binding.log`, `binding.read_tree`, …, each retry included), and answers g1t served from its own cache (`cache.*`, never operations). Which meters are operations is data: `operation_mapping` (`cost_operations` for g1t's bill, `billable_operations` for workspaces), read every 5 minutes, changed with `set_operation_mapping` without a deploy. Default: `git.fetch`, `git.receive_pack`, `internal.git.fetch`, `internal.git.receive_pack`, `binding.create`, `binding.fork`, `binding.delete` = 1, everything else 0. `git_operations` (what billing reads) is filled from the meters × `billable_operations`, by the hour. RPCs: `artifacts_usage { from, to, workspace?, by_repo? }` (raw meters and the mapping, for the reconciler), `operation_mapping`, `set_operation_mapping { meter, cost_operations, billable_operations, note? }`. Script: `scripts/ops/artifacts-usage.mjs`. | |
| 367 | | R13 | Built | Nothing on the request path writes D1 for counting. Meters add up per isolate and are written in one batch from `ctx.wait_until` after every request (and at the end of the cron and queue handlers); a failed write is kept for the next. The free-workspace slow-down decides from counts the isolate read back after its last write plus what it added since (`git_ops::standing`, at most 10 minutes old) and billing's plan answer kept 5 minutes. The 63–98 ms `kept` step's D1 upsert is gone. A workspace whose counts this isolate never read is not slowed: nothing slows anyone on a guess. | |
| 368 | | R2 | Built; the fork storage test is yours to run | `pull.merged` and `pull.closed` set the fork's `retire_after` (`FORK_RETENTION_DAYS`, 1 day in production; 7 when unset); `pull.reopened` clears it, or makes the fork again. The hourly sweep (`23 * * * *`, 25 a run) keeps the fork's head in its repository as `refs/pull/<pull id>/head` (only missing objects travel; an empty pack when merged), records `retired_at` and `retired_head`, then deletes the fork from the store (a failed delete puts the row back). Reads of a retired fork (the pull request's changes, divergence, tree, blob, log, branches) are answered from the repository with the fork's branch mapped to the kept head (`forks.rs` `Viewed`). Anything that writes or uses git on it (git over HTTPS, `git_access`, catch-up, land, `delete_branch`) makes it again first (`revive`: fork, then move its branch to the head) and schedules it to go again. Work never emits `pull.reopened` today; the handler is ready for it. Script: `scripts/ops/fork-storage-test.mjs`. | |
| 369 | | R3 | Built | Credentials g1t uses itself: TTL 3,600 s, reused for 50 minutes (isolate and KV, key `cred2:<key>:<scope>:internal`). `git_access` hands out its own: TTL 300 s, reused 180 s (`…:handout`); `refs_open` still uses 300 s. Every internal path (land, catch-up, mirrors, branch listing, commits, deleting a branch) now reuses kept credentials instead of minting each time. The remote is worked out as `https://<account>.artifacts.cloudflare.net/git/<namespace>/<name>.git` (the documented format, `api/git-protocol`), learned per namespace from the first `info()` an isolate makes, which runs alongside `createToken` and so costs no time; after that a mint is `get` and `createToken`. Optional `ARTIFACTS_REMOTE_BASE` skips even the first `info()`. | |
| 370 | | R4 | Built | Pushes are read as they arrive (`request.stream()`) and walked by `pack_limits::PackSizer` (each object inflated into a 32 KiB window and thrown away): an object over 32 MB (a delta measured by the object it makes), or a push taking the repository and its forks (`stored_bytes`) past `REPO_STORAGE_LIMIT_BYTES` (950 MB), is declined with `ng` lines and `remote:` text; a repository already at the limit is refused at the push's `info/refs` in plain text. Up to 24 MiB is kept, scanned and sent on as one copy, not three. Past 24 MiB push protection cannot read the push, so it is declined (`LARGE_PUSHES=refuse`, failing closed) with a command to push in parts, the 100 MB network limit named; `LARGE_PUSHES=unscanned` streams it to the store instead, still size-checked (a violation ends the stream before the pack's checksum, so the store keeps nothing). A pack too large for the scanner to inflate (48 MB inflated) is declined the same way instead of let through. Landing streams: upload-pack's side-band answer is taken apart chunk by chunk (`pack_limits::Sideband`) straight into the receive-pack body. | |
| 371 | | R5 | Built | `resilience.rs` sorts errors into rate limited, transient (`INTERNAL_ERROR`, `UPSTREAM_UNAVAILABLE`, `*_IN_PROGRESS`, no code, HTTP 5xx) and permanent. Binding reads, `get`, `info`, `createToken`, `create` and `delete` try up to 3 times with exponential backoff and jitter (80 ms base, 400 ms for rate limits, 2 s cap); `fork` and every receive-pack never retry. Git reads (`info/refs`, upload-pack) retry on 429 and 5xx. Per isolate, each namespace has a breaker that opens after 5 transient failures in a row, for 10 s, then lets one probe through. Busy answers reach git as 429 (rate limited) or 503, with `Retry-After: 5`; the site's read RPCs (`tree`, `blob`, `log`, `branches`, `blame`, `compare`) answer an `Outcome` failure saying the git storage is busy; other RPCs answer 503 with the same words. Health is counted by the minute (`store_health`) and served by the `store_health { minutes }` RPC; status.g1t.sh lists **Git storage** through a new `REPOS` service binding (down: 25% or more of at least 5 calls failed, or the breaker refused calls; degraded: rate limited, or a mean call over 1.5 s). | |
| 372 | | R9 | Built | `log(branch)`, `branches()` and `read_file(ref, path)` are kept in the colo cache under the repository's `refs_version` (5 minutes at most, and only while `refs_cache::usable`), and by commit hash for good; a log by branch also fills the by-hash entry; `readCommit` (parents) is kept for good. Read RPCs open repositories through `read_git`, which sets the version. | |
| 373 | | R10 | Built in repos; work unchanged | `divergence` works out the target's side once per target head per isolate (`coalesce.rs`: the head under the refs version, then the history by hash, kept 60 s), and what the target changed between two trees once per pair (10 minutes). `readCommit` and logs by hash come from the cache. Work's fan-out (`after_push`, up to 100 pull requests) is unchanged: its 100 `divergence` calls now cost one walk of the target instead of 100. | |
| 374 | | R7 | Built; the namespaces are yours to make | `shards.rs`: bindings named in `ARTIFACTS_NAMESPACES` (JSON, binding → namespace; `ARTIFACTS` → `g1t` always there), a repository's namespace kept in its `store` column as `<namespace>/<key>` (no prefix means the `ARTIFACTS` namespace, so every existing key reads the same), forks always in their repository's namespace. **Placing** (`Placement::choose`, loads from `namespaces.rs`): among `ARTIFACTS_NEW_REPOS`, the healthy namespaces (bound, taking writes, not failing, under `ARTIFACTS_NAMESPACE_LIMITS`' `max_repos`, busiest minute under 70% of the 12,000-a-minute limit) within 100 repositories or 5% of the emptiest, spread by an FNV hash of the id; loads are read (one D1 query each for the registry and `store_health`, kept a minute) only when there is more than one to choose from. **Moving** (`moves.rs`): `move_repository` or `scripts/ops/artifacts-namespaces.mjs move` queues one; the hourly sweep pauses writes, copies every ref of the repository and its working copies over git (one streamed upload-pack into one receive-pack each), switches every `store` key in one batch, and deletes the old copies after 7 days. **EU residency**: identity's `workspaces.data_residency`, set by an owner in the workspace's settings (shown only once `storage_options` says an EU namespace takes repositories), read by the repos service at creation only while `ARTIFACTS_EU_NAMESPACE` is set; an EU workspace's repository goes to that namespace or is not made. **Health and limits**: `namespaces` RPC and `scripts/ops/artifacts-namespaces.mjs`. Nothing changes until bindings and variables name new namespaces. See "R7: sharding, moves and EU residency" below. | |
| 375 | | R8 | Built | `crates/runner/src/clone.rs`: every sandbox clones at `--depth=1` (a full g1t clone took 5.4 s, depth 1 took 3.8 s). Work that merges (catch-up, the merge queue, merge checks, a review's diff) deepens 50, 500, then 5000 commits until the two sides share one, and fetches everything only as the last resort (`share_history`). `G1T_CLONE_DEPTH` (0 or `full` for everything) and `G1T_CLONE_FILTER=blob:none` change it per runner. | |
| 376 | | R6 | Built; the bucket must exist before it deploys | `pack_cache.rs`: an upload-pack POST with wants and no `have` or `shallow` lines (a fresh clone, the sandboxes' `deepen 1` ones included), uncompressed and at most 1 MiB, is keyed `packs/<repo id>/<refs_version>/<sha256>` over the request normalized: protocol v2 capabilities without `agent=`/`session-id=` and its arguments, each sorted and deduplicated; v0/v1 wants sorted, the first want's capabilities split off, sorted and without `agent=`, then `deepen`/`filter` lines, a flush and `done`. Only while `refs_cache::usable` (the version known, and no push credential out of g1t's hands), so never across a refs change. Looked up after authorization, alongside the free-workspace limits and the kept refs answer; a hit streams from the bucket (`Server-Timing` `pack;desc=hit`). A miss streams the store's 200 to git through a tee that copies it to a fill in `ctx.wait_until` (at most 5 MiB queued between them, 2 fills per isolate, one per key): under 5 MiB it is one `put` once it all arrived; larger, 5 MiB multipart parts completed only after the last part and a check that it is one whole side-band pack (well-formed pkt-lines, `PACK` on channel 1, no `ERR` or channel 3, a closing flush). Over 200 MB, a queue that falls behind, git going away or the store's stream failing lets the fill go and aborts the upload; nothing partial can be read. Meters `pack_cache.hit` (with the bytes served) and `pack_cache.miss` (counted with `record`, bytes added at the end), neither an operation by default; a hit records no `git.fetch`. Storage is behind the `PackStore` port, whose adapter puts the shared `BlobStore` (`crates/blobstore`) behind it: R2 hosted (`GIT_PACKS`, bucket `g1t-git-packs`, lifecycle: packs deleted after 7 days, unfinished uploads after 1), or with `PACK_STORE=s3` the S3 bucket `PACK_S3_BUCKET` names (self-hosted: MinIO's `g1t-git-packs`, packs deleted after 7 days, unfinished uploads by MinIO after 24 hours; `node services/repos/dev/clone-check.mjs --s3` checks it). With neither, nothing is kept. | |
| 377 | | R11 | Built; not yet deployed | Nightly `git bundle` backups to the `g1t-backups` R2 bucket, and a restore drill. Migration `0013_backups.sql` (`repo_backups`, and an `operation_mapping` row). See "R11: backups and the restore drill" below. | |
| 378 | | R12 | Built; the host is yours to make | `scripts/ops/restore-to-gitstore.mjs` rebuilds every repository from its bundle chain into the git store (`deploy/self-host/gitstore`, now with namespaced keys, `<root>/<namespace>/<name>.git`, and `GITSTORE_READ_ONLY=1`), on the host or over its API, and later lists and reconciles what the fallback took. `fallback.rs`: with `GIT_FALLBACK_URL`, `GIT_FALLBACK_SECRET` and `GIT_FALLBACK_NAMESPACES` set, a namespace's `GitStore` calls go to the git store's API instead of the Artifacts binding (same metering, retries and breaker, its own health as `<namespace>@fallback`), read-only unless `GIT_FALLBACK_WRITES=allow`: writes are refused before they are asked, and say so in words; kept ref listings and packs are not used, nor backups cut. status.g1t.sh shows Git storage degraded meanwhile. See "R12: the fallback store and the outage runbook" below. | |
| 379 | |
| 380 | ### R1: reading `scripts/ops/artifacts-usage.mjs` |
| 381 | |
| 382 | ```sh |
| 383 | export CLOUDFLARE_API_TOKEN=<token with Account Analytics: Read (and D1: Read, or set CLOUDFLARE_D1_TOKEN)> |
| 384 | node scripts/ops/artifacts-usage.mjs # last 31 days, a table |
| 385 | node scripts/ops/artifacts-usage.mjs --days 7 --json > usage.json |
| 386 | ARTIFACTS_NAMESPACE=g1t node scripts/ops/artifacts-usage.mjs |
| 387 | ``` |
| 388 | |
| 389 | It prints, per day, Cloudflare's `pull`, `push`, `create`, `fork` and `delete` events and its |
| 390 | errors beside g1t's fetch and push meters and `git_operations`, then, for each Cloudflare event, |
| 391 | the ratio Cloudflare ÷ g1t for several combinations of meters. Read it like this: |
| 392 | |
| 393 | - `pull` ≈ `git.fetch + internal.git.fetch` (ratio 1.00): Cloudflare counts one pull per |
| 394 | upload-pack fetch, as assumed. Keep the default mapping. |
| 395 | - `pull` ≈ a combination with `git.ls_refs` or `git.info_refs`: listing refs counts too. Set |
| 396 | `cost_operations` for those meters to 1 (`set_operation_mapping`), and decide whether |
| 397 | `billable_operations` follows (cost pass-through says yes). |
| 398 | - Every ratio well under 1: Cloudflare counts per clone or fetch session, not per request. |
| 399 | Ratios over 1: something reaches Artifacts that g1t does not meter, such as sandboxes pushing |
| 400 | directly with handed-out credentials. |
| 401 | - Only days after the meters were deployed compare; before that only `git_operations` exists. |
| 402 | - Binding calls do appear: Cloudflare's events include `read` and `token_create` actions (and |
| 403 | `namespace_*`) besides the five documented ones. If Cloudflare says they are billed, map the |
| 404 | `binding.*` meters in `operation_mapping`. |
| 405 | |
| 406 | A global API key works in place of the token: `CLOUDFLARE_API_KEY` with `CLOUDFLARE_EMAIL` |
| 407 | (both scripts). |
| 408 | |
| 409 | **Result, 2026-10-06** (31 days; g1t's meters cover only 2026-10-06, the day they shipped): |
| 410 | |
| 411 | | Cloudflare event | 31 days | 2026-10-06 | g1t's meters, 2026-10-06 | |
| 412 | | --- | --- | --- | --- | |
| 413 | | `read` | 137,225 | 107,616 | `binding.get` 85,206, `read_file` 49,846, `read_tree` 9,208, `read_blob` 4,964, `log` 2,798 | |
| 414 | | `pull` | 646 | 101 | `git.fetch` 29, `git.ls_refs` 26, `git.info_refs` 426 (+ 129 internal) | |
| 415 | | `push` | 385 | 10 | `git.receive_pack` 3 | |
| 416 | | `token_create` | 2,721 | 338 | `binding.create_token` 34 | |
| 417 | | `fork` / `create` / `delete` | 96 / 14 / 8 | 2 / 0 / 0 | | |
| 418 | | errors | 699 (688 client) | 476 client | | |
| 419 | |
| 420 | - `pull` is not one per upload-pack fetch: 101 pulls against 29 fetches on the one day both |
| 421 | exist. Over 31 days `pull` ≈ fetch + ls-refs + info/refs (ratio 1.06), so listing refs likely |
| 422 | counts as a pull. Not yet changed in `operation_mapping`: one day of meters is too little, and |
| 423 | re-check after a week before setting `cost_operations` for `git.info_refs` and `git.ls_refs`. |
| 424 | - `read` is the open question that matters. If reads are billed as operations at $0.15 per |
| 425 | 1,000, today's demo-scale traffic alone is about 3.2 million a month (~$480). Ask Cloudflare |
| 426 | (Q1) before 2026-10-14. Either way the volume is mostly waste: every repos call opens a handle |
| 427 | with `get` even when the answer is cached, and the object cache may not be hitting (no hit/miss |
| 428 | meter yet). Done on 2026-10-06: the handle's `get` waits for the first call that needs the |
| 429 | store (an answer from a cache, or `branches` over git, costs none); objects named by hash are |
| 430 | kept in the isolate (16 MB, oldest out first) ahead of the Cache API, and every look is |
| 431 | metered (`cache.memory_hit`, `cache.edge_hit`, `cache.miss`); issue and comment events start |
| 432 | nothing and read nothing when the synced `workflows` table has no workflow listening. Next: |
| 433 | read `cache.edge_hit` against `cache.miss` after a day; if the Cache API never hits from a |
| 434 | Worker reached only by service bindings, put objects in KV instead. Still to do: caller |
| 435 | attribution in the meters. |
| 436 | - 476 client errors on 2026-10-06 are unexplained; the fetch fix below accounts for some (every |
| 437 | failed negotiation was one). |
| 438 | |
| 439 | ### R2: running and reading `scripts/ops/fork-storage-test.mjs` |
| 440 | |
| 441 | ```sh |
| 442 | export CLOUDFLARE_API_TOKEN=<token: Artifacts edit, Account Analytics read> |
| 443 | node scripts/ops/fork-storage-test.mjs schema # which Artifacts analytics datasets exist |
| 444 | node scripts/ops/fork-storage-test.mjs run --keep # namespace g1t-storage-test: 100 MB repository, 5 forks |
| 445 | node scripts/ops/fork-storage-test.mjs measure # again tomorrow (storage is billed as a daily peak) |
| 446 | node scripts/ops/fork-storage-test.mjs cleanup # delete the 6 repositories |
| 447 | ``` |
| 448 | |
| 449 | It works only in its own namespace, through Cloudflare's REST API, never through g1t. Read: |
| 450 | |
| 451 | - Fork timing and response: a fork that returns in well under a second, with `objects` near the |
| 452 | source's count, is metadata (sharing). Seconds per fork, growing with size, suggests copying. |
| 453 | - Storage figures (any dataset `schema` lists besides `artifactsEventsAdaptiveGroups`): about |
| 454 | 100 MB after the forks means sharing; about 600 MB means each fork copied. The documentation |
| 455 | lists no storage dataset today, so this may print nothing: then the next day's usage in the |
| 456 | dashboard (Billing → Artifacts storage) is the measure, and the question stays with Cloudflare (Q2). |
| 457 | - `events`: `storageLimitReached` or other errors during the test. |
| 458 | |
| 459 | Either way R2 retires forks; the answer decides `FORK_RETENTION_DAYS` (shared: a week is fine; |
| 460 | copied: shorten it to 1 or 2 days and ask Cloudflare to raise the 1 TB account limit). |
| 461 | |
| 462 | **Result, 2026-10-06: forks are stored and billed as copies.** A 100 MB source took 57 s to |
| 463 | push; each of 5 forks took 4–6 s. The storage dataset (`artifactsStorageAdaptiveGroups`, |
| 464 | `max.repositorySizeBytes`) gave every fork the source's full 105,582,592 bytes: about 633 MB for |
| 465 | the six, not about 106 MB. Whatever Artifacts shares underneath, storage billing and the 1 TB |
| 466 | account limit see full copies. So: |
| 467 | |
| 468 | - `FORK_RETENTION_DAYS` is 1 in production (changed 2026-10-07). |
| 469 | - g1t meters a workspace's storage once per repository (`stored_bytes`), so an open pull |
| 470 | request's working copy is Cloudflare cost g1t absorbs: about $0.05 a month per 100 MB per open |
| 471 | pull request. Small now; decide whether open working copies count toward a workspace's |
| 472 | storage before agent pull requests reach thousands. |
| 473 | - Ask Cloudflare whether forks share objects physically, and for a higher account limit. |
| 474 | |
| 475 | Also found while testing (fixed in `git_http.rs`): the store answers a protocol v2 fetch that is |
| 476 | still negotiating, and whose `have`s it does not know, with `acknowledgments`, `NAK`, then a pack. |
| 477 | git refuses that ("expected no other sections to be sent after no 'ready'"). g1t now ends such |
| 478 | an answer after the acknowledgments with a flush, and the client negotiates again. Report it to |
| 479 | Cloudflare. |
| 480 | |
| 481 | ### R7: sharding, moves and EU residency |
| 482 | |
| 483 | Every piece is built and off. Production behaves exactly as before until the namespaces below are |
| 484 | made, bound and named: with only `ARTIFACTS` bound and `ARTIFACTS_NEW_REPOS` empty, new repositories |
| 485 | go to `g1t`, nothing extra is read from D1 when one is made, and identity is never asked about |
| 486 | residency. |
| 487 | |
| 488 | | Variable (`services/repos/wrangler.jsonc`) | What it does | |
| 489 | | --- | --- | |
| 490 | | `ARTIFACTS_NAMESPACES` | JSON, binding to namespace. `ARTIFACTS` is always there (`g1t` unless named). A name without a binding is logged and left out. | |
| 491 | | `ARTIFACTS_NEW_REPOS` | Comma-separated namespaces new repositories go to. Empty: `g1t`. A name that is not bound is passed over. | |
| 492 | | `ARTIFACTS_EU_NAMESPACE` | The namespace EU workspaces' new repositories go to. Unset: residency is never read, and the setting is never offered. | |
| 493 | | `ARTIFACTS_NAMESPACE_LIMITS` | Optional JSON, `{"g1t": {"max_repos": 50000}}`. A namespace at its limit takes no new repositories while another can. | |
| 494 | |
| 495 | **Placing a new repository** (`shards.rs` `Placement::choose`, loads from `namespaces.rs`). For a |
| 496 | workspace that keeps its data anywhere: among the namespaces in `ARTIFACTS_NEW_REPOS`, the healthy |
| 497 | ones (bound, taking writes, not failing, under `max_repos`, their busiest minute in the last hour |
| 498 | under 70% of the 12,000-a-minute control-plane limit), and of those the ones within 100 repositories |
| 499 | or 5% of the emptiest, spread by the id's FNV hash. If none is healthy, the usable ones the same way |
| 500 | (never a read-only one); if none is usable, `g1t`. Failing means this isolate's breaker is open, or a |
| 501 | quarter of at least 5 calls in the last 5 minutes failed (`store_health`). Loads cost two D1 queries |
| 502 | (the registry grouped by namespace, `store_health` for the last hour), kept a minute per isolate, and |
| 503 | are read only when more than one namespace could take the repository. A pull request's working copy |
| 504 | always goes where its repository is. For an EU workspace: `ARTIFACTS_EU_NAMESPACE` if it is bound and |
| 505 | takes writes; otherwise the repository is not made, and the person is told why (409, "This workspace |
| 506 | keeps its data in the EU, and EU storage cannot take new repositories right now."). It is never placed |
| 507 | elsewhere. |
| 508 | |
| 509 | **EU residency, as a workspace sees it.** Identity keeps `workspaces.data_residency` (NULL for |
| 510 | anywhere, `eu`), changed by an owner with `set_workspace_residency` and read with |
| 511 | `workspace_residency`; a change is audited as `workspace.residency_changed`. The workspace's |
| 512 | **Settings** page shows **Data residency** only when the repos service's `storage_options` says |
| 513 | `euAvailable` (an EU namespace is bound and takes writes), or when the workspace already chose the EU. |
| 514 | It applies to repositories made after it is saved, by any path that creates one (the site, the API, |
| 515 | push to create, imports). Existing repositories stay where they are until moved. A transfer keeps a |
| 516 | repository's store key, so a repository transferred into an EU workspace stays where it was: move it. |
| 517 | The guide is `apps/docs/src/content/docs/guides/workspaces.md`, "Data residency". Not in the public |
| 518 | API or MCP yet. |
| 519 | |
| 520 | **Moving a repository** (`moves.rs`, migration 0014). It keeps its id, path, rows and history; only |
| 521 | `store` changes. |
| 522 | |
| 523 | ```sh |
| 524 | node scripts/ops/artifacts-namespaces.mjs move acme/rocket g1t-us-1 # queues it (one INSERT) |
| 525 | node scripts/ops/artifacts-namespaces.mjs moves # queued, moving, moved, failed, cleaned, diverged |
| 526 | ``` |
| 527 | |
| 528 | Services can queue one with the `move_repository { repo_id, namespace, requested_by? }` RPC, which |
| 529 | checks it first, and list them with `repository_moves { limit? }`. The hourly sweep (`23 * * * *`) |
| 530 | runs one queued move per hour: |
| 531 | |
| 532 | 1. Writes to the repository and every working copy of its pull requests are paused |
| 533 | (`writes_paused_until`, 20 minutes at most, so a move that dies releases them on its own). A push |
| 534 | waits up to 20 seconds, polling every 2, and then is told: "acme/rocket is paused for maintenance |
| 535 | (moving to g1t-us-1); changes to it wait a few minutes. Try again shortly." Merges, catch-ups, |
| 536 | commits from the web, branch changes, mirror catch-ups, new pull request working copies and push |
| 537 | credentials for sandboxes wait the same way. Removing a working copy waits for the next sweep. |
| 538 | 2. Push credentials already handed out reach the store directly, so the move waits for |
| 539 | `refs_open_until` to pass (up to 7 minutes in the run; longer goes back in the queue), then 5 |
| 540 | seconds for pushes in flight. |
| 541 | 3. Each one is made in the new namespace under the same name, and every ref is copied with one |
| 542 | upload-pack from the old copy streamed into one receive-pack to the new (`land.rs` `copy_refs`), |
| 543 | never held in memory. Until both list the same refs and the old one did not move during the copy, |
| 544 | only what changed is copied again, three rounds at most. Removed working copies have nothing to |
| 545 | copy: their rows follow. |
| 546 | 4. One D1 batch points every row at its new key, moves its `refs_version` (so no kept ref listing, |
| 547 | pack or versioned read is used again), lifts the pause, and records each copy's refs in |
| 548 | `repo_move_copies`. |
| 549 | 5. After 7 days the sweep deletes each old copy whose refs still say what was copied. One that |
| 550 | changed (a push that slipped past the pause and landed in the old copy) is kept, and the move is |
| 551 | marked `diverged` with the keys; push its refs to the new copy by hand. While an old copy is kept, |
| 552 | its name stays taken, so no new repository adopts it. |
| 553 | |
| 554 | A move that fails before step 4 deletes what it made in the new namespace, lifts the pause, and is |
| 555 | tried again in the next sweep, three times in all. The copy is metered like landing |
| 556 | (`internal.git.fetch` and `internal.git.receive_pack`, billable 1 each to the repository's workspace |
| 557 | by default), plus `binding.create`. A copy is bounded by the cron's wall time (15 minutes), which |
| 558 | streams well past the 1 GB repository limit. Not verified against Artifacts yet: run the first move on |
| 559 | a test repository and compare `git ls-remote` of both copies. |
| 560 | |
| 561 | **Health and limits.** `namespaces` (RPC) answers each bound namespace's repositories, working copies |
| 562 | and stored bytes from the registry; its busiest minute in the last hour against 12,000 a minute; |
| 563 | calls, errors and rate limits in the last hour; whether it is failing, on the fallback, writable, |
| 564 | default, EU, and takes new repositories; and its `max_repos`. |
| 565 | |
| 566 | ```sh |
| 567 | node scripts/ops/artifacts-namespaces.mjs # a table, and a line for anything to act on (exit 1 then) |
| 568 | node scripts/ops/artifacts-namespaces.mjs --cloudflare # also Cloudflare's event counts and each namespace's jurisdiction |
| 569 | node scripts/ops/artifacts-namespaces.mjs --json |
| 570 | ``` |
| 571 | |
| 572 | It reads `services/repos/wrangler.jsonc` for what is configured, so it says what the next deploy |
| 573 | will do: a namespace named but not bound or not made, an EU namespace made without the EU |
| 574 | jurisdiction, one past 70% of the limit or at its `max_repos`, one rate limited, one served from the |
| 575 | fallback. |
| 576 | |
| 577 | #### What you must create (R7) |
| 578 | |
| 579 | None of this is needed until one namespace is not enough, or an EU customer asks. |
| 580 | |
| 581 | 1. Make the namespaces, with a token that can edit Artifacts. A namespace's jurisdiction is fixed when |
| 582 | it is made, and is not part of the binding (Wrangler's schema has only `binding`, `namespace` and |
| 583 | `remote`): |
| 584 | |
| 585 | ```sh |
| 586 | # A US shard, unrestricted like today's g1t, and an EU one. |
| 587 | curl -X POST "https://api.cloudflare.com/client/v4/accounts/1e6f2cffa3f445920836e8ebe446bb58/artifacts/namespaces" \ |
| 588 | -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json" \ |
| 589 | --data '{"namespace":"g1t-us-1"}' |
| 590 | curl -X POST "https://api.cloudflare.com/client/v4/accounts/1e6f2cffa3f445920836e8ebe446bb58/artifacts/namespaces" \ |
| 591 | -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json" \ |
| 592 | --data '{"namespace":"g1t-eu","jurisdiction":"eu"}' |
| 593 | node scripts/ops/artifacts-namespaces.mjs --cloudflare # both listed, g1t-eu with jurisdiction eu |
| 594 | ``` |
| 595 | |
| 596 | 2. Bind them, and deploy `g1t-repos` (migrations 0014 and identity's 0026 go first, as the deploy tool |
| 597 | always does). Nothing is placed in them yet: |
| 598 | |
| 599 | ```jsonc |
| 600 | "artifacts": [ |
| 601 | { "binding": "ARTIFACTS", "namespace": "g1t" }, |
| 602 | { "binding": "ARTIFACTS_1", "namespace": "g1t-us-1" }, |
| 603 | { "binding": "ARTIFACTS_EU", "namespace": "g1t-eu" } |
| 604 | ], |
| 605 | "vars": { |
| 606 | "ARTIFACTS_NAMESPACES": "{\"ARTIFACTS\":\"g1t\",\"ARTIFACTS_1\":\"g1t-us-1\",\"ARTIFACTS_EU\":\"g1t-eu\"}", |
| 607 | "ARTIFACTS_NEW_REPOS": "" |
| 608 | } |
| 609 | ``` |
| 610 | |
| 611 | 3. Name them, and deploy again: `"ARTIFACTS_NEW_REPOS": "g1t,g1t-us-1"` spreads new repositories over |
| 612 | both (the emptier first), and `"ARTIFACTS_EU_NAMESPACE": "g1t-eu"` puts **Data residency** in every |
| 613 | workspace's settings. Optionally `"ARTIFACTS_NAMESPACE_LIMITS": "{\"g1t\":{\"max_repos\":50000}}"`. |
| 614 | 4. Move a test repository there and back (`move`, then `moves` after the next :23), and compare both |
| 615 | copies with `git ls-remote`. |
| 616 | |
| 617 | More shards later are the same steps (`g1t-us-2`, `ARTIFACTS_2`). Deploy a binding before naming its |
| 618 | namespace anywhere; a name that is not bound is passed over, never used. |
| 619 | |
| 620 | ### R11: backups and the restore drill |
| 621 | |
| 622 | Every repository whose refs moved is bundled once a night and kept outside the git store, so a |
| 623 | repository can be rebuilt without Artifacts. The flow is in `crates/contracts/src/backups.rs`; |
| 624 | the chain, the manifest and the record are in `services/repos/src/backups.rs`. |
| 625 | |
| 626 | 1. **Queued.** At 02:53 UTC (`53 2 * * *` in `services/repos/wrangler.jsonc`) the repos service |
| 627 | queues the repositories that are due, at most `BACKUPS_PER_NIGHT` (200), the longest since |
| 628 | their last backup first. A repository is due when it has never been backed up, when its |
| 629 | `refs_version` went past the one its last backup was cut at, or when a credential that can |
| 630 | push was handed out (`refs_open_until`) after that backup's clone began: a push with such a |
| 631 | credential does not move `refs_version`. Deleted repositories, retired working copies and |
| 632 | pull request working copies (`pulls/…`, whose heads end up in their repository as |
| 633 | `refs/pull/<id>/head`) are not backed up. |
| 634 | 2. **Claimed.** The runner's five-minute sweep claims `BACKUPS_PER_SWEEP` (4) at a time, with at |
| 635 | most `BACKUPS_RUNNING` (6) running (`claim_backups`), and starts a sandbox for each in |
| 636 | `MODE=backup` (`crates/runner/src/backup.rs`). The sandbox is given the job's id and a token |
| 637 | for it, nothing else; the repos service keeps only the token's hash. It has a 60-minute time |
| 638 | cap. Its time is g1t's: it is not metered to the workspace. |
| 639 | 3. **Cut.** The sandbox asks for its job (`POST api.g1t.sh/backups/{job}/spec`, the token in |
| 640 | `x-g1t-backup-token`) and gets a read-only credential for the repository in the store (a |
| 641 | `git_access`-style handout, 5 minutes), the bundle's kind, and the commits the last bundle |
| 642 | ended at. It clones with `--mirror` (every ref, never shallow), writes those commits as refs |
| 643 | of its own, and runs `git bundle create --all --not <them>`, then `git bundle verify`. |
| 644 | When the clone has exactly the refs of the last backup, or git finds nothing new to bundle |
| 645 | (a branch deleted, a ref moved to a commit already kept), no bundle is cut and only the refs |
| 646 | are recorded. |
| 647 | 4. **Sent.** The bundle goes in 32 MiB parts (`PUT /backups/{job}/parts/{n}`), which the API |
| 648 | passes to the repos service and the repos service to an R2 multipart upload; then |
| 649 | `POST /backups/{job}/complete` with every ref, the size, the SHA-256 and the parts. A failure |
| 650 | is `POST /backups/{job}/fail`; a sandbox that dies is failed by the runner. A job is tried 3 |
| 651 | times a night; one running past 3 hours is queued again. |
| 652 | 5. **Recorded.** The manifest gains the entry, and `repo_backups` the refs version the clone began |
| 653 | at, so a push during the backup leaves the repository due the next night. |
| 654 | |
| 655 | Storage, through the `BlobStore` port in `crates/blobstore` (the adapters packages already used): |
| 656 | the `BACKUPS` binding (bucket `g1t-backups`) with `BACKUP_STORE=r2`; any S3-compatible store with |
| 657 | `BACKUP_STORE=s3` and `BACKUP_S3_BUCKET` (self-hosted: MinIO). Without either, backups are off and |
| 658 | the nightly cron does nothing. |
| 659 | |
| 660 | ```text |
| 661 | backups/<repo id>/manifest.json |
| 662 | backups/<repo id>/20261006T025300Z-full.bundle |
| 663 | backups/<repo id>/20261007T025302Z-incr.bundle |
| 664 | ``` |
| 665 | |
| 666 | The manifest (version 1) lists `chain`, oldest first, and `previous`, the chain before it. Each |
| 667 | entry has `id`, `kind` (`full` or `incremental`), `key` (null when only refs moved), |
| 668 | `created_at`, `refs_version`, `refs` (every ref and `HEAD` once it is applied), `prerequisites`, |
| 669 | `size` and `sha256`. The first backup is full; the next ones are incremental, their |
| 670 | prerequisites the last entry's tips, until the chain holds `BACKUP_FULL_EVERY` (30) incremental |
| 671 | ones, when a full one starts a new chain. The chain before that is kept until the next full one |
| 672 | replaces it, so the oldest backup kept is about two chains old. Backups of purged repositories |
| 673 | are removed the night after (50 a night). |
| 674 | |
| 675 | Meters: the clone counts as `internal.git.info_refs` and `internal.git.backup_fetch` with the |
| 676 | bytes it read, on the repository (they show in `artifacts_usage` and |
| 677 | `scripts/ops/artifacts-usage.mjs`). `operation_mapping` has `internal.git.backup_fetch` at 1 for |
| 678 | `cost_operations` and 0 for `billable_operations`: an operation on g1t's bill, never on the |
| 679 | workspace's. The credential's `binding.create_token` is metered as before. |
| 680 | |
| 681 | **The restore drill** (read-only against production: SELECTs on `g1t-repos`, reads of |
| 682 | `g1t-backups` through Wrangler, `git ls-remote` of the live repository): |
| 683 | |
| 684 | ```sh |
| 685 | node scripts/ops/backup-restore-drill.mjs # a repository unchanged since its last backup |
| 686 | node scripts/ops/backup-restore-drill.mjs --repo acme/rocket # this one |
| 687 | G1T_USER=you G1T_TOKEN=g1t_... node scripts/ops/backup-restore-drill.mjs --repo acme/private-thing |
| 688 | ``` |
| 689 | |
| 690 | It downloads the manifest and each bundle of the chain, checks each against its size and SHA-256, |
| 691 | `git bundle verify`s it, fetches it into a new bare repository without following tags, sets every |
| 692 | ref to what the last entry says (and removes the rest), points `HEAD` at the branch at its commit, |
| 693 | and runs `git fsck --connectivity-only`. Then it compares every ref with the manifest and with |
| 694 | `git ls-remote` of the live repository and prints each difference. Exit 0: every ref matches; |
| 695 | 1: a difference; 2: it could not run (a bundle that does not match its manifest is this). Picked |
| 696 | at random, the repository is one whose refs have not moved since its last backup, so any |
| 697 | difference is the backup's. Run it after the first night, then monthly, and after any change to |
| 698 | `backups.rs` or `backup.rs`. `--bundles <dir>` reads a local copy of the bucket instead |
| 699 | (self-hosted: `mc mirror local/g1t-backups <dir>`), with `--repo-id` and `--live <url or path>`. |
| 700 | `npm run test:ops` runs it against bundles cut with git. |
| 701 | |
| 702 | **A real restore into the store**, as it can be done today: |
| 703 | |
| 704 | 1. Run the drill for the repository with `--keep`. It prints where the restored copy is |
| 705 | (`…/restored.git`). Go on only if every ref matches the manifest; differences from the live |
| 706 | repository are what the restore is for. |
| 707 | 2. Tell the workspace, and stop the repository's agents and merge queue for the time. |
| 708 | 3. If its default branch is protected, turn protection off in the repository's settings for the |
| 709 | push: a push that changes a protected branch is declined. |
| 710 | 4. From the restored copy, push every ref as an owner, with an access token that has |
| 711 | `code:write`: |
| 712 | |
| 713 | ```sh |
| 714 | cd /tmp/g1t-drill-…/restored.git |
| 715 | git -c "http.extraHeader=Authorization: Basic $(printf 'you:g1t_...' | base64)" \ |
| 716 | push --force https://g1t.sh/acme/rocket.git 'refs/*:refs/*' |
| 717 | ``` |
| 718 | |
| 719 | It goes through the git door like any push: size limits, push protection (pushes over 24 MiB |
| 720 | per `LARGE_PUSHES`) and the audit log apply, and the refs version moves, so the next night |
| 721 | backs the repository up again. `--force` rewinds refs that went wrong; refs the live |
| 722 | repository has that the backup does not are left alone (`git push --mirror` would delete |
| 723 | them). |
| 724 | 5. Turn protection back on, and run the drill again: every ref now matches the live repository. |
| 725 | |
| 726 | When the repository is gone from the store itself (its key answers not found), there is still no |
| 727 | operator call to make an empty repository under an existing row's key in Artifacts. Until there is, |
| 728 | the fallback store can serve it from its backup (R12). |
| 729 | |
| 730 | To deploy: make the bucket (`npx wrangler r2 bucket create g1t-backups`, a setup step of `repos` |
| 731 | in `deploy/stack.jsonc`), then migration 0013, then `g1t-repos` (the `BACKUPS` binding and the |
| 732 | new cron), `g1t-api` (the `/backups/` door), and `g1t-runner` (a new image: the `backup` mode). |
| 733 | Until the runner is out, queued backups wait; nothing fails. Set `BACKUPS_PER_SWEEP` to `0` on the |
| 734 | runner to stop starting them. |
| 735 | |
| 736 | What is not covered: a pull request's working copy while its pull request is open (its head is |
| 737 | kept in the repository only once the working copy is retired), and anything that is not a git |
| 738 | ref (issues, pull requests and the rest live in D1, which has its own Time Travel). Every backup |
| 739 | clones the whole repository, so a night reads each changed repository in full from the store; |
| 740 | incremental bundles save storage, not reads. |
| 741 | |
| 742 | ### R12: the fallback store and the outage runbook |
| 743 | |
| 744 | A cold fallback, not a hot standby: when Artifacts is down for a namespace, g1t can serve that |
| 745 | namespace's repositories from the self-hosted git store (`deploy/self-host/gitstore`), rebuilt from the |
| 746 | nightly backups (R11). Reads work from the last backup; writes wait, unless you choose otherwise. It is |
| 747 | switched by configuration, per namespace, in seconds and without a build. |
| 748 | |
| 749 | | | Kept restored nightly (recommended) | Restored when needed | |
| 750 | | --- | --- | --- | |
| 751 | | Data served | As of the last backup: at most about a day old | The same | |
| 752 | | Time to switch | Minutes: a last restore pass, then one secret | Download plus restore: about an hour per 100 GB of bundles, 4 at a time | |
| 753 | | Cost | The host, always on | The host only while it is needed, if you make it then | |
| 754 | |
| 755 | **The pieces.** |
| 756 | |
| 757 | - `scripts/ops/restore-to-gitstore.mjs restore` rebuilds each repository from its chain as the restore |
| 758 | drill does (each bundle checked against its size and SHA-256, `git bundle verify`, fetched in order, |
| 759 | refs set to the last entry's, `git fsck --connectivity-only`) into `<root>/<namespace>/<name>.git`, |
| 760 | configured as the git store configures its own, with a `g1t.json` that records what it was restored |
| 761 | from. A second run skips repositories already restored from the same last backup, so a nightly run |
| 762 | only does what changed. `--into <root>` writes on the host; `--gitstore <url>` (with `GITSTORE_SECRET`) |
| 763 | goes through the store's API and git, for a store that is not read-only. `--bundles <dir>` reads a |
| 764 | local copy of the bucket; `--offline` takes the repositories from the manifests in it, so the host |
| 765 | needs no database access (a repository moved between namespaces since its last backup is restored |
| 766 | under its old namespace until the next backup records the new one). |
| 767 | - The git store (`deploy/self-host/gitstore/server.mjs`) now takes namespaced keys |
| 768 | (`g1t-us-1/acme--rocket`, served at `/git/g1t-us-1/acme--rocket.git`, the shape Artifacts gives |
| 769 | remotes) beside plain ones, and `GITSTORE_READ_ONLY=1` refuses pushes, write tokens, and making, |
| 770 | forking or deleting repositories. |
| 771 | - `services/repos/src/fallback.rs` and `store.rs`: for each namespace in `GIT_FALLBACK_NAMESPACES` |
| 772 | (`*` for all), the `GitStore` sends every call it would make on the Artifacts binding (`create`, |
| 773 | `get`, `delete`, `info`, `createToken`, `log`, `readCommit`, `readTree`, `readBlob`, `readFile`, |
| 774 | `fork`) to the store's API at `GIT_FALLBACK_URL` with `GIT_FALLBACK_SECRET`, and git's smart HTTP to |
| 775 | its remotes. Calls keep their retries and breaker, counted as `<namespace>@fallback` in |
| 776 | `store_health`, never as Artifacts' own health, and are not metered as binding calls (git requests |
| 777 | still are). Credentials are kept apart from Artifacts' (`fallback:<key>`). Answers kept under a refs |
| 778 | version (ref listings, packs, logs and files by branch) are neither used nor kept, since the |
| 779 | fallback may be behind them; objects named by their hash are. |
| 780 | - Read-only, the default (`GIT_FALLBACK_WRITES` unset or `refuse`): a write is refused before it is |
| 781 | asked. Git hears 503 with `Retry-After: 300` and "g1t's git storage is read-only while it |
| 782 | recovers: clones, fetches and pages work, and pushes, merges and new repositories wait until it is |
| 783 | back."; the site's reads work, and its writes fail with the same words. New repositories are placed |
| 784 | in a namespace that still takes writes, if `ARTIFACTS_NEW_REPOS` has one. Backups are not cut from a |
| 785 | switched namespace (they would record an older state). status.g1t.sh shows **Git storage** |
| 786 | degraded: "Served from the backup store: reads work, pushes and merges wait". |
| 787 | |
| 788 | **What does not work while switched.** Working copies of open pull requests are not backed up, so |
| 789 | their changes and branches do not read; working copies already removed read from their repository's |
| 790 | `refs/pull/<id>/head` as usual. Anything pushed after the last backup is not there until Artifacts is |
| 791 | back. Agent runs that only read work (their sandboxes clone from the fallback through handed-out |
| 792 | credentials); runs that push wait. Mirror catch-ups wait; pushes out to mirrors work. |
| 793 | |
| 794 | #### The host |
| 795 | |
| 796 | | Need | Why | What | |
| 797 | | --- | --- | --- | |
| 798 | | Outside Cloudflare | It is the exit if Artifacts, or the account, is the problem | A VM with a provider of your choice | |
| 799 | | Persistent disk | Repositories and a copy of the bucket live there; Containers' disk is ephemeral and at most 20 GB | Block storage or a local SSD that survives reboots | |
| 800 | | Disk size | Restored repositories about equal the newest full bundles; the bucket's copy holds up to two chains | 2.5 times the bucket's size; today 100 GB is ample, at 3,000 workspaces (about 250 GB stored, section 5) 1 TB | |
| 801 | | Near the repos Worker | Every read crosses to it; `g1t-repos` runs near D1 in WNAM | US West, for example Oregon. The EU namespace's fallback on a second, EU host, so EU data stays in the EU | |
| 802 | | HTTPS on a public name | Workers reach it with `fetch` | Caddy in front of port 8080, a name such as `fallback-git.g1t.sh` | |
| 803 | | Software | | Docker (the git store's image), or Node 24 and git; `rclone` | |
| 804 | | CPU and memory | `git upload-pack` for clones, restores 4 at a time | 4 vCPU, 8 GB | |
| 805 | |
| 806 | **What it costs**, at list prices for such a host (check before buying): a 4 vCPU, 8 GB VM is about |
| 807 | $15 to $50 a month depending on the provider; block storage $0.04 to $0.10 per GB-month, so $4 to $10 |
| 808 | a month for 100 GB today and $40 to $100 for 1 TB at launch scale. Reading the bucket costs nothing in |
| 809 | egress (R2 charges none) and a few cents a month in R2 operations for a nightly `rclone sync`. In all, |
| 810 | about $20 to $60 a month now and $60 to $150 at 3,000 workspaces, per host; the EU host only once there |
| 811 | is an EU namespace. |
| 812 | |
| 813 | #### What you must create (R12) |
| 814 | |
| 815 | 1. The host above, with a DNS name and TLS. |
| 816 | 2. An R2 API token that can only read `g1t-backups` (Cloudflare dashboard, R2, Manage API tokens: |
| 817 | Object Read, that bucket), for `rclone` on the host: |
| 818 | |
| 819 | ```ini |
| 820 | # ~/.config/rclone/rclone.conf on the host |
| 821 | [r2] |
| 822 | type = s3 |
| 823 | provider = Cloudflare |
| 824 | access_key_id = <the token's access key id> |
| 825 | secret_access_key = <its secret> |
| 826 | endpoint = https://1e6f2cffa3f445920836e8ebe446bb58.r2.cloudflarestorage.com |
| 827 | ``` |
| 828 | |
| 829 | 3. The git store and its secret, read-only from the start: |
| 830 | |
| 831 | ```sh |
| 832 | git clone https://g1t.sh/flagon-io/g1t.git /opt/g1t # node, git and rclone installed |
| 833 | openssl rand -hex 32 > /srv/gitstore.secret |
| 834 | GITSTORE_ROOT=/srv/gitstore GITSTORE_PORT=8080 GITSTORE_URL=https://fallback-git.g1t.sh \ |
| 835 | GITSTORE_SECRET="$(cat /srv/gitstore.secret)" GITSTORE_READ_ONLY=1 \ |
| 836 | node /opt/g1t/deploy/self-host/gitstore/server.mjs # as a systemd unit, or the image in deploy/self-host/gitstore |
| 837 | # Caddyfile: fallback-git.g1t.sh { reverse_proxy 127.0.0.1:8080 } |
| 838 | ``` |
| 839 | |
| 840 | 4. The first restore, then every night after the backups (02:53 UTC) have run, say at 07:00 UTC: |
| 841 | |
| 842 | ```sh |
| 843 | rclone sync r2:g1t-backups /srv/backups |
| 844 | node /opt/g1t/scripts/ops/restore-to-gitstore.mjs restore --into /srv/gitstore --bundles /srv/backups --offline --jobs 4 |
| 845 | ``` |
| 846 | |
| 847 | 5. Tell `g1t-repos` where it is, ahead of time. These change nothing until a namespace is named: |
| 848 | |
| 849 | ```sh |
| 850 | cd services/repos |
| 851 | echo https://fallback-git.g1t.sh | npx wrangler secret put GIT_FALLBACK_URL |
| 852 | npx wrangler secret put GIT_FALLBACK_SECRET # paste /srv/gitstore.secret |
| 853 | ``` |
| 854 | |
| 855 | 6. Drill it once a quarter on a namespace that holds only test repositories (make `g1t-drill` as in |
| 856 | R7, bind it, move a test repository there), with the runbook below. |
| 857 | |
| 858 | #### Runbook: an Artifacts outage |
| 859 | |
| 860 | **Detect.** |
| 861 | |
| 862 | 1. status.g1t.sh shows **Git storage** down, or `node scripts/ops/artifacts-namespaces.mjs` shows a |
| 863 | namespace failing (errors, `rejected` calls from an open breaker). `npx wrangler tail g1t-repos` |
| 864 | shows `git store <namespace>: ... failed`. |
| 865 | 2. Check Cloudflare's status page and the Artifacts metrics (`serverError`, `rateLimited`). |
| 866 | 3. Switch when it has lasted 15 minutes with no sign of ending, or at once if Cloudflare says it will |
| 867 | be long. A short blip needs nothing: retries and the breaker already answer git with 503 and |
| 868 | `Retry-After`. |
| 869 | |
| 870 | **Switch.** |
| 871 | |
| 872 | 1. On the host: `curl -s https://fallback-git.g1t.sh/healthz` answers `ok read-only`. If the nightly |
| 873 | restore did not run today, run step 4 of the setup; a run over a restored store only does what |
| 874 | changed. |
| 875 | 2. Switch the failing namespace (or `*`). A secret takes effect in seconds, with no build: |
| 876 | |
| 877 | ```sh |
| 878 | cd services/repos |
| 879 | echo g1t | npx wrangler secret put GIT_FALLBACK_NAMESPACES |
| 880 | ``` |
| 881 | |
| 882 | 3. Check: `git ls-remote https://g1t.sh/flagon-io/hello.git` answers; a repository page loads; |
| 883 | status.g1t.sh shows Git storage degraded; `artifacts-namespaces.mjs` lists `@fallback` calls. |
| 884 | 4. Open an incident on status.g1t.sh: reads work from last night's backup; pushes, merges and new |
| 885 | repositories wait. |
| 886 | |
| 887 | **Serve reads.** Nothing more to do. Watch the host's disk and load; `upload-pack` is the work. |
| 888 | |
| 889 | **Take writes, only if the outage will be long.** Everything pushed then must be sent back |
| 890 | afterwards, and anything pushed to Artifacts after the last backup will conflict with it. |
| 891 | |
| 892 | ```sh |
| 893 | # On the host: restart the git store without GITSTORE_READ_ONLY. Then: |
| 894 | echo allow | npx wrangler secret put GIT_FALLBACK_WRITES |
| 895 | ``` |
| 896 | |
| 897 | **Switch back**, once Artifacts answers again (`artifacts-namespaces.mjs` shows no errors from it; |
| 898 | the namespace's own calls are none while switched, so check Cloudflare's status and the metrics): |
| 899 | |
| 900 | 1. If writes were taken: stop them first (`echo refuse | npx wrangler secret put GIT_FALLBACK_WRITES`, |
| 901 | and restart the git store with `GITSTORE_READ_ONLY=1`), then list and send back what came in, while |
| 902 | the namespace is still switched, so nothing else writes to Artifacts meanwhile: |
| 903 | |
| 904 | ```sh |
| 905 | node scripts/ops/restore-to-gitstore.mjs changed --into /srv/gitstore |
| 906 | CLOUDFLARE_API_TOKEN=<Artifacts edit, D1 edit> node scripts/ops/restore-to-gitstore.mjs reconcile --into /srv/gitstore |
| 907 | ``` |
| 908 | |
| 909 | A ref Artifacts still has as it was backed up takes the fallback's value (leased on that value, so |
| 910 | nothing newer is overwritten). One that moved on both sides keeps Artifacts' value, and the |
| 911 | fallback's goes beside it as `refs/fallback/<rest of the name>` (exit 3 says some did): tell the |
| 912 | repository's owners to merge it. Each reconciled repository's `refs_version` is moved. To push, |
| 913 | `reconcile` mints a write token with Cloudflare's REST API |
| 914 | (`POST /accounts/<id>/artifacts/namespaces/<ns>/repos/<name>/tokens`, taken to mirror the binding's |
| 915 | `createToken`); that endpoint is not verified yet, so run `reconcile` on one repository in the |
| 916 | drill before relying on it. |
| 917 | 2. Switch back: `npx wrangler secret delete GIT_FALLBACK_NAMESPACES` (and `GIT_FALLBACK_WRITES`). |
| 918 | 3. Check as in "Switch", step 3: Git storage is no longer degraded once the fallback's calls age out |
| 919 | of the five-minute window. |
| 920 | 4. Backups resume the next night. Close the incident. |
| 921 | |
| 922 | Tested by `npm run test:ops` (`scripts/ops/restore-to-gitstore.test.mjs`): bundles cut as the runner |
| 923 | cuts them are restored into a store's root, served read-only by the git store itself (a namespaced |
| 924 | remote, a read token, a clone; write tokens and new repositories refused), pushed to, listed by |
| 925 | `changed`, reconciled into a live copy, and reconciled again after the live copy moved too; and a |
| 926 | restore through the store's API. `cargo test` covers the routing, answers, read-only refusals and |
| 927 | kept credentials (`fallback.rs`, `store.rs`, `resilience.rs`). Not yet run against production: the |
| 928 | switch itself, which wants the host. |
| 929 | |
| 930 | ### Deploy order and what to watch |
| 931 | |
| 932 | R7 and R12 (2026-10-07): migrations `repos/0014` and `identity/0026` first (the registry reads |
| 933 | `writes_paused_until`, and `claim_store_key` reads `repo_move_copies`, so the new repos code must not |
| 934 | run before 0014); then `g1t-repos`, `g1t-identity`, `g1t-web` and `g1t-status`. No new bindings or |
| 935 | variables: with today's configuration nothing is placed, moved, offered or switched. Watch that |
| 936 | repository creation still answers as fast (it reads nothing new), and that `repository_moves` stays |
| 937 | empty. |
| 938 | |
| 939 | For 2026-10-06's work: |
| 940 | |
| 941 | 1. Migration 0011 (the deploy tool applies migrations first). `forks_of` reads `retired_at`, so |
| 942 | the new code must not run before it. |
| 943 | 2. `g1t-repos` (new vars in `wrangler.jsonc`; no new bindings). |
| 944 | 3. `g1t-status` (a new `REPOS` service binding; **Git storage** appears once deployed). |
| 945 | 4. After a day: `scripts/ops/artifacts-usage.mjs`; then the fork test. |
| 946 | |
| 947 | Expected: `Server-Timing` `kept` on POSTs drops from 63–98 ms to the KV and Cache lookups only (a |
| 948 | few ms); `mint` happens about once per repository and scope every 50 minutes per isolate instead |
| 949 | of every 3 minutes, and costs `get` and `createToken` (about 0.5 s instead of 0.8 s) once an |
| 950 | isolate knows its namespace's prefix; branch pages and repeated tree and file views skip |
| 951 | Artifacts while the refs version holds; a burst of 100 mergeability checks walks the target once. |
| 952 | |
| 953 | Risks: what Cloudflare bills is still theirs to confirm (the mapping makes changing it cheap). |
| 954 | Counts held by an isolate that is evicted before its `wait_until` write are lost (seconds of |
| 955 | traffic). Pushes over 24 MiB now fail closed: bringing a large existing repository needs the |
| 956 | push-in-parts command (in `guides/git.md`). Moving a revived fork's branch back to its old head |
| 957 | is a non-fast-forward update, which depends on Artifacts accepting it as git does by default; if |
| 958 | it refuses, the fork of a closed pull request cannot be made again and the error says so. |
| 959 | `refs/pull/*` refs appear in full ref advertisements (mirrors, `--mirror` clones). Operations on |
| 960 | pull request forks are metered under the workspace `pulls`, so they are counted for g1t's bill |
| 961 | but not charged to a workspace (except the fork itself, metered on its repository). |