| 1 | # Cloudflare Artifacts: due diligence for g1t at launch scale |
| 2 | |
| 3 | Status: research document, 2026-10-06; R1–R5, R9, R10, R13 and R7 groundwork were built the same day (section 9). |
| 4 | Scope: everything g1t stores in Cloudflare Artifacts (open beta since 2026-10-01; billing from 2026-10-14), |
| 5 | measured against what Cloudflare documents, and what we must build so that a few thousand workspaces |
| 6 | can run on it. |
| 7 | |
| 8 | ## Summary |
| 9 | |
| 10 | Artifacts fits what g1t does: one Durable Object per repository, smart HTTP for clones and pushes, a |
| 11 | read-only binding for commits, trees and blobs, forks and short-lived tokens. Nothing we rely on is |
| 12 | missing outright. Five things are not yet safe at a few thousand workspaces. |
| 13 | |
| 14 | 1. **We do not know what an "operation" is.** Pricing says "repo operations, such as `create`, `push`, |
| 15 | `pull`, and `clone`". Metrics list only `create`, `fork`, `push`, `pull`, `delete`. If binding reads |
| 16 | (`get`, `info`, `createToken`, `log`, `readTree`, `readBlob`, `readCommit`, `readFile`) and every git |
| 17 | HTTP request also count, our bill at launch scale is roughly **15× larger** (about $31k a month instead |
| 18 | of about $1.8k). We must settle this before 2026-10-14. |
| 19 | 2. **Pull request forks are never deleted.** Every pull request is an Artifacts fork (`pulls--<id>`). They |
| 20 | are purged only with their parent repository. Cloudflare does not say whether a fork shares objects |
| 21 | with its source. If forks copy objects, an agent-heavy workload reaches the **1 TB account storage |
| 22 | limit in days**, and every push then fails. If they share, storage still grows without bound. |
| 23 | 3. **One namespace carries everything.** All repositories and forks live in the `g1t` namespace. The |
| 24 | control-plane limit is **2,000 requests per 10 seconds per namespace** (200 per second). If binding |
| 25 | calls count against it, page views, token mints and mergeability checks together exceed it at launch |
| 26 | peaks. |
| 27 | 4. **Hard limits are not enforced in front of Artifacts.** 1 GB per repository, 32 MB per file, and a |
| 28 | 128 MB Worker isolate that buffers each push body twice. Large pushes and imports fail late, without a |
| 29 | message git can show. |
| 30 | 5. **No backup, no exit drill.** Cloudflare replicates data, but there is no SLA, no documented export |
| 31 | besides git itself, and the self-host git store is not a production fallback yet. Nightly bundles |
| 32 | to R2 and a restore drill are now built (R11, section 9); the fallback store is not (R12). |
| 33 | |
| 34 | None of these blocks an invite-only launch. Items 1 and 2 must be answered before billing starts on |
| 35 | 2026-10-14, and the fork cleanup must ship before agent pull requests reach thousands a day. |
| 36 | |
| 37 | ## 1. What Cloudflare documents |
| 38 | |
| 39 | All quotes are from developers.cloudflare.com, retrieved 2026-10-06. |
| 40 | |
| 41 | ### Limits ([Artifacts limits](https://developers.cloudflare.com/artifacts/platform/limits/), updated 2026-10-01) |
| 42 | |
| 43 | | Limit | Value | |
| 44 | | --- | --- | |
| 45 | | Control-plane request rate | 2,000 requests per 10 seconds **per namespace** | |
| 46 | | Git request rate | 2,000 requests per 10 seconds **per repository** | |
| 47 | | Storage per repository | **1 GB** | |
| 48 | | Largest file or blob | **32 MB** | |
| 49 | | Storage per account | **1 TB** (can be raised on request) | |
| 50 | | Repositories, namespaces | Unlimited | |
| 51 | | Names | 2 to 63 characters; letters, digits, `.`, `_`, `-`; start with a letter or digit | |
| 52 | |
| 53 | ### Pricing ([Artifacts pricing](https://developers.cloudflare.com/artifacts/platform/pricing/)) |
| 54 | |
| 55 | - Workers Paid only. "Cloudflare will begin billing for Artifacts operations and storage on October 14, 2026." |
| 56 | - Operations: first 10,000 a month, then **$0.15 per 1,000**. Defined only as "the number of repo |
| 57 | operations, such as `create`, `push`, `pull`, and `clone`." |
| 58 | - Storage: first 1 GB, then **$0.50 per GB-month**, "calculated by averaging peak storage per day over a |
| 59 | 30-day billing period". "Replicas do not add storage charges." "Repos remain stored until you |
| 60 | explicitly delete them." |
| 61 | |
| 62 | ### Architecture ([announcement blog](https://blog.cloudflare.com/artifacts-git-for-agents-beta/), 2026-04-16, Matt Carey and Matt Silverlock) |
| 63 | |
| 64 | - Each repository is a Durable Object; the git server is Zig compiled to WebAssembly (about 100 KB). |
| 65 | - "Files are stored in the underlying Durable Object's SQLite database." "Durable Object storage has a |
| 66 | 2MB max row size, so large Git objects are chunked and stored across multiple rows." |
| 67 | - "DOs have ~128MB memory limits." Fetch and push stream (`ReadableStream<Uint8Array>`). |
| 68 | - "Artifacts also uses R2 (for snapshots) and KV (for tracking auth tokens)." |
| 69 | - Deltas are stored as received; "if the requesting client already has the base object, Zig emits the |
| 70 | delta instead." |
| 71 | - Durability ([How Artifacts works](https://developers.cloudflare.com/artifacts/concepts/how-artifacts-works/)): |
| 72 | "Cloudflare replicates repo data synchronously across multiple data centers and copies it |
| 73 | asynchronously to object storage and snapshots." |
| 74 | - A repository is "a single logical instance that Cloudflare can route to from any region", like a |
| 75 | Durable Object. Durable Objects have a soft limit of 1,000 requests per second each |
| 76 | ([DO limits](https://developers.cloudflare.com/durable-objects/platform/limits/)). |
| 77 | |
| 78 | ### Git protocol ([Git protocol](https://developers.cloudflare.com/artifacts/api/git-protocol/)) |
| 79 | |
| 80 | | Feature | Documented support | |
| 81 | | --- | --- | |
| 82 | | Clone and fetch | Protocol v1 and v2 (`ls-refs`, `fetch`); v1 shallow and deepen | |
| 83 | | Push | v1 receive-pack only; v2 receive-pack not supported | |
| 84 | | `filter` (partial clone), `include-tag` | "not supported" for v1 | |
| 85 | | Tokens | `art_v1_<40 hex>?expires=<unix seconds>`; scopes `read`, `write`; TTL 60 s to 1 year, default 24 h | |
| 86 | |
| 87 | **Measured differently:** a protocol v2 `git clone --filter=blob:none` of `flagon-io/g1t` through g1t |
| 88 | returned a real partial clone (2,789 objects, 903 KB) instead of the full 6,187 objects and 5.5 MB. So |
| 89 | v2 filtering works today, despite the table. Ask Cloudflare whether this is supported or accidental. |
| 90 | |
| 91 | ### Binding API ([Workers binding](https://developers.cloudflare.com/artifacts/api/workers-binding/); generated types in `services/*/worker-configuration.d.ts`) |
| 92 | |
| 93 | - Namespace: `create`, `get`, `list`, `import`, `delete`. `get()` is a lookup that throws `NOT_FOUND` or |
| 94 | `*_IN_PROGRESS`, so it costs a round trip. |
| 95 | - Repository handle: `info()` ("Each call performs a fresh lookup"), `createToken`, `listTokens`, |
| 96 | `revokeToken`, `log` (first-parent only, at most 1,000), `readCommit`, `readTree` (one level), |
| 97 | `readBlob`, `readFile`, `fork` (`defaultBranchOnly` defaults to true). |
| 98 | - Not available: listing refs, updating refs, writing objects, repository size, gc or repack, hooks. |
| 99 | g1t works around the first three over smart HTTP (`refs.rs`, `land.rs`, `catch_up.rs`). |
| 100 | - Errors: `MEMORY_LIMIT` (10402) "if the object cannot be buffered safely", `INTERNAL_ERROR` (10400), |
| 101 | `UPSTREAM_UNAVAILABLE`, `IMPORT_IN_PROGRESS`, `FORK_IN_PROGRESS`, `CREATE_IN_PROGRESS` |
| 102 | ([Errors](https://developers.cloudflare.com/artifacts/api/errors/)). |
| 103 | - Doc drift: the [isomorphic-git example](https://developers.cloudflare.com/artifacts/examples/isomorphic-git/) |
| 104 | still says the binding "cannot read or write files inside them". It can read; it cannot write. |
| 105 | |
| 106 | ### Observability, events, data location |
| 107 | |
| 108 | - Metrics: GraphQL dataset `artifactsEventsAdaptiveGroups`, 31 days, with `eventType` in `create`, |
| 109 | `fork`, `push`, `pull`, `delete`, and errors `storageLimitReached`, `serverError`, `clientError`, |
| 110 | `rateLimited` ([Metrics](https://developers.cloudflare.com/artifacts/observability/metrics/)). |
| 111 | **This is the fastest way to learn what Cloudflare counts.** |
| 112 | - Events: account level `repo.created|deleted|forked|imported`; per-repository `pushed`, `cloned`, |
| 113 | `fetched`, `token.created|revoked`, which need one subscription per repository |
| 114 | ([Event subscriptions](https://developers.cloudflare.com/artifacts/guides/event-subscriptions/)). |
| 115 | g1t reports pushes itself for that reason (`record_push`). |
| 116 | - Jurisdictions `eu` and `us`, chosen when a namespace is created and never changeable |
| 117 | ([Data localization](https://developers.cloudflare.com/artifacts/guides/data-localization/)). A binding |
| 118 | names one namespace, so EU residency needs a second binding. |
| 119 | - Best practices: "Do not keep every repo in one default namespace once usage grows" and "When one |
| 120 | namespace becomes hot, shard new repos into additional namespaces" |
| 121 | ([Best practices](https://developers.cloudflare.com/artifacts/concepts/best-practices/)). |
| 122 | - Beta: the docs state no SLA, no support tier, and no backup or export feature beyond git itself. |
| 123 | |
| 124 | ### Platform limits that bound us |
| 125 | |
| 126 | | Limit | Value | Source | |
| 127 | | --- | --- | --- | |
| 128 | | Worker memory | 128 MB per isolate, shared by concurrent requests | [Workers limits](https://developers.cloudflare.com/workers/platform/limits/) | |
| 129 | | Request body | 100 MB on Free and Pro zones, 200 MB Business, up to 5 GB Enterprise | same | |
| 130 | | Response body | No limit (CDN cache 512 MB) | same | |
| 131 | | CPU | 30 s default, up to 5 min | same | |
| 132 | | Subrequests | 10,000 per invocation (Paid); 6 connections waiting for headers at once | same | |
| 133 | | Cache API | Per data center only; calls share the subrequest quota | [Cache](https://developers.cloudflare.com/workers/runtime-apis/cache/) | |
| 134 | | KV | 1 write per second per key; 25 MiB values; 60 s minimum TTL | [KV limits](https://developers.cloudflare.com/kv/platform/limits/) | |
| 135 | | D1 | 10 GB per database; 30 s per query; one writer | [D1 limits](https://developers.cloudflare.com/d1/platform/limits/) | |
| 136 | | Cloudflare REST API | 1,200 requests per 5 minutes per user | [API rate limits](https://developers.cloudflare.com/fundamentals/api/reference/limits/) | |
| 137 | | Containers | Up to 4 vCPU, 12 GiB, 20 GB disk (ephemeral) | [Containers limits](https://developers.cloudflare.com/containers/platform-details/limits/) | |
| 138 | | Smart Placement | Applies to fetch handlers, not RPC entrypoints | [Placement](https://developers.cloudflare.com/workers/configuration/placement/) | |
| 139 | |
| 140 | The REST API's 1,200 requests per 5 minutes means the hot path must use the binding, never the REST API. |
| 141 | |
| 142 | ## 2. How g1t uses Artifacts |
| 143 | |
| 144 | All Artifacts traffic goes through `services/repos` (`g1t-repos`), binding `ARTIFACTS`, namespace `g1t`, |
| 145 | placement off. Store keys are `<workspace>--<repo>`; pull request forks are `pulls--<pull id>`. |
| 146 | |
| 147 | ### Calls per user action |
| 148 | |
| 149 | "Binding" counts calls on `env.ARTIFACTS`; "git" counts HTTP requests to the repository's remote. |
| 150 | |
| 151 | | Action | Path in code | Binding calls | Git requests to Artifacts | |
| 152 | | --- | --- | --- | --- | |
| 153 | | Credential (mint) | `store.rs` `mint_access` | 3: `get`, then `info` and `createToken` at once | 0 | |
| 154 | | Clone, protocol v2 | `lib.rs` `answer_git` → `git_http::forward` | mint if none kept (3) | `info/refs` and `ls-refs` (0 on a `refs_cache` hit), `fetch` 1 | |
| 155 | | Clone, protocol v0 | same | mint if none kept | `info/refs` (0 on hit), `upload-pack` 1 | |
| 156 | | Fetch, nothing new | same | as above | `ls-refs` (0 on hit), `fetch` 1 | |
| 157 | | Push | same, then `record_push` | mint; then `get` + `log(branch, 1)` per pushed branch | `info/refs` 1, `receive-pack` 1 | |
| 158 | | Tree page | `Repos::tree` | `get`, `log(ref, 1)`, `readTree` per path segment (Cache API), `readBlob` README (cached) | 0 | |
| 159 | | File page | `Repos::blob` | `get`, `readFile` (not cached) | 0 | |
| 160 | | Branches list | `GitRepo::branches` (`refs.rs`) | mint (3) | `info/refs` 1 (does not use `refs_cache`) | |
| 161 | | Blame | `blame.rs` | `log(ref, 400)`, then trees and blobs per commit | 0 (one view took 6.6 s wall in the tail) | |
| 162 | | Open a pull request | `fork_for_pull` | `get`, `fork` | 0 | |
| 163 | | Mergeability (each time the target branch moves, for up to 100 open pull requests) | `divergence` | `get` ×2, `log(…, 1000)` ×2, `readCommit` per commit off the first-parent chain, `readTree` per changed directory | 0 | |
| 164 | | Catch-up (no conflicts) | `catch_up.rs` | `get` ×2, `log` ×2, trees | `upload-pack` 1, `receive-pack` 1 | |
| 165 | | Land a pull request | `land.rs` | 2 mints (6) | `upload-pack` 1 from the fork, `receive-pack` 1 to the target | |
| 166 | | Commit one file | `commit_file.rs` | mint, `log` | `info/refs` 1, `receive-pack` 1 | |
| 167 | | Mirror sync or import | `mirror.rs`, `import.rs` | mint | `info/refs` 1–2, `upload-pack` and `receive-pack` 1 each; packs capped at 40 MB | |
| 168 | | Search indexing | `listing.rs` | trees by level, blobs in groups | 0 | |
| 169 | | Push protection | `secret_scan.rs` | trees and blobs for bases, up to 24 MB scanned | 0 | |
| 170 | | Delete (purge) | `lifecycle.rs` | `delete` per key, including forks | 0 | |
| 171 | |
| 172 | Every sandbox job clones in full (`crates/runner/src/{main,checks,review,plan,queue,reply,update,deploy,mergecheck}.rs`: |
| 173 | `git clone --quiet`, no depth, no filter), and most then fetch an upstream branch. Only Actions' |
| 174 | checkout fetches with `--depth=1` (`crates/runner/src/actions/uses.rs`). |
| 175 | |
| 176 | ### What already saves calls |
| 177 | |
| 178 | | Mitigation | Where | What it saves | |
| 179 | | --- | --- | --- | |
| 180 | | Credential cache | `store.rs` `Credentials` (per isolate) and `shared.rs` (KV, sealed), TTL 300 s, reused 180 s | 3 binding calls and about 800 ms per git request on a hit | |
| 181 | | Ref listing cache | `refs_cache.rs`, keyed by `refs_version`, 60 s TTL, colo Cache API then KV | the `info/refs` and `ls-refs` round trip (330–400 ms) on a hit; measured hits answer in 3–6 ms | |
| 182 | | Object cache | `store.rs` `cached`/`keep`, Cache API, immutable | `readTree`, `readBlob` (≤ 1 MB), and `log` by commit hash | |
| 183 | | Recent row cache | `registry.by_path_recent` | D1, not Artifacts | |
| 184 | | Soft delete | `lifecycle.rs` | an accidental `delete` cannot lose data for `RESTORE_DAYS` | |
| 185 | |
| 186 | Gaps in those caches: the Cache API is per data center, so each colo pays its own misses; `log` by branch |
| 187 | name, `readFile`, `branches()` and `readCommit` are never cached; the credential reuse window is 180 s, |
| 188 | so a busy repository mints about 20 times an hour per scope. |
| 189 | |
| 190 | ## 3. Measured in production |
| 191 | |
| 192 | Read-only, from a client in Denver (`CF-Ray …-DEN`), 2026-10-06 07:40 UTC, git 2.45 user agent. |
| 193 | `flagon-io/hello`: 239 objects, 44 KB pack. `flagon-io/g1t`: 6,187 objects, 5.5 MB pack. |
| 194 | |
| 195 | | Measurement | hello | g1t | |
| 196 | | --- | --- | --- | |
| 197 | | `info/refs`, cold: credential minted | n/a | 1,306–1,425 ms total: `mint` 807–856 ms, `store` 334–399 ms | |
| 198 | | `info/refs`, `refs_cache` hit | 3–6 ms server, 80–150 ms client | 3–6 ms server | |
| 199 | | `ls-refs` / v2 advertisement miss with kept credential | 436 ms (`store` 349) | 458 ms (`store` 332) | |
| 200 | | Full upload-pack (replayed v0 want, kept credential) | 674–1,104 ms, `store` 607–997 | 1,960–2,223 ms to first byte, `store` 1,855–2,156; 3.2 s total | |
| 201 | | `git clone --bare`, first / repeats | 5.1 s / 2.0–2.4 s | 5.4 s / 4.2 s (4.8–5.8 s during the tail) | |
| 202 | | `git fetch`, nothing new | 0.99 s first, then 0.39–0.42 s | 0.95 s first, then 0.35–0.41 s | |
| 203 | | `clone --depth=1` | 1.6 s | 3.8 s | |
| 204 | | `clone --filter=blob:none` | 1.4 s | 3.4 s, 903 KB | |
| 205 | | `kept` step on every POST | 63–98 ms | same | |
| 206 | |
| 207 | Observations: |
| 208 | |
| 209 | - Artifacts builds the whole pack before the first byte: about 2 s for 5.5 MB. Every full clone pays |
| 210 | this, every time, because nothing caches packs. |
| 211 | - Minting a credential costs about 0.8 s, almost all in `get` then `info` and `createToken`. |
| 212 | - The 63–98 ms `kept` step on every POST is not Artifacts. It is `git_limits` doing a D1 upsert into |
| 213 | `git_operations` before the request is forwarded. |
| 214 | - `wrangler tail g1t-repos` for 75 s: 87 events, all `ok`, no exceptions or error logs. The slowest were a |
| 215 | blame RPC (6.6 s wall, 178 ms CPU) and an `info/refs` miss with a mint (2.6 s). |
| 216 | |
| 217 | ## 4. Where we and the docs disagree |
| 218 | |
| 219 | | # | Topic | Documented | What g1t does or assumes | Risk | Fix | |
| 220 | | --- | --- | --- | --- | --- | --- | |
| 221 | | M1 | What an operation is | "create, push, pull, clone"; metrics events `create`, `fork`, `push`, `pull`, `delete` | `git_ops.rs` bills workspaces per upload-pack or receive-pack POST, including `ls-refs` answered from our cache; binding reads assumed free | Cost could be about 15× the plan; customers billed for operations that never reach Artifacts | R1 | |
| 222 | | M2 | Fork storage | Not documented; fork returns `objects: N` | `store.rs` calls forks "copy-on-write"; forks never deleted until the parent is purged | 1 TB account limit; storage grows forever | R2 | |
| 223 | | M3 | Namespace rate | 2,000 per 10 s per namespace | One namespace for all repositories and forks | 429s across all of g1t at peak | R7 | |
| 224 | | M4 | Repository size | 1 GB | Free private storage is 1 GB per workspace; no per-repository check; imports capped at 40 MB | Late `storageLimitReached` failures with no explanation | R4 | |
| 225 | | M5 | File size | 32 MB | No check before forwarding | Push fails inside Artifacts | R4 | |
| 226 | | M6 | Push bodies | Workers 128 MB per isolate; 100 MB body cap on our zone plan | `git_http::forward` reads the body (`request.bytes()`) and copies it into a `Uint8Array`; push protection also reads it | Pushes above roughly 40–50 MB can exceed isolate memory, taking concurrent requests with them | R4 | |
| 227 | | M7 | Landing and mirrors | Fetch streams | `land::fetch_pack` buffers the pack, `unpack_sideband` copies it, `push_pack` copies again | Large pull requests can fail to land | R4 | |
| 228 | | M8 | Token lifetime | 60 s to 1 year | 300 s, reused 180 s | Mint load and 0.8 s latency on each miss | R3 | |
| 229 | | M9 | `info()` | "Each call performs a fresh lookup" | Called on every mint only to read `remote` | One wasted call per mint; `remote` is fixed per key | R3 | |
| 230 | | M10 | Partial clone | v1 `filter` unsupported | v2 `blob:none` works in production | Building on undocumented behaviour | Q3 | |
| 231 | | M11 | Read-after-write | Not documented | `record_push` reads `log(branch, 1)` right after the push response; `refs_moved` assumes refs are final when the response ends | Missed push events if a read lags | Q5 | |
| 232 | | M12 | Errors | Typed `ArtifactsError`, `rateLimited` events | `ALREADY_EXISTS`/`NOT_FOUND` tolerated; everything else returns 500; no retry, no breaker | Brief Artifacts errors become user-visible failures | R5 | |
| 233 | | M13 | Durability | Synchronous replication, asynchronous snapshots; no SLA; no export | No copy outside Artifacts | Beta incident or account issue with no recovery path | R11 | |
| 234 | | M14 | Data location | Jurisdiction per namespace, fixed | `PLAN.md` promises residency per workspace; only `g1t` exists | EU customers cannot be offered residency | R7 | |
| 235 | | M15 | Direct credentials | Tokens are bearer, repo-scoped | `git_access` hands out raw write tokens; pushes with them skip branch protection and push protection (`refs_open` only stops caching) | Policy bypass if a token leaks out of a sandbox | Keep TTL minimal on this path | |
| 236 | | M16 | Hot repository | 2,000 git requests per 10 s per repository; DO soft limit 1,000 req/s | Agents clone the same repository many times per pull request | Not near the limit today; a monorepo with many agents could be | R6 | |
| 237 | |
| 238 | ## 5. Capacity model: 3,000 workspaces |
| 239 | |
| 240 | ### Assumptions |
| 241 | |
| 242 | - 3,000 workspaces, 10,000 repositories, 6,000 active people, average repository 25 MB stored. |
| 243 | - People: 6 fetches and 3 pushes per person per day. |
| 244 | - Agent pull requests: 15,000 a day (5 per workspace). Human pull requests: 3,000 a day. |
| 245 | - Per agent pull request: 1 fork; 1.5 agent runs × (clone + upstream fetch + push); 1.5 check clones; |
| 246 | review clone + fetch; mergecheck in 30% of cases (clone + fetch); 0.5 catch-up push; merge queue clone + |
| 247 | fetch; landing upload-pack + receive-pack; 0.3 deploy clones. About **14 git operations**. |
| 248 | - Actions: 1.5 checkouts per push. Mirrors: 1,000 repositories × 24 syncs × 2. |
| 249 | - Web: 60 page views per person per day, about 5 uncached binding calls each. |
| 250 | - Mergeability: 18,000 default-branch moves a day × 8 open pull requests × about 20 binding calls. |
| 251 | |
| 252 | ### Operations per day |
| 253 | |
| 254 | | Source | Scenario A: git data operations only | Scenario B: every Artifacts call | |
| 255 | | --- | ---: | ---: | |
| 256 | | People: fetch and push | 54,000 | 54,000 | |
| 257 | | Agent pull requests (15,000 × 14) | 210,000 | 210,000 | |
| 258 | | Human pull requests (3,000 × 8) | 24,000 | 24,000 | |
| 259 | | Actions checkouts | 60,000 | 60,000 | |
| 260 | | Mirrors | 48,000 | 48,000 | |
| 261 | | Credential mints (≈ 200,000 × 3) | 0 | 600,000 | |
| 262 | | Ref listing misses (`info/refs`, `ls-refs`) | 0 | 300,000 | |
| 263 | | Web pages (360,000 × 5) | 0 | 1,800,000 | |
| 264 | | Mergeability (18,000 × 8 × 20) | 0 | 2,900,000 | |
| 265 | | Search indexing, push protection, `record_push` | 0 | 900,000 | |
| 266 | | **Total per day** | **≈ 400,000** | **≈ 6,900,000** | |
| 267 | | **Per month** | **≈ 12 M** | **≈ 207 M** | |
| 268 | | **Cost per month at $0.15 / 1,000** | **≈ $1,800** | **≈ $31,000** | |
| 269 | |
| 270 | Rates: scenario A averages 5 operations per second. Scenario B averages 80 calls per second and peaks |
| 271 | around 250–400, above the 200 per second a single namespace allows, if binding calls count toward it. |
| 272 | |
| 273 | Hot spots, largest first in scenario B: mergeability fan-out (42%), web reads (26%), credential mints |
| 274 | (9%), indexing and scanning (13%), sandbox clones (6%, but they carry almost all the bytes and the |
| 275 | latency). |
| 276 | |
| 277 | ### Storage |
| 278 | |
| 279 | | Item | Estimate | Cost per month | |
| 280 | | --- | --- | ---: | |
| 281 | | 10,000 repositories × 25 MB | 250 GB | ≈ $125 | |
| 282 | | Pull request forks if shared (about 0.5 MB of new objects each) | +270 GB every month, never deleted | +$135, growing each month | |
| 283 | | Pull request forks if copied (25 MB each, 540,000 a month) | +13.5 TB a month | ≈ $6,750 the first month, and the 1 TB account limit is reached in about 2 days | |
| 284 | |
| 285 | ### Free tier exposure |
| 286 | |
| 287 | `GIT_OPERATIONS_FREE_CAP` is 50,000 operations per workspace per month, never charged. At $0.15 per 1,000 |
| 288 | that is at most $7.50 per free workspace, or $22,500 a month if all 3,000 used it. Realistic use is far |
| 289 | lower, but the cap should be sized to what Cloudflare actually counts (R1). |
| 290 | |
| 291 | ## 6. Recommendations, ranked by risk × effort |
| 292 | |
| 293 | P0 means before 2026-10-14 or before agent volume ramps. Effort assumes one engineer. |
| 294 | |
| 295 | | # | Priority | What to build | Where | Expected effect | Effort | |
| 296 | | --- | --- | --- | --- | --- | --- | |
| 297 | | R1 | P0 | Find out what counts. Query `artifactsEventsAdaptiveGroups` grouped by `eventType` for the last 31 days and compare with `git_operations`; ask Cloudflare in writing. Then make `git_ops::count` count what Cloudflare counts (skip `ls-refs` and cache hits if they are free). | `services/repos/src/git_ops.rs`, a script under `scripts/` | Removes the 15× uncertainty; billing matches cost | 0.5 day plus Cloudflare's answer | |
| 298 | | R2 | P0 | Delete pull request forks after merge or close plus a grace period (for example 14 days), and verify fork storage semantics with a 100 MB test repository and the storage metric. | `lifecycle.rs` sweep (`23 * * * *`), `work` events `pull.merged`/`pull.closed` | Storage proportional to open work instead of all history; removes the 1 TB cliff | 1–2 days | |
| 299 | | R3 | P0 | Mint less and faster: TTL 3,600 s with a 50 min reuse window for credentials that never leave the service; keep 300 s for `git_access`. Derive `remote` from the key (`https://<account>.artifacts.cloudflare.net/git/g1t/<key>.git`) or keep it in the registry `store` column instead of calling `info()`. Retry a 401 once with a fresh token (already done for GET). | `store.rs` `TOKEN_TTL_SECONDS`, `TOKEN_REUSE_MS`, `ArtifactsRepo::access` | Mints about 15× fewer; cold git requests about 0.5 s faster | 0.5 day | |
| 300 | | R4 | P0 | Enforce limits in front of Artifacts: refuse objects over 32 MB and pushes that would take a repository over about 950 MB, with a git `ng` line (same framing as `declined`); stream the push body to Artifacts instead of buffering it twice, keeping only the command section and at most `MAX_SCANNED_PUSH` for scanning; stream `land` packs straight from upload-pack into receive-pack. | `git_http.rs` `forward`, `secret_scan.rs`, `land.rs` | Clear errors instead of late failures; no isolate OOM on 50–100 MB pushes | 2–3 days | |
| 301 | | R5 | P0 | Resilience: map `ArtifactsError.code` and HTTP 429/5xx to retry with jittered backoff (reads, mints, ref listings only; never a receive-pack), a per-isolate circuit breaker, `Retry-After` to git, and an "Artifacts" component on status.g1t.sh fed by the GraphQL error metrics (`rateLimited`, `serverError`, `storageLimitReached`). | `store.rs`, `git_http.rs`, `apps/status` | Brief Artifacts errors stop reaching users; incidents are visible | 2 days | |
| 302 | | R6 | P1 | Pack cache for full clones: when an upload-pack request has wants and no haves, key it by (repo id, `refs_version`, hash of the request) and serve the pack from R2; fill it on a miss with a tee of the response. Sandboxes clone the same commit repeatedly per pull request. | `git_http.rs`, `refs_cache.rs` pattern, new R2 bucket | Removes about 2 s of pack building per repeat clone and a large share of `pull` operations | 3–4 days | |
| 303 | | R7 | P1 | Shard namespaces: bindings `ARTIFACTS_0…N` plus `ARTIFACTS_EU`; record each repository's namespace in the registry `store` column (already read by `remember_store`); new repositories and forks go to the least-loaded shard; forks in their own namespaces. | `services/repos/wrangler.jsonc`, `store.rs`, `registry.rs` | Multiplies the control-plane ceiling; enables EU residency | 3 days | |
| 304 | | R8 | P1 | Sandboxes clone less: `--depth=1` for checks and deploy; `--filter=blob:none` only where lazy blob fetches are few (review of a small diff); keep full clones for agent runs and merges. Or restore a cached sandbox snapshot and `git fetch` (Sandbox SDK backups). | `crates/runner/src/*.rs` clone calls | Measured: 3.8 s (depth 1) and 3.4 s (blobless) against 5.4 s for g1t; fewer bytes from Artifacts | 1–2 days | |
| 305 | | R9 | P1 | Cache mutable reads by `refs_version`: `log(branch, n)`, `branches()` (parse the kept `refs_cache` advertisement instead of a new `info/refs`), and `readFile` (resolve the path through cached trees, then a cached `readBlob`). | `store.rs`, `refs.rs`, `lib.rs` `tree`/`blob`/`branches` | Most web page views stop reaching Artifacts | 2 days | |
| 306 | | R10 | P1 | Calm mergeability: coalesce re-checks per repository (one pass per target head, at most once a minute), resolve the target head once and `log` by hash (already cached), and cache `readCommit` like other objects. | `services/work/src/mergeability.rs`, `lib.rs` `divergence`, `descends_from` | The largest scenario-B hot spot drops by an order of magnitude | 1–2 days | |
| 307 | | R11 | P2 | Back up every repository to R2: a nightly incremental `git bundle` per repository whose `refs_version` changed, made in a Container (Workers cannot run git), with a restore drill. | new job in `services/runner` or a Container | Durability independent of the beta; also the export path | 3–5 days | |
| 308 | | R12 | P2 | Make the git store a real fallback: run `deploy/self-host/gitstore` on a host with persistent disk (Containers' disk is ephemeral and at most 20 GB), restore from the R2 bundles, and switch `GitStore` per namespace shard. | `deploy/self-host/gitstore`, `store.rs` | A tested exit path; not a hot standby | 1–2 weeks | |
| 309 | | R13 | P2 | Take the operation counter off the request path: record the count in `waitUntil` (or Analytics Engine), and check limits against a value cached for a minute. | `lib.rs` `git_limits`, `git_ops.rs` | Saves 63–98 ms on every fetch and push; removes a hot D1 row per workspace-hour | 1 day | |
| 310 | | R14 | P2 | Try Smart Placement or a placement hint for `g1t-repos`, which makes several sequential calls to Artifacts per request; compare `Server-Timing` before and after. | `services/repos/wrangler.jsonc` | Possibly lower latency for distant users | 0.5 day | |
| 311 | | R15 | P3 | Large files: an LFS endpoint backed by R2 for files over 32 MB. | new route in `services/repos` | Repositories with binaries can move to g1t | 1–2 weeks | |
| 312 | |
| 313 | ## 7. Questions for Cloudflare |
| 314 | |
| 315 | Ask in the Artifacts beta channel (the [beta request form](https://forms.gle/DwBoPRa3CWQ8ajFp7) is the |
| 316 | listed contact), through our account team, and in the Cloudflare Developers Discord. As a git |
| 317 | competition entrant (due 2026-10-14) we can also ask the competition organisers. Get answers in writing. |
| 318 | |
| 319 | 1. Exactly what is billed as an operation? Is each `upload-pack` and `receive-pack` request one, or each |
| 320 | clone or fetch? Do `info/refs`, `ls-refs`, binding calls (`get`, `info`, `createToken`, `log`, |
| 321 | `readTree`, `readBlob`, `readCommit`, `readFile`) or token creation count? |
| 322 | 2. Does `fork` share objects with its source or copy them? How is a fork's storage billed? Does |
| 323 | `objects` in the fork response mean objects copied? |
| 324 | 3. Is protocol v2 `filter` (partial clone) supported, given it works today but the docs say it does not? |
| 325 | 4. Do binding calls count toward "2,000 requests per 10 seconds per namespace"? What happens past it |
| 326 | (429, `rateLimited`, queueing)? Can it be raised per namespace? |
| 327 | 5. Read-after-write: once a receive-pack response has ended, are refs visible to `log`, `info/refs` and |
| 328 | other locations at once? |
| 329 | 6. Is `receive-pack` with `atomic` supported for multi-ref pushes? |
| 330 | 7. Where does a repository's Durable Object live, can it move, and can we hint a region per repository? |
| 331 | 8. What SLA, support path and incident communication apply during open beta? When is GA? |
| 332 | 9. Is there an export or snapshot restore for a deleted or corrupted repository? How long are R2 |
| 333 | snapshots kept? |
| 334 | 10. Can the 1 GB per repository and 32 MB per file limits be raised, and what does a client see when a |
| 335 | push crosses them? |
| 336 | 11. Does Artifacts garbage-collect or repack? Do deleted branches and force pushes free storage, and when? |
| 337 | 12. Can the 1 TB account limit be raised now, ahead of launch? |
| 338 | 13. Are per-repository event subscriptions practical at tens of thousands of repositories, or is an |
| 339 | account-level `pushed` event planned? |
| 340 | |
| 341 | ## 8. Launch blockers |
| 342 | |
| 343 | - **Before 2026-10-14:** R1 (know what is billed and count the same thing) and R2's verification (fork |
| 344 | storage semantics). If forks copy objects, R2's deletion is a launch blocker for agent-heavy |
| 345 | workspaces, because the 1 TB account limit would stop every push. |
| 346 | - **Before agent volume ramps (about 1,000 pull requests a day):** R2 deletion, R3, R4, R5. |
| 347 | - **Before a few thousand active workspaces:** R6, R7, R9, R10. |
| 348 | - Everything else is resilience and exit planning, ideally done while the product is still invite-only. |
| 349 | |
| 350 | ## 9. What was built (2026-10-06) |
| 351 | |
| 352 | Code in `services/repos` unless named; one migration, |
| 353 | `migrations/0011_artifacts_meters_forks_health.sql` (new columns on `repos`, new tables |
| 354 | `artifacts_meters`, `operation_mapping`, `store_health`; additive, no backfill). R11 added |
| 355 | `migrations/0013_backups.sql` (a new table, `repo_backups`, and one `operation_mapping` row; |
| 356 | additive). |
| 357 | |
| 358 | | # | Status | What | |
| 359 | | --- | --- | --- | |
| 360 | | R1 | Built; Cloudflare's answer still needed | Every interaction with the store is metered raw (`meters.rs` → `artifacts_meters`, per day, namespace, repository, workspace and meter, with bytes where known): client git (`git.info_refs`, `git.ls_refs`, `git.fetch`, `git.receive_pack`), g1t's own git (`internal.git.*`: landing, catch-up, mirrors, branch listings, fork retirement), every binding call (`binding.get`, `binding.create_token`, `binding.log`, `binding.read_tree`, …, each retry included), and answers g1t served from its own cache (`cache.*`, never operations). Which meters are operations is data: `operation_mapping` (`cost_operations` for g1t's bill, `billable_operations` for workspaces), read every 5 minutes, changed with `set_operation_mapping` without a deploy. Default: `git.fetch`, `git.receive_pack`, `internal.git.fetch`, `internal.git.receive_pack`, `binding.create`, `binding.fork`, `binding.delete` = 1, everything else 0. `git_operations` (what billing reads) is filled from the meters × `billable_operations`, by the hour. RPCs: `artifacts_usage { from, to, workspace?, by_repo? }` (raw meters and the mapping, for the reconciler), `operation_mapping`, `set_operation_mapping { meter, cost_operations, billable_operations, note? }`. Script: `scripts/ops/artifacts-usage.mjs`. | |
| 361 | | R13 | Built | Nothing on the request path writes D1 for counting. Meters add up per isolate and are written in one batch from `ctx.wait_until` after every request (and at the end of the cron and queue handlers); a failed write is kept for the next. The free-workspace slow-down decides from counts the isolate read back after its last write plus what it added since (`git_ops::standing`, at most 10 minutes old) and billing's plan answer kept 5 minutes. The 63–98 ms `kept` step's D1 upsert is gone. A workspace whose counts this isolate never read is not slowed: nothing slows anyone on a guess. | |
| 362 | | R2 | Built; the fork storage test is yours to run | `pull.merged` and `pull.closed` set the fork's `retire_after` (`FORK_RETENTION_DAYS`, 1 day in production; 7 when unset); `pull.reopened` clears it, or makes the fork again. The hourly sweep (`23 * * * *`, 25 a run) keeps the fork's head in its repository as `refs/pull/<pull id>/head` (only missing objects travel; an empty pack when merged), records `retired_at` and `retired_head`, then deletes the fork from the store (a failed delete puts the row back). Reads of a retired fork (the pull request's changes, divergence, tree, blob, log, branches) are answered from the repository with the fork's branch mapped to the kept head (`forks.rs` `Viewed`). Anything that writes or uses git on it (git over HTTPS, `git_access`, catch-up, land, `delete_branch`) makes it again first (`revive`: fork, then move its branch to the head) and schedules it to go again. Work never emits `pull.reopened` today; the handler is ready for it. Script: `scripts/ops/fork-storage-test.mjs`. | |
| 363 | | R3 | Built | Credentials g1t uses itself: TTL 3,600 s, reused for 50 minutes (isolate and KV, key `cred2:<key>:<scope>:internal`). `git_access` hands out its own: TTL 300 s, reused 180 s (`…:handout`); `refs_open` still uses 300 s. Every internal path (land, catch-up, mirrors, branch listing, commits, deleting a branch) now reuses kept credentials instead of minting each time. The remote is worked out as `https://<account>.artifacts.cloudflare.net/git/<namespace>/<name>.git` (the documented format, `api/git-protocol`), learned per namespace from the first `info()` an isolate makes, which runs alongside `createToken` and so costs no time; after that a mint is `get` and `createToken`. Optional `ARTIFACTS_REMOTE_BASE` skips even the first `info()`. | |
| 364 | | R4 | Built | Pushes are read as they arrive (`request.stream()`) and walked by `pack_limits::PackSizer` (each object inflated into a 32 KiB window and thrown away): an object over 32 MB (a delta measured by the object it makes), or a push taking the repository and its forks (`stored_bytes`) past `REPO_STORAGE_LIMIT_BYTES` (950 MB), is declined with `ng` lines and `remote:` text; a repository already at the limit is refused at the push's `info/refs` in plain text. Up to 24 MiB is kept, scanned and sent on as one copy, not three. Past 24 MiB push protection cannot read the push, so it is declined (`LARGE_PUSHES=refuse`, failing closed) with a command to push in parts, the 100 MB network limit named; `LARGE_PUSHES=unscanned` streams it to the store instead, still size-checked (a violation ends the stream before the pack's checksum, so the store keeps nothing). A pack too large for the scanner to inflate (48 MB inflated) is declined the same way instead of let through. Landing streams: upload-pack's side-band answer is taken apart chunk by chunk (`pack_limits::Sideband`) straight into the receive-pack body. | |
| 365 | | R5 | Built | `resilience.rs` sorts errors into rate limited, transient (`INTERNAL_ERROR`, `UPSTREAM_UNAVAILABLE`, `*_IN_PROGRESS`, no code, HTTP 5xx) and permanent. Binding reads, `get`, `info`, `createToken`, `create` and `delete` try up to 3 times with exponential backoff and jitter (80 ms base, 400 ms for rate limits, 2 s cap); `fork` and every receive-pack never retry. Git reads (`info/refs`, upload-pack) retry on 429 and 5xx. Per isolate, each namespace has a breaker that opens after 5 transient failures in a row, for 10 s, then lets one probe through. Busy answers reach git as 429 (rate limited) or 503, with `Retry-After: 5`; the site's read RPCs (`tree`, `blob`, `log`, `branches`, `blame`, `compare`) answer an `Outcome` failure saying the git storage is busy; other RPCs answer 503 with the same words. Health is counted by the minute (`store_health`) and served by the `store_health { minutes }` RPC; status.g1t.sh lists **Git storage** through a new `REPOS` service binding (down: 25% or more of at least 5 calls failed, or the breaker refused calls; degraded: rate limited, or a mean call over 1.5 s). | |
| 366 | | R9 | Built | `log(branch)`, `branches()` and `read_file(ref, path)` are kept in the colo cache under the repository's `refs_version` (5 minutes at most, and only while `refs_cache::usable`), and by commit hash for good; a log by branch also fills the by-hash entry; `readCommit` (parents) is kept for good. Read RPCs open repositories through `read_git`, which sets the version. | |
| 367 | | R10 | Built in repos; work unchanged | `divergence` works out the target's side once per target head per isolate (`coalesce.rs`: the head under the refs version, then the history by hash, kept 60 s), and what the target changed between two trees once per pair (10 minutes). `readCommit` and logs by hash come from the cache. Work's fan-out (`after_push`, up to 100 pull requests) is unchanged: its 100 `divergence` calls now cost one walk of the target instead of 100. | |
| 368 | | R7 | Groundwork | `shards.rs`: bindings named in `ARTIFACTS_NAMESPACES` (JSON, binding → namespace; `ARTIFACTS` → `g1t` always there), a repository's namespace kept in its `store` column as `<namespace>/<key>` (no prefix means the `ARTIFACTS` namespace, so every existing key reads the same), new repositories placed by `ARTIFACTS_NEW_REPOS` (comma-separated, spread by an FNV hash of the repository id; names not bound are skipped), forks always in their repository's namespace, `ARTIFACTS_EU_NAMESPACE` reserved for EU residency (no workspace setting yet). Works with only `ARTIFACTS` bound, as today. | |
| 369 | | R8 | Built | `crates/runner/src/clone.rs`: every sandbox clones at `--depth=1` (a full g1t clone took 5.4 s, depth 1 took 3.8 s). Work that merges (catch-up, the merge queue, merge checks, a review's diff) deepens 50, 500, then 5000 commits until the two sides share one, and fetches everything only as the last resort (`share_history`). `G1T_CLONE_DEPTH` (0 or `full` for everything) and `G1T_CLONE_FILTER=blob:none` change it per runner. | |
| 370 | | R11 | Built; not yet deployed | Nightly `git bundle` backups to the `g1t-backups` R2 bucket, and a restore drill. Migration `0013_backups.sql` (`repo_backups`, and an `operation_mapping` row). See "R11: backups and the restore drill" below. | |
| 371 | |
| 372 | ### R1: reading `scripts/ops/artifacts-usage.mjs` |
| 373 | |
| 374 | ```sh |
| 375 | export CLOUDFLARE_API_TOKEN=<token with Account Analytics: Read (and D1: Read, or set CLOUDFLARE_D1_TOKEN)> |
| 376 | node scripts/ops/artifacts-usage.mjs # last 31 days, a table |
| 377 | node scripts/ops/artifacts-usage.mjs --days 7 --json > usage.json |
| 378 | ARTIFACTS_NAMESPACE=g1t node scripts/ops/artifacts-usage.mjs |
| 379 | ``` |
| 380 | |
| 381 | It prints, per day, Cloudflare's `pull`, `push`, `create`, `fork` and `delete` events and its |
| 382 | errors beside g1t's fetch and push meters and `git_operations`, then, for each Cloudflare event, |
| 383 | the ratio Cloudflare ÷ g1t for several combinations of meters. Read it like this: |
| 384 | |
| 385 | - `pull` ≈ `git.fetch + internal.git.fetch` (ratio 1.00): Cloudflare counts one pull per |
| 386 | upload-pack fetch, as assumed. Keep the default mapping. |
| 387 | - `pull` ≈ a combination with `git.ls_refs` or `git.info_refs`: listing refs counts too. Set |
| 388 | `cost_operations` for those meters to 1 (`set_operation_mapping`), and decide whether |
| 389 | `billable_operations` follows (cost pass-through says yes). |
| 390 | - Every ratio well under 1: Cloudflare counts per clone or fetch session, not per request. |
| 391 | Ratios over 1: something reaches Artifacts that g1t does not meter, such as sandboxes pushing |
| 392 | directly with handed-out credentials. |
| 393 | - Only days after the meters were deployed compare; before that only `git_operations` exists. |
| 394 | - Binding calls do appear: Cloudflare's events include `read` and `token_create` actions (and |
| 395 | `namespace_*`) besides the five documented ones. If Cloudflare says they are billed, map the |
| 396 | `binding.*` meters in `operation_mapping`. |
| 397 | |
| 398 | A global API key works in place of the token: `CLOUDFLARE_API_KEY` with `CLOUDFLARE_EMAIL` |
| 399 | (both scripts). |
| 400 | |
| 401 | **Result, 2026-10-06** (31 days; g1t's meters cover only 2026-10-06, the day they shipped): |
| 402 | |
| 403 | | Cloudflare event | 31 days | 2026-10-06 | g1t's meters, 2026-10-06 | |
| 404 | | --- | --- | --- | --- | |
| 405 | | `read` | 137,225 | 107,616 | `binding.get` 85,206, `read_file` 49,846, `read_tree` 9,208, `read_blob` 4,964, `log` 2,798 | |
| 406 | | `pull` | 646 | 101 | `git.fetch` 29, `git.ls_refs` 26, `git.info_refs` 426 (+ 129 internal) | |
| 407 | | `push` | 385 | 10 | `git.receive_pack` 3 | |
| 408 | | `token_create` | 2,721 | 338 | `binding.create_token` 34 | |
| 409 | | `fork` / `create` / `delete` | 96 / 14 / 8 | 2 / 0 / 0 | | |
| 410 | | errors | 699 (688 client) | 476 client | | |
| 411 | |
| 412 | - `pull` is not one per upload-pack fetch: 101 pulls against 29 fetches on the one day both |
| 413 | exist. Over 31 days `pull` ≈ fetch + ls-refs + info/refs (ratio 1.06), so listing refs likely |
| 414 | counts as a pull. Not yet changed in `operation_mapping`: one day of meters is too little, and |
| 415 | re-check after a week before setting `cost_operations` for `git.info_refs` and `git.ls_refs`. |
| 416 | - `read` is the open question that matters. If reads are billed as operations at $0.15 per |
| 417 | 1,000, today's demo-scale traffic alone is about 3.2 million a month (~$480). Ask Cloudflare |
| 418 | (Q1) before 2026-10-14. Either way the volume is mostly waste: every repos call opens a handle |
| 419 | with `get` even when the answer is cached, and the object cache may not be hitting (no hit/miss |
| 420 | meter yet). Done on 2026-10-06: the handle's `get` waits for the first call that needs the |
| 421 | store (an answer from a cache, or `branches` over git, costs none); objects named by hash are |
| 422 | kept in the isolate (16 MB, oldest out first) ahead of the Cache API, and every look is |
| 423 | metered (`cache.memory_hit`, `cache.edge_hit`, `cache.miss`); issue and comment events start |
| 424 | nothing and read nothing when the synced `workflows` table has no workflow listening. Next: |
| 425 | read `cache.edge_hit` against `cache.miss` after a day; if the Cache API never hits from a |
| 426 | Worker reached only by service bindings, put objects in KV instead. Still to do: caller |
| 427 | attribution in the meters. |
| 428 | - 476 client errors on 2026-10-06 are unexplained; the fetch fix below accounts for some (every |
| 429 | failed negotiation was one). |
| 430 | |
| 431 | ### R2: running and reading `scripts/ops/fork-storage-test.mjs` |
| 432 | |
| 433 | ```sh |
| 434 | export CLOUDFLARE_API_TOKEN=<token: Artifacts edit, Account Analytics read> |
| 435 | node scripts/ops/fork-storage-test.mjs schema # which Artifacts analytics datasets exist |
| 436 | node scripts/ops/fork-storage-test.mjs run --keep # namespace g1t-storage-test: 100 MB repository, 5 forks |
| 437 | node scripts/ops/fork-storage-test.mjs measure # again tomorrow (storage is billed as a daily peak) |
| 438 | node scripts/ops/fork-storage-test.mjs cleanup # delete the 6 repositories |
| 439 | ``` |
| 440 | |
| 441 | It works only in its own namespace, through Cloudflare's REST API, never through g1t. Read: |
| 442 | |
| 443 | - Fork timing and response: a fork that returns in well under a second, with `objects` near the |
| 444 | source's count, is metadata (sharing). Seconds per fork, growing with size, suggests copying. |
| 445 | - Storage figures (any dataset `schema` lists besides `artifactsEventsAdaptiveGroups`): about |
| 446 | 100 MB after the forks means sharing; about 600 MB means each fork copied. The documentation |
| 447 | lists no storage dataset today, so this may print nothing: then the next day's usage in the |
| 448 | dashboard (Billing → Artifacts storage) is the measure, and the question stays with Cloudflare (Q2). |
| 449 | - `events`: `storageLimitReached` or other errors during the test. |
| 450 | |
| 451 | Either way R2 retires forks; the answer decides `FORK_RETENTION_DAYS` (shared: a week is fine; |
| 452 | copied: shorten it to 1 or 2 days and ask Cloudflare to raise the 1 TB account limit). |
| 453 | |
| 454 | **Result, 2026-10-06: forks are stored and billed as copies.** A 100 MB source took 57 s to |
| 455 | push; each of 5 forks took 4–6 s. The storage dataset (`artifactsStorageAdaptiveGroups`, |
| 456 | `max.repositorySizeBytes`) gave every fork the source's full 105,582,592 bytes: about 633 MB for |
| 457 | the six, not about 106 MB. Whatever Artifacts shares underneath, storage billing and the 1 TB |
| 458 | account limit see full copies. So: |
| 459 | |
| 460 | - `FORK_RETENTION_DAYS` is 1 in production (changed 2026-10-07). |
| 461 | - g1t meters a workspace's storage once per repository (`stored_bytes`), so an open pull |
| 462 | request's working copy is Cloudflare cost g1t absorbs: about $0.05 a month per 100 MB per open |
| 463 | pull request. Small now; decide whether open working copies count toward a workspace's |
| 464 | storage before agent pull requests reach thousands. |
| 465 | - Ask Cloudflare whether forks share objects physically, and for a higher account limit. |
| 466 | |
| 467 | Also found while testing (fixed in `git_http.rs`): the store answers a protocol v2 fetch that is |
| 468 | still negotiating, and whose `have`s it does not know, with `acknowledgments`, `NAK`, then a pack. |
| 469 | git refuses that ("expected no other sections to be sent after no 'ready'"). g1t now ends such |
| 470 | an answer after the acknowledgments with a flush, and the client negotiates again. Report it to |
| 471 | Cloudflare. |
| 472 | |
| 473 | ### R7: making more namespaces (yours to run, when needed) |
| 474 | |
| 475 | ```sh |
| 476 | # A US shard, unrestricted like today's g1t, and an EU one. |
| 477 | curl -X POST "https://api.cloudflare.com/client/v4/accounts/1e6f2cffa3f445920836e8ebe446bb58/artifacts/namespaces" \ |
| 478 | -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json" \ |
| 479 | --data '{"namespace":"g1t-us-1"}' |
| 480 | curl -X POST "https://api.cloudflare.com/client/v4/accounts/1e6f2cffa3f445920836e8ebe446bb58/artifacts/namespaces" \ |
| 481 | -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json" \ |
| 482 | --data '{"namespace":"g1t-eu","jurisdiction":"eu"}' |
| 483 | ``` |
| 484 | |
| 485 | Then in `services/repos/wrangler.jsonc`: |
| 486 | |
| 487 | ```jsonc |
| 488 | "artifacts": [ |
| 489 | { "binding": "ARTIFACTS", "namespace": "g1t" }, |
| 490 | { "binding": "ARTIFACTS_1", "namespace": "g1t-us-1" }, |
| 491 | { "binding": "ARTIFACTS_EU", "namespace": "g1t-eu" } |
| 492 | ], |
| 493 | "vars": { |
| 494 | "ARTIFACTS_NAMESPACES": "{\"ARTIFACTS\":\"g1t\",\"ARTIFACTS_1\":\"g1t-us-1\",\"ARTIFACTS_EU\":\"g1t-eu\"}", |
| 495 | "ARTIFACTS_NEW_REPOS": "g1t,g1t-us-1", // new repositories spread over both |
| 496 | "ARTIFACTS_EU_NAMESPACE": "g1t-eu" // used once a workspace can choose the EU |
| 497 | } |
| 498 | ``` |
| 499 | |
| 500 | Existing repositories stay where they are (`store` without a prefix). Deploy the binding before |
| 501 | naming its namespace in `ARTIFACTS_NEW_REPOS`; a name that is not bound is skipped, never used. |
| 502 | Moving an existing repository between namespaces is not built (a clone and push, then a `store` |
| 503 | update). |
| 504 | |
| 505 | ### R11: backups and the restore drill |
| 506 | |
| 507 | Every repository whose refs moved is bundled once a night and kept outside the git store, so a |
| 508 | repository can be rebuilt without Artifacts. The flow is in `crates/contracts/src/backups.rs`; |
| 509 | the chain, the manifest and the record are in `services/repos/src/backups.rs`. |
| 510 | |
| 511 | 1. **Queued.** At 02:53 UTC (`53 2 * * *` in `services/repos/wrangler.jsonc`) the repos service |
| 512 | queues the repositories that are due, at most `BACKUPS_PER_NIGHT` (200), the longest since |
| 513 | their last backup first. A repository is due when it has never been backed up, when its |
| 514 | `refs_version` went past the one its last backup was cut at, or when a credential that can |
| 515 | push was handed out (`refs_open_until`) after that backup's clone began: a push with such a |
| 516 | credential does not move `refs_version`. Deleted repositories, retired working copies and |
| 517 | pull request working copies (`pulls/…`, whose heads end up in their repository as |
| 518 | `refs/pull/<id>/head`) are not backed up. |
| 519 | 2. **Claimed.** The runner's five-minute sweep claims `BACKUPS_PER_SWEEP` (4) at a time, with at |
| 520 | most `BACKUPS_RUNNING` (6) running (`claim_backups`), and starts a sandbox for each in |
| 521 | `MODE=backup` (`crates/runner/src/backup.rs`). The sandbox is given the job's id and a token |
| 522 | for it, nothing else; the repos service keeps only the token's hash. It has a 60-minute time |
| 523 | cap. Its time is g1t's: it is not metered to the workspace. |
| 524 | 3. **Cut.** The sandbox asks for its job (`POST api.g1t.sh/backups/{job}/spec`, the token in |
| 525 | `x-g1t-backup-token`) and gets a read-only credential for the repository in the store (a |
| 526 | `git_access`-style handout, 5 minutes), the bundle's kind, and the commits the last bundle |
| 527 | ended at. It clones with `--mirror` (every ref, never shallow), writes those commits as refs |
| 528 | of its own, and runs `git bundle create --all --not <them>`, then `git bundle verify`. |
| 529 | When the clone has exactly the refs of the last backup, or git finds nothing new to bundle |
| 530 | (a branch deleted, a ref moved to a commit already kept), no bundle is cut and only the refs |
| 531 | are recorded. |
| 532 | 4. **Sent.** The bundle goes in 32 MiB parts (`PUT /backups/{job}/parts/{n}`), which the API |
| 533 | passes to the repos service and the repos service to an R2 multipart upload; then |
| 534 | `POST /backups/{job}/complete` with every ref, the size, the SHA-256 and the parts. A failure |
| 535 | is `POST /backups/{job}/fail`; a sandbox that dies is failed by the runner. A job is tried 3 |
| 536 | times a night; one running past 3 hours is queued again. |
| 537 | 5. **Recorded.** The manifest gains the entry, and `repo_backups` the refs version the clone began |
| 538 | at, so a push during the backup leaves the repository due the next night. |
| 539 | |
| 540 | Storage, through the `BlobStore` port in `crates/blobstore` (the adapters packages already used): |
| 541 | the `BACKUPS` binding (bucket `g1t-backups`) with `BACKUP_STORE=r2`; any S3-compatible store with |
| 542 | `BACKUP_STORE=s3` and `BACKUP_S3_BUCKET` (self-hosted: MinIO). Without either, backups are off and |
| 543 | the nightly cron does nothing. |
| 544 | |
| 545 | ```text |
| 546 | backups/<repo id>/manifest.json |
| 547 | backups/<repo id>/20261006T025300Z-full.bundle |
| 548 | backups/<repo id>/20261007T025302Z-incr.bundle |
| 549 | ``` |
| 550 | |
| 551 | The manifest (version 1) lists `chain`, oldest first, and `previous`, the chain before it. Each |
| 552 | entry has `id`, `kind` (`full` or `incremental`), `key` (null when only refs moved), |
| 553 | `created_at`, `refs_version`, `refs` (every ref and `HEAD` once it is applied), `prerequisites`, |
| 554 | `size` and `sha256`. The first backup is full; the next ones are incremental, their |
| 555 | prerequisites the last entry's tips, until the chain holds `BACKUP_FULL_EVERY` (30) incremental |
| 556 | ones, when a full one starts a new chain. The chain before that is kept until the next full one |
| 557 | replaces it, so the oldest backup kept is about two chains old. Backups of purged repositories |
| 558 | are removed the night after (50 a night). |
| 559 | |
| 560 | Meters: the clone counts as `internal.git.info_refs` and `internal.git.backup_fetch` with the |
| 561 | bytes it read, on the repository (they show in `artifacts_usage` and |
| 562 | `scripts/ops/artifacts-usage.mjs`). `operation_mapping` has `internal.git.backup_fetch` at 1 for |
| 563 | `cost_operations` and 0 for `billable_operations`: an operation on g1t's bill, never on the |
| 564 | workspace's. The credential's `binding.create_token` is metered as before. |
| 565 | |
| 566 | **The restore drill** (read-only against production: SELECTs on `g1t-repos`, reads of |
| 567 | `g1t-backups` through Wrangler, `git ls-remote` of the live repository): |
| 568 | |
| 569 | ```sh |
| 570 | node scripts/ops/backup-restore-drill.mjs # a repository unchanged since its last backup |
| 571 | node scripts/ops/backup-restore-drill.mjs --repo acme/rocket # this one |
| 572 | G1T_USER=you G1T_TOKEN=g1t_... node scripts/ops/backup-restore-drill.mjs --repo acme/private-thing |
| 573 | ``` |
| 574 | |
| 575 | It downloads the manifest and each bundle of the chain, checks each against its size and SHA-256, |
| 576 | `git bundle verify`s it, fetches it into a new bare repository without following tags, sets every |
| 577 | ref to what the last entry says (and removes the rest), points `HEAD` at the branch at its commit, |
| 578 | and runs `git fsck --connectivity-only`. Then it compares every ref with the manifest and with |
| 579 | `git ls-remote` of the live repository and prints each difference. Exit 0: every ref matches; |
| 580 | 1: a difference; 2: it could not run (a bundle that does not match its manifest is this). Picked |
| 581 | at random, the repository is one whose refs have not moved since its last backup, so any |
| 582 | difference is the backup's. Run it after the first night, then monthly, and after any change to |
| 583 | `backups.rs` or `backup.rs`. `--bundles <dir>` reads a local copy of the bucket instead |
| 584 | (self-hosted: `mc mirror local/g1t-backups <dir>`), with `--repo-id` and `--live <url or path>`. |
| 585 | `npm run test:ops` runs it against bundles cut with git. |
| 586 | |
| 587 | **A real restore into the store**, as it can be done today: |
| 588 | |
| 589 | 1. Run the drill for the repository with `--keep`. It prints where the restored copy is |
| 590 | (`…/restored.git`). Go on only if every ref matches the manifest; differences from the live |
| 591 | repository are what the restore is for. |
| 592 | 2. Tell the workspace, and stop the repository's agents and merge queue for the time. |
| 593 | 3. If its default branch is protected, turn protection off in the repository's settings for the |
| 594 | push: a push that changes a protected branch is declined. |
| 595 | 4. From the restored copy, push every ref as an owner, with an access token that has |
| 596 | `code:write`: |
| 597 | |
| 598 | ```sh |
| 599 | cd /tmp/g1t-drill-…/restored.git |
| 600 | git -c "http.extraHeader=Authorization: Basic $(printf 'you:g1t_...' | base64)" \ |
| 601 | push --force https://g1t.sh/acme/rocket.git 'refs/*:refs/*' |
| 602 | ``` |
| 603 | |
| 604 | It goes through the git door like any push: size limits, push protection (pushes over 24 MiB |
| 605 | per `LARGE_PUSHES`) and the audit log apply, and the refs version moves, so the next night |
| 606 | backs the repository up again. `--force` rewinds refs that went wrong; refs the live |
| 607 | repository has that the backup does not are left alone (`git push --mirror` would delete |
| 608 | them). |
| 609 | 5. Turn protection back on, and run the drill again: every ref now matches the live repository. |
| 610 | |
| 611 | When the repository is gone from the store itself (its key answers not found), there is no |
| 612 | operator call yet to make an empty repository under an existing row's key; that is part of R12. |
| 613 | |
| 614 | To deploy: make the bucket (`npx wrangler r2 bucket create g1t-backups`, a setup step of `repos` |
| 615 | in `deploy/stack.jsonc`), then migration 0013, then `g1t-repos` (the `BACKUPS` binding and the |
| 616 | new cron), `g1t-api` (the `/backups/` door), and `g1t-runner` (a new image: the `backup` mode). |
| 617 | Until the runner is out, queued backups wait; nothing fails. Set `BACKUPS_PER_SWEEP` to `0` on the |
| 618 | runner to stop starting them. |
| 619 | |
| 620 | What is not covered: a pull request's working copy while its pull request is open (its head is |
| 621 | kept in the repository only once the working copy is retired), and anything that is not a git |
| 622 | ref (issues, pull requests and the rest live in D1, which has its own Time Travel). Every backup |
| 623 | clones the whole repository, so a night reads each changed repository in full from the store; |
| 624 | incremental bundles save storage, not reads. |
| 625 | |
| 626 | ### Deploy order and what to watch |
| 627 | |
| 628 | 1. Migration 0011 (the deploy tool applies migrations first). `forks_of` reads `retired_at`, so |
| 629 | the new code must not run before it. |
| 630 | 2. `g1t-repos` (new vars in `wrangler.jsonc`; no new bindings). |
| 631 | 3. `g1t-status` (a new `REPOS` service binding; **Git storage** appears once deployed). |
| 632 | 4. After a day: `scripts/ops/artifacts-usage.mjs`; then the fork test. |
| 633 | |
| 634 | Expected: `Server-Timing` `kept` on POSTs drops from 63–98 ms to the KV and Cache lookups only (a |
| 635 | few ms); `mint` happens about once per repository and scope every 50 minutes per isolate instead |
| 636 | of every 3 minutes, and costs `get` and `createToken` (about 0.5 s instead of 0.8 s) once an |
| 637 | isolate knows its namespace's prefix; branch pages and repeated tree and file views skip |
| 638 | Artifacts while the refs version holds; a burst of 100 mergeability checks walks the target once. |
| 639 | |
| 640 | Risks: what Cloudflare bills is still theirs to confirm (the mapping makes changing it cheap). |
| 641 | Counts held by an isolate that is evicted before its `wait_until` write are lost (seconds of |
| 642 | traffic). Pushes over 24 MiB now fail closed: bringing a large existing repository needs the |
| 643 | push-in-parts command (in `guides/git.md`). Moving a revived fork's branch back to its old head |
| 644 | is a non-fast-forward update, which depends on Artifacts accepting it as git does by default; if |
| 645 | it refuses, the fork of a closed pull request cannot be made again and the error says so. |
| 646 | `refs/pull/*` refs appear in full ref advertisements (mirrors, `--mirror` clones). Operations on |
| 647 | pull request forks are metered under the workspace `pulls`, so they are counted for g1t's bill |
| 648 | but not charged to a workspace (except the fork itself, metered on its repository). |