| 28 | 28 | | 128 MB Worker isolate that buffers each push body twice. Large pushes and imports fail late, without a |
| 29 | 29 | | message git can show. |
| 30 | 30 | | 5. **No backup, no exit drill.** Cloudflare replicates data, but there is no SLA, no documented export |
| 31 | | − | besides git itself, and the self-host git store is not a production fallback yet. |
| 31 | + | besides git itself, and the self-host git store is not a production fallback yet. Nightly bundles |
| 32 | + | to R2 and a restore drill are now built (R11, section 9); the fallback store is not (R12). |
| 32 | 33 | | |
| 33 | 34 | | None of these blocks an invite-only launch. Items 1 and 2 must be answered before billing starts on |
| 34 | 35 | | 2026-10-14, and the fork cleanup must ship before agent pull requests reach thousands a day. |
| ⋯ |
| 350 | 351 | | |
| 351 | 352 | | Code in `services/repos` unless named; one migration, |
| 352 | 353 | | `migrations/0011_artifacts_meters_forks_health.sql` (new columns on `repos`, new tables |
| 353 | | − | `artifacts_meters`, `operation_mapping`, `store_health`; additive, no backfill). |
| 354 | + | `artifacts_meters`, `operation_mapping`, `store_health`; additive, no backfill). R11 added |
| 355 | + | `migrations/0013_backups.sql` (a new table, `repo_backups`, and one `operation_mapping` row; |
| 356 | + | additive). |
| 354 | 357 | | |
| 355 | 358 | | | # | Status | What | |
| 356 | 359 | | | --- | --- | --- | |
| ⋯ |
| 364 | 367 | | | R10 | Built in repos; work unchanged | `divergence` works out the target's side once per target head per isolate (`coalesce.rs`: the head under the refs version, then the history by hash, kept 60 s), and what the target changed between two trees once per pair (10 minutes). `readCommit` and logs by hash come from the cache. Work's fan-out (`after_push`, up to 100 pull requests) is unchanged: its 100 `divergence` calls now cost one walk of the target instead of 100. | |
| 365 | 368 | | | R7 | Groundwork | `shards.rs`: bindings named in `ARTIFACTS_NAMESPACES` (JSON, binding → namespace; `ARTIFACTS` → `g1t` always there), a repository's namespace kept in its `store` column as `<namespace>/<key>` (no prefix means the `ARTIFACTS` namespace, so every existing key reads the same), new repositories placed by `ARTIFACTS_NEW_REPOS` (comma-separated, spread by an FNV hash of the repository id; names not bound are skipped), forks always in their repository's namespace, `ARTIFACTS_EU_NAMESPACE` reserved for EU residency (no workspace setting yet). Works with only `ARTIFACTS` bound, as today. | |
| 366 | 369 | | | R8 | Built | `crates/runner/src/clone.rs`: every sandbox clones at `--depth=1` (a full g1t clone took 5.4 s, depth 1 took 3.8 s). Work that merges (catch-up, the merge queue, merge checks, a review's diff) deepens 50, 500, then 5000 commits until the two sides share one, and fetches everything only as the last resort (`share_history`). `G1T_CLONE_DEPTH` (0 or `full` for everything) and `G1T_CLONE_FILTER=blob:none` change it per runner. | |
| 370 | + | | R11 | Built; not yet deployed | Nightly `git bundle` backups to the `g1t-backups` R2 bucket, and a restore drill. Migration `0013_backups.sql` (`repo_backups`, and an `operation_mapping` row). See "R11: backups and the restore drill" below. | |
| 367 | 371 | | |
| 368 | 372 | | ### R1: reading `scripts/ops/artifacts-usage.mjs` |
| 369 | 373 | | |
| ⋯ |
| 498 | 502 | | Moving an existing repository between namespaces is not built (a clone and push, then a `store` |
| 499 | 503 | | update). |
| 500 | 504 | | |
| 505 | + | ### R11: backups and the restore drill |
| 506 | + | |
| 507 | + | Every repository whose refs moved is bundled once a night and kept outside the git store, so a |
| 508 | + | repository can be rebuilt without Artifacts. The flow is in `crates/contracts/src/backups.rs`; |
| 509 | + | the chain, the manifest and the record are in `services/repos/src/backups.rs`. |
| 510 | + | |
| 511 | + | 1. **Queued.** At 02:53 UTC (`53 2 * * *` in `services/repos/wrangler.jsonc`) the repos service |
| 512 | + | queues the repositories that are due, at most `BACKUPS_PER_NIGHT` (200), the longest since |
| 513 | + | their last backup first. A repository is due when it has never been backed up, when its |
| 514 | + | `refs_version` went past the one its last backup was cut at, or when a credential that can |
| 515 | + | push was handed out (`refs_open_until`) after that backup's clone began: a push with such a |
| 516 | + | credential does not move `refs_version`. Deleted repositories, retired working copies and |
| 517 | + | pull request working copies (`pulls/…`, whose heads end up in their repository as |
| 518 | + | `refs/pull/<id>/head`) are not backed up. |
| 519 | + | 2. **Claimed.** The runner's five-minute sweep claims `BACKUPS_PER_SWEEP` (4) at a time, with at |
| 520 | + | most `BACKUPS_RUNNING` (6) running (`claim_backups`), and starts a sandbox for each in |
| 521 | + | `MODE=backup` (`crates/runner/src/backup.rs`). The sandbox is given the job's id and a token |
| 522 | + | for it, nothing else; the repos service keeps only the token's hash. It has a 60-minute time |
| 523 | + | cap. Its time is g1t's: it is not metered to the workspace. |
| 524 | + | 3. **Cut.** The sandbox asks for its job (`POST api.g1t.sh/backups/{job}/spec`, the token in |
| 525 | + | `x-g1t-backup-token`) and gets a read-only credential for the repository in the store (a |
| 526 | + | `git_access`-style handout, 5 minutes), the bundle's kind, and the commits the last bundle |
| 527 | + | ended at. It clones with `--mirror` (every ref, never shallow), writes those commits as refs |
| 528 | + | of its own, and runs `git bundle create --all --not <them>`, then `git bundle verify`. |
| 529 | + | When the clone has exactly the refs of the last backup, or git finds nothing new to bundle |
| 530 | + | (a branch deleted, a ref moved to a commit already kept), no bundle is cut and only the refs |
| 531 | + | are recorded. |
| 532 | + | 4. **Sent.** The bundle goes in 32 MiB parts (`PUT /backups/{job}/parts/{n}`), which the API |
| 533 | + | passes to the repos service and the repos service to an R2 multipart upload; then |
| 534 | + | `POST /backups/{job}/complete` with every ref, the size, the SHA-256 and the parts. A failure |
| 535 | + | is `POST /backups/{job}/fail`; a sandbox that dies is failed by the runner. A job is tried 3 |
| 536 | + | times a night; one running past 3 hours is queued again. |
| 537 | + | 5. **Recorded.** The manifest gains the entry, and `repo_backups` the refs version the clone began |
| 538 | + | at, so a push during the backup leaves the repository due the next night. |
| 539 | + | |
| 540 | + | Storage, through the `BlobStore` port in `crates/blobstore` (the adapters packages already used): |
| 541 | + | the `BACKUPS` binding (bucket `g1t-backups`) with `BACKUP_STORE=r2`; any S3-compatible store with |
| 542 | + | `BACKUP_STORE=s3` and `BACKUP_S3_BUCKET` (self-hosted: MinIO). Without either, backups are off and |
| 543 | + | the nightly cron does nothing. |
| 544 | + | |
| 545 | + | ```text |
| 546 | + | backups/<repo id>/manifest.json |
| 547 | + | backups/<repo id>/20261006T025300Z-full.bundle |
| 548 | + | backups/<repo id>/20261007T025302Z-incr.bundle |
| 549 | + | ``` |
| 550 | + | |
| 551 | + | The manifest (version 1) lists `chain`, oldest first, and `previous`, the chain before it. Each |
| 552 | + | entry has `id`, `kind` (`full` or `incremental`), `key` (null when only refs moved), |
| 553 | + | `created_at`, `refs_version`, `refs` (every ref and `HEAD` once it is applied), `prerequisites`, |
| 554 | + | `size` and `sha256`. The first backup is full; the next ones are incremental, their |
| 555 | + | prerequisites the last entry's tips, until the chain holds `BACKUP_FULL_EVERY` (30) incremental |
| 556 | + | ones, when a full one starts a new chain. The chain before that is kept until the next full one |
| 557 | + | replaces it, so the oldest backup kept is about two chains old. Backups of purged repositories |
| 558 | + | are removed the night after (50 a night). |
| 559 | + | |
| 560 | + | Meters: the clone counts as `internal.git.info_refs` and `internal.git.backup_fetch` with the |
| 561 | + | bytes it read, on the repository (they show in `artifacts_usage` and |
| 562 | + | `scripts/ops/artifacts-usage.mjs`). `operation_mapping` has `internal.git.backup_fetch` at 1 for |
| 563 | + | `cost_operations` and 0 for `billable_operations`: an operation on g1t's bill, never on the |
| 564 | + | workspace's. The credential's `binding.create_token` is metered as before. |
| 565 | + | |
| 566 | + | **The restore drill** (read-only against production: SELECTs on `g1t-repos`, reads of |
| 567 | + | `g1t-backups` through Wrangler, `git ls-remote` of the live repository): |
| 568 | + | |
| 569 | + | ```sh |
| 570 | + | node scripts/ops/backup-restore-drill.mjs # a repository unchanged since its last backup |
| 571 | + | node scripts/ops/backup-restore-drill.mjs --repo acme/rocket # this one |
| 572 | + | G1T_USER=you G1T_TOKEN=g1t_... node scripts/ops/backup-restore-drill.mjs --repo acme/private-thing |
| 573 | + | ``` |
| 574 | + | |
| 575 | + | It downloads the manifest and each bundle of the chain, checks each against its size and SHA-256, |
| 576 | + | `git bundle verify`s it, fetches it into a new bare repository without following tags, sets every |
| 577 | + | ref to what the last entry says (and removes the rest), points `HEAD` at the branch at its commit, |
| 578 | + | and runs `git fsck --connectivity-only`. Then it compares every ref with the manifest and with |
| 579 | + | `git ls-remote` of the live repository and prints each difference. Exit 0: every ref matches; |
| 580 | + | 1: a difference; 2: it could not run (a bundle that does not match its manifest is this). Picked |
| 581 | + | at random, the repository is one whose refs have not moved since its last backup, so any |
| 582 | + | difference is the backup's. Run it after the first night, then monthly, and after any change to |
| 583 | + | `backups.rs` or `backup.rs`. `--bundles <dir>` reads a local copy of the bucket instead |
| 584 | + | (self-hosted: `mc mirror local/g1t-backups <dir>`), with `--repo-id` and `--live <url or path>`. |
| 585 | + | `npm run test:ops` runs it against bundles cut with git. |
| 586 | + | |
| 587 | + | **A real restore into the store**, as it can be done today: |
| 588 | + | |
| 589 | + | 1. Run the drill for the repository with `--keep`. It prints where the restored copy is |
| 590 | + | (`…/restored.git`). Go on only if every ref matches the manifest; differences from the live |
| 591 | + | repository are what the restore is for. |
| 592 | + | 2. Tell the workspace, and stop the repository's agents and merge queue for the time. |
| 593 | + | 3. If its default branch is protected, turn protection off in the repository's settings for the |
| 594 | + | push: a push that changes a protected branch is declined. |
| 595 | + | 4. From the restored copy, push every ref as an owner, with an access token that has |
| 596 | + | `code:write`: |
| 597 | + | |
| 598 | + | ```sh |
| 599 | + | cd /tmp/g1t-drill-…/restored.git |
| 600 | + | git -c "http.extraHeader=Authorization: Basic $(printf 'you:g1t_...' | base64)" \ |
| 601 | + | push --force https://g1t.sh/acme/rocket.git 'refs/*:refs/*' |
| 602 | + | ``` |
| 603 | + | |
| 604 | + | It goes through the git door like any push: size limits, push protection (pushes over 24 MiB |
| 605 | + | per `LARGE_PUSHES`) and the audit log apply, and the refs version moves, so the next night |
| 606 | + | backs the repository up again. `--force` rewinds refs that went wrong; refs the live |
| 607 | + | repository has that the backup does not are left alone (`git push --mirror` would delete |
| 608 | + | them). |
| 609 | + | 5. Turn protection back on, and run the drill again: every ref now matches the live repository. |
| 610 | + | |
| 611 | + | When the repository is gone from the store itself (its key answers not found), there is no |
| 612 | + | operator call yet to make an empty repository under an existing row's key; that is part of R12. |
| 613 | + | |
| 614 | + | To deploy: make the bucket (`npx wrangler r2 bucket create g1t-backups`, a setup step of `repos` |
| 615 | + | in `deploy/stack.jsonc`), then migration 0013, then `g1t-repos` (the `BACKUPS` binding and the |
| 616 | + | new cron), `g1t-api` (the `/backups/` door), and `g1t-runner` (a new image: the `backup` mode). |
| 617 | + | Until the runner is out, queued backups wait; nothing fails. Set `BACKUPS_PER_SWEEP` to `0` on the |
| 618 | + | runner to stop starting them. |
| 619 | + | |
| 620 | + | What is not covered: a pull request's working copy while its pull request is open (its head is |
| 621 | + | kept in the repository only once the working copy is retired), and anything that is not a git |
| 622 | + | ref (issues, pull requests and the rest live in D1, which has its own Time Travel). Every backup |
| 623 | + | clones the whole repository, so a night reads each changed repository in full from the store; |
| 624 | + | incremental bundles save storage, not reads. |
| 625 | + | |
| 501 | 626 | | ### Deploy order and what to watch |
| 502 | 627 | | |
| 503 | 628 | | 1. Migration 0011 (the deploy tool applies migrations first). `forks_of` reads `retired_at`, so |