Sign in

flagon-io/g1t

public

Commit

Repositories shard across git store namespaces, move between them, and can keep to the EU; a namespace can be served read-only from the self-hosted git store, rebuilt from the nightly backups (#20, #25)

R7 (#20): new repositories go to the emptier healthy namespaces named in ARTIFACTS_NEW_REPOS (namespaces.rs loads, read only when there is a choice); moves.rs pauses writes, copies every ref over git, switches the store keys in one batch and cleans the old copies after 7 days; EU residency is a workspace setting (identity 0026), shown once an EU namespace takes repositories, applied at creation; artifacts-namespaces.mjs reports each namespace and queues moves. Migration repos 0014. R12 (#25): restore-to-gitstore.mjs rebuilds repositories from their bundle chains into the git store (now with namespaced keys and a read-only mode), lists what it took while serving, and reconciles it back; fallback.rs switches a namespace's GitStore to it by configuration, read-only unless told otherwise; status shows Git storage degraded meanwhile. Runbook, host and cost in docs/ARTIFACTS.md. Nothing changes in production until namespaces, bindings and the fallback host exist.

syntaqxcommitted Parent73aad90Browse files
39 files+2395−490/39 viewed
+1−0
2929 | `repo.collaborator_added`, `repo.collaborator_role_changed`, `repo.collaborator_removed` | Someone was given a role on it, had it changed, or lost it. See [access and roles](/guides/access-and-roles/). |
3030 | `repo.invitation_created`, `repo.invitation_revoked` | Someone was invited to it, or an invitation was withdrawn. |
3131 | `workspace.base_permission_changed` | An owner changed what members get on every repository. |
32+| `workspace.residency_changed` | An owner changed where the workspace's new repositories are stored. See [data residency](/guides/workspaces/#data-residency). |
3233 | `workspace.deleted`, `workspace.restored`, `workspace.purged` | An owner deleted the workspace, g1t's support restored it, or it was removed for good. See [deleting a workspace](/guides/workspaces/#delete-a-workspace). |
3334
3435 Through the API and the MCP server, the call itself is recorded under its
+1−1
2222 | Website and sign-in | Loads `g1t.sh/login`, and asks the API about an access token no one holds, which the account service must refuse with `401` |
2323 | API | Loads `api.g1t.sh/` |
2424 | Git and repositories | Lists the branches of a public repository over HTTPS (`info/refs`), the first step of every clone |
25−| Git storage | How the store that keeps every repository (Cloudflare Artifacts) answered g1t over the last five minutes. Down when a quarter or more of its calls failed, or g1t stopped asking after repeated failures; degraded when it rate limited g1t or its calls took over 1.5 seconds on average |
25+| Git storage | How the store that keeps every repository (Cloudflare Artifacts) answered g1t over the last five minutes. Down when a quarter or more of its calls failed, or g1t stopped asking after repeated failures; degraded when it rate limited g1t, its calls took over 1.5 seconds on average, or g1t is serving repositories from its backup store (reads work, pushes and merges wait) |
2626 | Page speed | Loads a public project page (`g1t.sh/flagon-io/g1t`) and `g1t.sh/explore`, timed to the first byte of each answer |
2727 | MCP server | Loads `mcp.g1t.sh/` |
2828 | Documentation | Loads `docs.g1t.sh/` |
+33−0
141141 digits and single hyphens, up to 39 characters, not a reserved word, and
142142 not another workspace's slug or someone else's username.
143143
144+## Data residency
145+
146+Data residency says where the git data of the workspace's new repositories
147+is stored. The section appears in **Settings** once g1t can store
148+repositories in the EU. Until then it is not shown, and every repository is
149+stored wherever g1t stores repositories.
150+
151+| Setting | What it does |
152+| --- | --- |
153+| Anywhere | New repositories are stored wherever g1t stores repositories. The default. |
154+| EU only | New repositories are stored in the EU. If EU storage cannot take one right now, the repository is not made, and you are told why. It is never stored somewhere else instead. |
155+
156+To change it:
157+
158+1. Open the workspace, then **Settings**. Only owners see the page.
159+2. Under **Data residency**, choose **Anywhere** or **EU only**.
160+3. Select **Save**.
161+
162+The setting applies to repositories made after you save it, however they
163+are made: from the site, with the API, by pushing to a new address, or by
164+importing. Repositories the workspace already has stay where they are. To
165+move them, contact support; a move keeps each repository's address, history
166+and settings, and pushes to it wait a few minutes while it happens.
167+
168+Data residency covers the git data: commits, branches, tags and files,
169+including pull requests' working copies, which are stored with their
170+repository. Issues, pull requests, comments and settings are not affected.
171+A repository [transferred](/guides/transferring-repositories/) to another
172+workspace stays where it is stored.
173+
174+Changing the setting is recorded in the
175+[audit log](/guides/audit-log/) as `workspace.residency_changed`.
176+
144177 ## Delete a workspace
145178
146179 Deleting a workspace takes everything in it with it, in one step: its
+3−0
6161 // Rate limited: degraded, not down.
6262 const limited = judgeStorage(report(row(100, 2, 2, 0, 10_000)));
6363 assert.deepEqual([limited.ok, limited.degraded], [true, "Rate limited 2 times in 5 minutes"]);
64+ // Served from the fallback store: degraded, whatever its own calls did.
65+ const onFallback = judgeStorage(report({ ...row(20, 0, 0, 0, 200), store: "g1t@fallback" }));
66+ assert.deepEqual([onFallback.ok, onFallback.degraded], [true, "Served from the backup store: reads work, pushes and merges wait"]);
6467 const viaCheck = await runCheck({ kind: "storage" }, { fetch: answer(200), billing: null, storage: async () => report(row(1, 0, 0, 0, 5)) });
6568 assert.deepEqual(viaCheck, { ok: true, ms: 5 });
6669 assert.equal(await runCheck({ kind: "storage" }, { fetch: answer(200), billing: null }), null);
+6−0
5050 return { ok: false, ms, error: `${Math.round((100 * errors) / calls)}% of calls failed` };
5151 }
5252 if (limited > 0) return { ok: true, ms, degraded: `Rate limited ${limited} times in ${report.minutes} minutes` };
53+ // A namespace served from the fallback store (`<namespace>@fallback`,
54+ // repos src/fallback.rs): reads work from the last backup, writes wait.
55+ const fallback = report.stores.filter((row) => row.store.endsWith("@fallback") && Number(row.calls) > 0);
56+ if (fallback.length > 0) {
57+ return { ok: true, ms, degraded: "Served from the backup store: reads work, pushes and merges wait" };
58+ }
5359 return { ok: true, ms };
5460 }
5561
+86−2
22 import { Form, data, redirect, useFetcher, useNavigation } from "react-router";
33
44 import {
5+ type DataResidency,
56 RENAME_COOLDOWN_HOURS,
67 SLUG_HOLD_DAYS,
78 WORKSPACE_RESTORE_DAYS,
2627 } from "../../components/ui/alert-dialog";
2728 import { FieldDescription, FieldLabel, Field as FormField } from "../../components/ui/field";
2829 import { InputAddon, InputGroup, Input as TextInput } from "../../components/ui/input";
30+import { RadioGroup, RadioOption } from "../../components/ui/radio-group";
2931 import { readAvatarUpload } from "../../lib/avatar-upload";
30−import { deployments, identity } from "../../lib/services.server";
32+import { deployments, identity, repos } from "../../lib/services.server";
3133 import { assertSameOrigin, getViewer, requireUser, roleIn } from "../../lib/session.server";
3234 import { forgetWorkspace } from "../../lib/workspace-choice";
3335 import { confirmsSlug, deletionRefusal, whatGoes } from "../../lib/workspace-deletion";
7072 deletion = found?.ok ? found.value : null;
7173 apps = usage?.ok ? usage.value.apps : null;
7274 }
73− return { workspace, check, deletion, apps };
75+ // Where its repositories are kept: offered once g1t can keep them in the
76+ // EU, and shown to a workspace that chose it whatever happens since.
77+ const [storage, residency] = await Promise.all([
78+ repos.storageOptions().catch(() => ({ euAvailable: false })),
79+ identity.workspaceResidency(params.owner).catch(() => null),
80+ ]);
81+ return { workspace, check, deletion, apps, euAvailable: storage.euAvailable, residency: residency ?? "anywhere" };
7482 }
7583
7684 export async function action({ request, params, context }: Route.ActionArgs) {
101109 const secure = new URL(request.url).protocol === "https:";
102110 throw redirect("/", { headers: { "Set-Cookie": forgetWorkspace(secure) } });
103111 }
112+ // Where new repositories keep their data: identity checks the owner.
113+ if (intent === "residency") {
114+ const wanted = form.get("residency") === "eu" ? "eu" : "anywhere";
115+ const result = await identity.setWorkspaceResidency(user, params.owner, wanted);
116+ if (!result.ok) return { residencyError: result.error.message };
117+ return { saved: "residency" as const };
118+ }
104119 if (intent === "rename") {
105120 const newSlug = String(form.get("newSlug") ?? "").trim().toLowerCase();
106121 const result = await identity.renameWorkspace(user, params.owner, newSlug);
154169 error={actionData && "renameError" in actionData ? actionData.renameError : undefined}
155170 />
156171
172+ {(loaderData.euAvailable || loaderData.residency === "eu") && (
173+ <ResidencySection
174+ residency={loaderData.residency}
175+ euAvailable={loaderData.euAvailable}
176+ saved={Boolean(actionData && "saved" in actionData && actionData.saved === "residency")}
177+ error={actionData && "residencyError" in actionData ? actionData.residencyError : undefined}
178+ />
179+ )}
180+
157181 <DangerZone>
158182 <DeleteAction
159183 workspace={workspace}
167191 }
168192
169193 /**
194+ * Where the workspace's new repositories keep their git data. Shown only
195+ * once g1t has EU storage (or to a workspace that already chose it), so
196+ * nobody is offered a choice that does nothing.
197+ */
198+function ResidencySection({
199+ residency,
200+ euAvailable,
201+ saved,
202+ error,
203+}: {
204+ residency: DataResidency;
205+ euAvailable: boolean;
206+ saved: boolean;
207+ error?: string;
208+}) {
209+ const [choice, setChoice] = useState<DataResidency>(residency);
210+ const navigation = useNavigation();
211+ const saving = navigation.state !== "idle" && navigation.formData?.get("intent") === "residency";
212+ return (
213+ <section>
214+ <h2 className="font-medium">Data residency</h2>
215+ <p className="mt-1.5 text-xs text-faint">
216+ Where the git data of repositories made from now on is stored. Repositories the workspace already has stay
217+ where they are; ask support to move them. Issues, pull requests and settings are not affected.
218+ </p>
219+ <Form method="post" className="mt-5 space-y-4">
220+ <input type="hidden" name="intent" value="residency" />
221+ <RadioGroup
222+ name="residency"
223+ value={choice}
224+ onValueChange={(value) => setChoice(value as DataResidency)}
225+ aria-label="Where new repositories are stored"
226+ >
227+ <RadioOption value="anywhere" label="Anywhere" description="Wherever g1t stores repositories. The default." />
228+ <RadioOption
229+ value="eu"
230+ label="EU only"
231+ description={
232+ euAvailable
233+ ? "New repositories are stored in the EU, and are not made if EU storage cannot take them."
234+ : "EU storage cannot take new repositories right now."
235+ }
236+ disabled={!euAvailable && residency !== "eu"}
237+ />
238+ </RadioGroup>
239+ <ErrorText>{error}</ErrorText>
240+ {saved && !error && (
241+ <p role="status" className="text-xs text-muted">
242+ Saved.
243+ </p>
244+ )}
245+ <Button type="submit" disabled={saving || choice === residency}>
246+ Save
247+ </Button>
248+ </Form>
249+ </section>
250+ );
251+}
252+
253+/**
170254 * Deleting the workspace, with everything in it, in one step. Owners only,
171255 * as the whole page is; typed out to confirm. It is kept for
172256 * `WORKSPACE_RESTORE_DAYS`, when support can restore it. A protected
+39−0
317317 pub avatar: Option<String>,
318318 }
319319
320+/// Where a workspace keeps its repositories' git data: anywhere g1t
321+/// stores it (the default), or in the EU only. It applies to repositories
322+/// made after it is set; the repos service reads it when it places a new
323+/// one (`storage_options` says whether the EU can be chosen).
324+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
325+#[serde(rename_all = "lowercase")]
326+pub enum DataResidency {
327+ #[default]
328+ Anywhere,
329+ Eu,
330+}
331+
332+impl DataResidency {
333+ pub fn as_str(self) -> &'static str {
334+ match self {
335+ DataResidency::Anywhere => "anywhere",
336+ DataResidency::Eu => "eu",
337+ }
338+ }
339+
340+ pub fn parse(text: &str) -> Option<Self> {
341+ match text.trim().to_ascii_lowercase().as_str() {
342+ "anywhere" => Some(DataResidency::Anywhere),
343+ "eu" => Some(DataResidency::Eu),
344+ _ => None,
345+ }
346+ }
347+}
348+
349+/// `workspace_residency` takes [`SlugArgs`] and returns
350+/// `Option<DataResidency>` (null when there is no such workspace).
351+/// `set_workspace_residency`: owners only. Returns `Outcome<DataResidency>`.
352+#[derive(Debug, Serialize, Deserialize)]
353+pub struct SetResidencyArgs {
354+ pub actor: User,
355+ pub slug: String,
356+ pub residency: DataResidency,
357+}
358+
320359 /// `create_workspace`. Returns `Outcome<Workspace>`.
321360 #[derive(Debug, Serialize, Deserialize)]
322361 pub struct CreateWorkspaceArgs {
+11−0
4545 }
4646 }
4747
48+/// `storage_options` (no arguments, `{}`): what a workspace may choose
49+/// about where its repositories are kept. `eu_available`: an EU namespace
50+/// is configured and takes new repositories, so a workspace may keep its
51+/// data in the EU (`set_workspace_residency` on identity). Returns
52+/// `StorageOptions`.
53+#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
54+#[serde(rename_all = "camelCase")]
55+pub struct StorageOptions {
56+ pub eu_available: bool,
57+}
58+
4859 /// How long a deleted repository can be restored before it is purged.
4960 pub const RESTORE_DAYS: u64 = 30;
5061
+32−5
1212 // able to reach it: the API takes a shared secret, and git requests a
1313 // short-lived token the shim minted with the same secret.
1414 //
15+// A key is a repository's name (`acme--rocket`), or a namespace and a name
16+// (`g1t/acme--rocket`): hosted g1t's fallback store (docs/ARTIFACTS.md, R12)
17+// keeps each Artifacts namespace's repositories in a directory of their
18+// own, so a remote reads `<GITSTORE_URL>/git/<namespace>/<name>.git`, the
19+// shape Artifacts gives remotes.
20+//
21+// GITSTORE_READ_ONLY=1 refuses everything that writes: pushes, creating,
22+// forking, deleting, and minting write tokens. As a fallback the store
23+// serves reads until told otherwise; the repos service refuses writes too.
24+//
1525 // No dependencies beyond Node and git.
1626
1727 import { spawn } from "node:child_process";
2737 // How the repos service reaches this server; it becomes each repository's
2838 // `remote`, exactly as Artifacts hands one out.
2939 const PUBLIC_URL = (process.env.GITSTORE_URL ?? `http://localhost:${PORT}`).replace(/\/$/, "");
40+const READ_ONLY = ["1", "true", "yes"].includes(String(process.env.GITSTORE_READ_ONLY ?? "").toLowerCase());
3041
3142 /**
3243 * The secret shared with the Artifacts shim: GITSTORE_SECRET, or else the
5061 }
5162 mkdirSync(ROOT, { recursive: true });
5263
53−const KEY = /^[A-Za-z0-9_][A-Za-z0-9._-]{0,199}$/;
64+const NAME = /^[A-Za-z0-9_][A-Za-z0-9._-]{0,199}$/;
5465 const HASH = /^[0-9a-f]{40}$/;
5566
5667 class StoreError extends Error {
6172 }
6273 }
6374
75+/** Whether `key` is a name, or a namespace and a name. */
76+function validKey(key) {
77+ if (typeof key !== "string" || key.includes("..")) return false;
78+ const parts = key.split("/");
79+ return parts.length <= 2 && parts.every((part) => NAME.test(part));
80+}
81+
6482 function repoDir(key) {
65− if (!KEY.test(key) || key.includes("..")) {
83+ if (!validKey(key)) {
6684 throw new StoreError("INVALID_REPO_NAME", `invalid repository name: ${key}`);
6785 }
6886 return join(ROOT, `${key}.git`);
6987 }
7088
89+function refuseWrites(what) {
90+ if (READ_ONLY) throw new StoreError("READ_ONLY", `the git store is read-only: ${what} is refused`, 403);
91+}
92+
7193 function exists(key) {
7294 return existsSync(join(repoDir(key), "HEAD"));
7395 }
140162 }
141163
142164 async function create(key, { description, defaultBranch, readOnly, source } = {}) {
165+ refuseWrites("creating a repository");
143166 const dir = repoDir(key);
144167 if (exists(key)) throw new StoreError("ALREADY_EXISTS", `${key} already exists`, 409);
145168 mkdirSync(dir, { recursive: true });
164187 }
165188
166189 async function fork(key, target, { description, readOnly, defaultBranchOnly = true } = {}) {
190+ refuseWrites("forking");
167191 const source = requireRepo(key);
168192 const dir = repoDir(target);
169193 if (exists(target)) throw new StoreError("ALREADY_EXISTS", `${target} already exists`, 409);
298322 }
299323
300324 function mintToken(key, scope = "write", ttl = 86400) {
325+ if (scope === "write") refuseWrites("a write token");
301326 const seconds = Math.max(60, Math.min(Number(ttl) || 86400, 31536000));
302327 const expires = Math.floor(Date.now() / 1000) + seconds;
303328 const id = randomUUID();
343368 }
344369 const service = rest === "info/refs" ? new URLSearchParams(query).get("service") : rest;
345370 if (service === "git-receive-pack" && claims.s !== "write") return send(response, 403, "read-only token");
371+ if (service === "git-receive-pack" && READ_ONLY) return send(response, 403, "the git store is read-only");
346372 if (service !== "git-upload-pack" && service !== "git-receive-pack") return send(response, 404, "not found");
347373
348374 const env = {
445471 if (method === "GET" && !action) return send(response, 200, await info(key));
446472 // DELETE /api/repos/<key> delete (a purged repository)
447473 if (method === "DELETE" && !action) {
474+ refuseWrites("deleting a repository");
448475 if (!exists(key)) return send(response, 404, { code: "NOT_FOUND", message: "no such repository" });
449476 await rm(repoDir(key), { recursive: true, force: true });
450477 return send(response, 200, { deleted: true });
478505 const server = createServer(async (request, response) => {
479506 const url = new URL(request.url, "http://gitstore");
480507 try {
481− if (url.pathname === "/healthz") return send(response, 200, "ok");
482− const git = /^\/git\/([^/]+)\.git\/(info\/refs|git-upload-pack|git-receive-pack)$/.exec(url.pathname);
508+ if (url.pathname === "/healthz") return send(response, 200, READ_ONLY ? "ok read-only" : "ok");
509+ const git = /^\/git\/((?:[^/]+\/)?[^/]+)\.git\/(info\/refs|git-upload-pack|git-receive-pack)$/.exec(url.pathname);
483510 if (git) return smartHttp(request, response, decodeURIComponent(git[1]), git[2], url.search.slice(1));
484511 if (url.pathname === "/api/repos" || url.pathname.startsWith("/api/repos/")) {
485512 const parts = url.pathname.slice("/api/repos".length).split("/").filter(Boolean).map(decodeURIComponent);
496523 });
497524
498525 server.listen(PORT, () => {
499− console.log(`g1t gitstore: ${ROOT} on :${PORT} (remote ${PUBLIC_URL})`);
526+ console.log(`g1t gitstore: ${ROOT} on :${PORT} (remote ${PUBLIC_URL})${READ_ONLY ? ", read-only" : ""}`);
500527 });
501528
502529 for (const signal of ["SIGINT", "SIGTERM"]) {
+342−30
11 # Cloudflare Artifacts: due diligence for g1t at launch scale
22
33 Status: research document, 2026-10-06; R1–R5, R9, R10, R13 and R7 groundwork were built the same day (section 9).
4+R7 (sharding, moves, EU residency) and R12 (the fallback store) were built 2026-10-07, off until their
5+infrastructure exists ("What you must create", section 9).
46 Scope: everything g1t stores in Cloudflare Artifacts (open beta since 2026-10-01; billing from 2026-10-14),
57 measured against what Cloudflare documents, and what we must build so that a few thousand workspaces
68 can run on it.
2325 3. **One namespace carries everything.** All repositories and forks live in the `g1t` namespace. The
2426 control-plane limit is **2,000 requests per 10 seconds per namespace** (200 per second). If binding
2527 calls count against it, page views, token mints and mergeability checks together exceed it at launch
26− peaks.
28+ peaks. Sharding is now built (R7, section 9); the extra namespaces are not made yet.
2729 4. **Hard limits are not enforced in front of Artifacts.** 1 GB per repository, 32 MB per file, and a
2830 128 MB Worker isolate that buffers each push body twice. Large pushes and imports fail late, without a
2931 message git can show.
3032 5. **No backup, no exit drill.** Cloudflare replicates data, but there is no SLA, no documented export
3133 besides git itself, and the self-host git store is not a production fallback yet. Nightly bundles
32− to R2 and a restore drill are now built (R11, section 9); the fallback store is not (R12).
34+ to R2 and a restore drill are now built (R11, section 9), and so is the fallback path (R12): a
35+ restore into the git store and a switch by configuration. The host it runs on is not made yet.
3336
3437 None of these blocks an invite-only launch. Items 1 and 2 must be answered before billing starts on
3538 2026-10-14, and the fork cleanup must ship before agent pull requests reach thousands a day.
353356 `migrations/0011_artifacts_meters_forks_health.sql` (new columns on `repos`, new tables
354357 `artifacts_meters`, `operation_mapping`, `store_health`; additive, no backfill). R11 added
355358 `migrations/0013_backups.sql` (a new table, `repo_backups`, and one `operation_mapping` row;
356−additive).
359+additive). R7 added `migrations/0014_namespace_moves.sql` (two columns on `repos`,
360+`writes_paused_until` and `writes_paused_for`; new tables `repo_moves` and `repo_move_copies`;
361+additive) and, in identity, `services/identity/migrations/0026_workspace_residency.sql` (one column,
362+`workspaces.data_residency`; additive). R12 needs no migration.
357363
358364 | # | Status | What |
359365 | --- | --- | --- |
365371 | R5 | Built | `resilience.rs` sorts errors into rate limited, transient (`INTERNAL_ERROR`, `UPSTREAM_UNAVAILABLE`, `*_IN_PROGRESS`, no code, HTTP 5xx) and permanent. Binding reads, `get`, `info`, `createToken`, `create` and `delete` try up to 3 times with exponential backoff and jitter (80 ms base, 400 ms for rate limits, 2 s cap); `fork` and every receive-pack never retry. Git reads (`info/refs`, upload-pack) retry on 429 and 5xx. Per isolate, each namespace has a breaker that opens after 5 transient failures in a row, for 10 s, then lets one probe through. Busy answers reach git as 429 (rate limited) or 503, with `Retry-After: 5`; the site's read RPCs (`tree`, `blob`, `log`, `branches`, `blame`, `compare`) answer an `Outcome` failure saying the git storage is busy; other RPCs answer 503 with the same words. Health is counted by the minute (`store_health`) and served by the `store_health { minutes }` RPC; status.g1t.sh lists **Git storage** through a new `REPOS` service binding (down: 25% or more of at least 5 calls failed, or the breaker refused calls; degraded: rate limited, or a mean call over 1.5 s). |
366372 | R9 | Built | `log(branch)`, `branches()` and `read_file(ref, path)` are kept in the colo cache under the repository's `refs_version` (5 minutes at most, and only while `refs_cache::usable`), and by commit hash for good; a log by branch also fills the by-hash entry; `readCommit` (parents) is kept for good. Read RPCs open repositories through `read_git`, which sets the version. |
367373 | R10 | Built in repos; work unchanged | `divergence` works out the target's side once per target head per isolate (`coalesce.rs`: the head under the refs version, then the history by hash, kept 60 s), and what the target changed between two trees once per pair (10 minutes). `readCommit` and logs by hash come from the cache. Work's fan-out (`after_push`, up to 100 pull requests) is unchanged: its 100 `divergence` calls now cost one walk of the target instead of 100. |
368−| R7 | Groundwork | `shards.rs`: bindings named in `ARTIFACTS_NAMESPACES` (JSON, binding → namespace; `ARTIFACTS` → `g1t` always there), a repository's namespace kept in its `store` column as `<namespace>/<key>` (no prefix means the `ARTIFACTS` namespace, so every existing key reads the same), new repositories placed by `ARTIFACTS_NEW_REPOS` (comma-separated, spread by an FNV hash of the repository id; names not bound are skipped), forks always in their repository's namespace, `ARTIFACTS_EU_NAMESPACE` reserved for EU residency (no workspace setting yet). Works with only `ARTIFACTS` bound, as today. |
374+| R7 | Built; the namespaces are yours to make | `shards.rs`: bindings named in `ARTIFACTS_NAMESPACES` (JSON, binding → namespace; `ARTIFACTS` → `g1t` always there), a repository's namespace kept in its `store` column as `<namespace>/<key>` (no prefix means the `ARTIFACTS` namespace, so every existing key reads the same), forks always in their repository's namespace. **Placing** (`Placement::choose`, loads from `namespaces.rs`): among `ARTIFACTS_NEW_REPOS`, the healthy namespaces (bound, taking writes, not failing, under `ARTIFACTS_NAMESPACE_LIMITS`' `max_repos`, busiest minute under 70% of the 12,000-a-minute limit) within 100 repositories or 5% of the emptiest, spread by an FNV hash of the id; loads are read (one D1 query each for the registry and `store_health`, kept a minute) only when there is more than one to choose from. **Moving** (`moves.rs`): `move_repository` or `scripts/ops/artifacts-namespaces.mjs move` queues one; the hourly sweep pauses writes, copies every ref of the repository and its working copies over git (one streamed upload-pack into one receive-pack each), switches every `store` key in one batch, and deletes the old copies after 7 days. **EU residency**: identity's `workspaces.data_residency`, set by an owner in the workspace's settings (shown only once `storage_options` says an EU namespace takes repositories), read by the repos service at creation only while `ARTIFACTS_EU_NAMESPACE` is set; an EU workspace's repository goes to that namespace or is not made. **Health and limits**: `namespaces` RPC and `scripts/ops/artifacts-namespaces.mjs`. Nothing changes until bindings and variables name new namespaces. See "R7: sharding, moves and EU residency" below. |
369375 | R8 | Built | `crates/runner/src/clone.rs`: every sandbox clones at `--depth=1` (a full g1t clone took 5.4 s, depth 1 took 3.8 s). Work that merges (catch-up, the merge queue, merge checks, a review's diff) deepens 50, 500, then 5000 commits until the two sides share one, and fetches everything only as the last resort (`share_history`). `G1T_CLONE_DEPTH` (0 or `full` for everything) and `G1T_CLONE_FILTER=blob:none` change it per runner. |
370376 | R6 | Built; the bucket must exist before it deploys | `pack_cache.rs`: an upload-pack POST with wants and no `have` or `shallow` lines (a fresh clone, the sandboxes' `deepen 1` ones included), uncompressed and at most 1 MiB, is keyed `packs/<repo id>/<refs_version>/<sha256>` over the request normalized: protocol v2 capabilities without `agent=`/`session-id=` and its arguments, each sorted and deduplicated; v0/v1 wants sorted, the first want's capabilities split off, sorted and without `agent=`, then `deepen`/`filter` lines, a flush and `done`. Only while `refs_cache::usable` (the version known, and no push credential out of g1t's hands), so never across a refs change. Looked up after authorization, alongside the free-workspace limits and the kept refs answer; a hit streams from the bucket (`Server-Timing` `pack;desc=hit`). A miss streams the store's 200 to git through a tee that copies it to a fill in `ctx.wait_until` (at most 5 MiB queued between them, 2 fills per isolate, one per key): under 5 MiB it is one `put` once it all arrived; larger, 5 MiB multipart parts completed only after the last part and a check that it is one whole side-band pack (well-formed pkt-lines, `PACK` on channel 1, no `ERR` or channel 3, a closing flush). Over 200 MB, a queue that falls behind, git going away or the store's stream failing lets the fill go and aborts the upload; nothing partial can be read. Meters `pack_cache.hit` (with the bytes served) and `pack_cache.miss` (counted with `record`, bytes added at the end), neither an operation by default; a hit records no `git.fetch`. Storage is behind the `PackStore` port with an R2 adapter (`GIT_PACKS`, bucket `g1t-git-packs`, lifecycle: packs deleted after 7 days, unfinished uploads after 1); without the binding (self-hosted) nothing is kept. |
371377 | R11 | Built; not yet deployed | Nightly `git bundle` backups to the `g1t-backups` R2 bucket, and a restore drill. Migration `0013_backups.sql` (`repo_backups`, and an `operation_mapping` row). See "R11: backups and the restore drill" below. |
378+| R12 | Built; the host is yours to make | `scripts/ops/restore-to-gitstore.mjs` rebuilds every repository from its bundle chain into the git store (`deploy/self-host/gitstore`, now with namespaced keys, `<root>/<namespace>/<name>.git`, and `GITSTORE_READ_ONLY=1`), on the host or over its API, and later lists and reconciles what the fallback took. `fallback.rs`: with `GIT_FALLBACK_URL`, `GIT_FALLBACK_SECRET` and `GIT_FALLBACK_NAMESPACES` set, a namespace's `GitStore` calls go to the git store's API instead of the Artifacts binding (same metering, retries and breaker, its own health as `<namespace>@fallback`), read-only unless `GIT_FALLBACK_WRITES=allow`: writes are refused before they are asked, and say so in words; kept ref listings and packs are not used, nor backups cut. status.g1t.sh shows Git storage degraded meanwhile. See "R12: the fallback store and the outage runbook" below. |
372379
373380 ### R1: reading `scripts/ops/artifacts-usage.mjs`
374381
471478 an answer after the acknowledgments with a flush, and the client negotiates again. Report it to
472479 Cloudflare.
473480
474−### R7: making more namespaces (yours to run, when needed)
481+### R7: sharding, moves and EU residency
482+
483+Every piece is built and off. Production behaves exactly as before until the namespaces below are
484+made, bound and named: with only `ARTIFACTS` bound and `ARTIFACTS_NEW_REPOS` empty, new repositories
485+go to `g1t`, nothing extra is read from D1 when one is made, and identity is never asked about
486+residency.
475487
488+| Variable (`services/repos/wrangler.jsonc`) | What it does |
489+| --- | --- |
490+| `ARTIFACTS_NAMESPACES` | JSON, binding to namespace. `ARTIFACTS` is always there (`g1t` unless named). A name without a binding is logged and left out. |
491+| `ARTIFACTS_NEW_REPOS` | Comma-separated namespaces new repositories go to. Empty: `g1t`. A name that is not bound is passed over. |
492+| `ARTIFACTS_EU_NAMESPACE` | The namespace EU workspaces' new repositories go to. Unset: residency is never read, and the setting is never offered. |
493+| `ARTIFACTS_NAMESPACE_LIMITS` | Optional JSON, `{"g1t": {"max_repos": 50000}}`. A namespace at its limit takes no new repositories while another can. |
494+
495+**Placing a new repository** (`shards.rs` `Placement::choose`, loads from `namespaces.rs`). For a
496+workspace that keeps its data anywhere: among the namespaces in `ARTIFACTS_NEW_REPOS`, the healthy
497+ones (bound, taking writes, not failing, under `max_repos`, their busiest minute in the last hour
498+under 70% of the 12,000-a-minute control-plane limit), and of those the ones within 100 repositories
499+or 5% of the emptiest, spread by the id's FNV hash. If none is healthy, the usable ones the same way
500+(never a read-only one); if none is usable, `g1t`. Failing means this isolate's breaker is open, or a
501+quarter of at least 5 calls in the last 5 minutes failed (`store_health`). Loads cost two D1 queries
502+(the registry grouped by namespace, `store_health` for the last hour), kept a minute per isolate, and
503+are read only when more than one namespace could take the repository. A pull request's working copy
504+always goes where its repository is. For an EU workspace: `ARTIFACTS_EU_NAMESPACE` if it is bound and
505+takes writes; otherwise the repository is not made, and the person is told why (409, "This workspace
506+keeps its data in the EU, and EU storage cannot take new repositories right now."). It is never placed
507+elsewhere.
508+
509+**EU residency, as a workspace sees it.** Identity keeps `workspaces.data_residency` (NULL for
510+anywhere, `eu`), changed by an owner with `set_workspace_residency` and read with
511+`workspace_residency`; a change is audited as `workspace.residency_changed`. The workspace's
512+**Settings** page shows **Data residency** only when the repos service's `storage_options` says
513+`euAvailable` (an EU namespace is bound and takes writes), or when the workspace already chose the EU.
514+It applies to repositories made after it is saved, by any path that creates one (the site, the API,
515+push to create, imports). Existing repositories stay where they are until moved. A transfer keeps a
516+repository's store key, so a repository transferred into an EU workspace stays where it was: move it.
517+The guide is `apps/docs/src/content/docs/guides/workspaces.md`, "Data residency". Not in the public
518+API or MCP yet.
519+
520+**Moving a repository** (`moves.rs`, migration 0014). It keeps its id, path, rows and history; only
521+`store` changes.
522+
476523 ```sh
477−# A US shard, unrestricted like today's g1t, and an EU one.
478−curl -X POST "https://api.cloudflare.com/client/v4/accounts/1e6f2cffa3f445920836e8ebe446bb58/artifacts/namespaces" \
479− -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json" \
480− --data '{"namespace":"g1t-us-1"}'
481−curl -X POST "https://api.cloudflare.com/client/v4/accounts/1e6f2cffa3f445920836e8ebe446bb58/artifacts/namespaces" \
482− -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json" \
483− --data '{"namespace":"g1t-eu","jurisdiction":"eu"}'
524+node scripts/ops/artifacts-namespaces.mjs move acme/rocket g1t-us-1 # queues it (one INSERT)
525+node scripts/ops/artifacts-namespaces.mjs moves # queued, moving, moved, failed, cleaned, diverged
484526 ```
485527
486−Then in `services/repos/wrangler.jsonc`:
528+Services can queue one with the `move_repository { repo_id, namespace, requested_by? }` RPC, which
529+checks it first, and list them with `repository_moves { limit? }`. The hourly sweep (`23 * * * *`)
530+runs one queued move per hour:
531+
532+1. Writes to the repository and every working copy of its pull requests are paused
533+ (`writes_paused_until`, 20 minutes at most, so a move that dies releases them on its own). A push
534+ waits up to 20 seconds, polling every 2, and then is told: "acme/rocket is paused for maintenance
535+ (moving to g1t-us-1); changes to it wait a few minutes. Try again shortly." Merges, catch-ups,
536+ commits from the web, branch changes, mirror catch-ups, new pull request working copies and push
537+ credentials for sandboxes wait the same way. Removing a working copy waits for the next sweep.
538+2. Push credentials already handed out reach the store directly, so the move waits for
539+ `refs_open_until` to pass (up to 7 minutes in the run; longer goes back in the queue), then 5
540+ seconds for pushes in flight.
541+3. Each one is made in the new namespace under the same name, and every ref is copied with one
542+ upload-pack from the old copy streamed into one receive-pack to the new (`land.rs` `copy_refs`),
543+ never held in memory. Until both list the same refs and the old one did not move during the copy,
544+ only what changed is copied again, three rounds at most. Removed working copies have nothing to
545+ copy: their rows follow.
546+4. One D1 batch points every row at its new key, moves its `refs_version` (so no kept ref listing,
547+ pack or versioned read is used again), lifts the pause, and records each copy's refs in
548+ `repo_move_copies`.
549+5. After 7 days the sweep deletes each old copy whose refs still say what was copied. One that
550+ changed (a push that slipped past the pause and landed in the old copy) is kept, and the move is
551+ marked `diverged` with the keys; push its refs to the new copy by hand. While an old copy is kept,
552+ its name stays taken, so no new repository adopts it.
553+
554+A move that fails before step 4 deletes what it made in the new namespace, lifts the pause, and is
555+tried again in the next sweep, three times in all. The copy is metered like landing
556+(`internal.git.fetch` and `internal.git.receive_pack`, billable 1 each to the repository's workspace
557+by default), plus `binding.create`. A copy is bounded by the cron's wall time (15 minutes), which
558+streams well past the 1 GB repository limit. Not verified against Artifacts yet: run the first move on
559+a test repository and compare `git ls-remote` of both copies.
560+
561+**Health and limits.** `namespaces` (RPC) answers each bound namespace's repositories, working copies
562+and stored bytes from the registry; its busiest minute in the last hour against 12,000 a minute;
563+calls, errors and rate limits in the last hour; whether it is failing, on the fallback, writable,
564+default, EU, and takes new repositories; and its `max_repos`.
487565
488−```jsonc
489−"artifacts": [
490− { "binding": "ARTIFACTS", "namespace": "g1t" },
491− { "binding": "ARTIFACTS_1", "namespace": "g1t-us-1" },
492− { "binding": "ARTIFACTS_EU", "namespace": "g1t-eu" }
493−],
494−"vars": {
495− "ARTIFACTS_NAMESPACES": "{\"ARTIFACTS\":\"g1t\",\"ARTIFACTS_1\":\"g1t-us-1\",\"ARTIFACTS_EU\":\"g1t-eu\"}",
496− "ARTIFACTS_NEW_REPOS": "g1t,g1t-us-1", // new repositories spread over both
497− "ARTIFACTS_EU_NAMESPACE": "g1t-eu" // used once a workspace can choose the EU
498−}
566+```sh
567+node scripts/ops/artifacts-namespaces.mjs # a table, and a line for anything to act on (exit 1 then)
568+node scripts/ops/artifacts-namespaces.mjs --cloudflare # also Cloudflare's event counts and each namespace's jurisdiction
569+node scripts/ops/artifacts-namespaces.mjs --json
499570 ```
500571
501−Existing repositories stay where they are (`store` without a prefix). Deploy the binding before
502−naming its namespace in `ARTIFACTS_NEW_REPOS`; a name that is not bound is skipped, never used.
503−Moving an existing repository between namespaces is not built (a clone and push, then a `store`
504−update).
572+It reads `services/repos/wrangler.jsonc` for what is configured, so it says what the next deploy
573+will do: a namespace named but not bound or not made, an EU namespace made without the EU
574+jurisdiction, one past 70% of the limit or at its `max_repos`, one rate limited, one served from the
575+fallback.
576+
577+#### What you must create (R7)
505578
579+None of this is needed until one namespace is not enough, or an EU customer asks.
580+
581+1. Make the namespaces, with a token that can edit Artifacts. A namespace's jurisdiction is fixed when
582+ it is made, and is not part of the binding (Wrangler's schema has only `binding`, `namespace` and
583+ `remote`):
584+
585+ ```sh
586+ # A US shard, unrestricted like today's g1t, and an EU one.
587+ curl -X POST "https://api.cloudflare.com/client/v4/accounts/1e6f2cffa3f445920836e8ebe446bb58/artifacts/namespaces" \
588+ -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json" \
589+ --data '{"namespace":"g1t-us-1"}'
590+ curl -X POST "https://api.cloudflare.com/client/v4/accounts/1e6f2cffa3f445920836e8ebe446bb58/artifacts/namespaces" \
591+ -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H "Content-Type: application/json" \
592+ --data '{"namespace":"g1t-eu","jurisdiction":"eu"}'
593+ node scripts/ops/artifacts-namespaces.mjs --cloudflare # both listed, g1t-eu with jurisdiction eu
594+ ```
595+
596+2. Bind them, and deploy `g1t-repos` (migrations 0014 and identity's 0026 go first, as the deploy tool
597+ always does). Nothing is placed in them yet:
598+
599+ ```jsonc
600+ "artifacts": [
601+ { "binding": "ARTIFACTS", "namespace": "g1t" },
602+ { "binding": "ARTIFACTS_1", "namespace": "g1t-us-1" },
603+ { "binding": "ARTIFACTS_EU", "namespace": "g1t-eu" }
604+ ],
605+ "vars": {
606+ "ARTIFACTS_NAMESPACES": "{\"ARTIFACTS\":\"g1t\",\"ARTIFACTS_1\":\"g1t-us-1\",\"ARTIFACTS_EU\":\"g1t-eu\"}",
607+ "ARTIFACTS_NEW_REPOS": ""
608+ }
609+ ```
610+
611+3. Name them, and deploy again: `"ARTIFACTS_NEW_REPOS": "g1t,g1t-us-1"` spreads new repositories over
612+ both (the emptier first), and `"ARTIFACTS_EU_NAMESPACE": "g1t-eu"` puts **Data residency** in every
613+ workspace's settings. Optionally `"ARTIFACTS_NAMESPACE_LIMITS": "{\"g1t\":{\"max_repos\":50000}}"`.
614+4. Move a test repository there and back (`move`, then `moves` after the next :23), and compare both
615+ copies with `git ls-remote`.
616+
617+More shards later are the same steps (`g1t-us-2`, `ARTIFACTS_2`). Deploy a binding before naming its
618+namespace anywhere; a name that is not bound is passed over, never used.
619+
506620 ### R11: backups and the restore drill
507621
508622 Every repository whose refs moved is bundled once a night and kept outside the git store, so a
609723 them).
610724 5. Turn protection back on, and run the drill again: every ref now matches the live repository.
611725
612−When the repository is gone from the store itself (its key answers not found), there is no
613−operator call yet to make an empty repository under an existing row's key; that is part of R12.
726+When the repository is gone from the store itself (its key answers not found), there is still no
727+operator call to make an empty repository under an existing row's key in Artifacts. Until there is,
728+the fallback store can serve it from its backup (R12).
614729
615730 To deploy: make the bucket (`npx wrangler r2 bucket create g1t-backups`, a setup step of `repos`
616731 in `deploy/stack.jsonc`), then migration 0013, then `g1t-repos` (the `BACKUPS` binding and the
624739 clones the whole repository, so a night reads each changed repository in full from the store;
625740 incremental bundles save storage, not reads.
626741
742+### R12: the fallback store and the outage runbook
743+
744+A cold fallback, not a hot standby: when Artifacts is down for a namespace, g1t can serve that
745+namespace's repositories from the self-hosted git store (`deploy/self-host/gitstore`), rebuilt from the
746+nightly backups (R11). Reads work from the last backup; writes wait, unless you choose otherwise. It is
747+switched by configuration, per namespace, in seconds and without a build.
748+
749+| | Kept restored nightly (recommended) | Restored when needed |
750+| --- | --- | --- |
751+| Data served | As of the last backup: at most about a day old | The same |
752+| Time to switch | Minutes: a last restore pass, then one secret | Download plus restore: about an hour per 100 GB of bundles, 4 at a time |
753+| Cost | The host, always on | The host only while it is needed, if you make it then |
754+
755+**The pieces.**
756+
757+- `scripts/ops/restore-to-gitstore.mjs restore` rebuilds each repository from its chain as the restore
758+ drill does (each bundle checked against its size and SHA-256, `git bundle verify`, fetched in order,
759+ refs set to the last entry's, `git fsck --connectivity-only`) into `<root>/<namespace>/<name>.git`,
760+ configured as the git store configures its own, with a `g1t.json` that records what it was restored
761+ from. A second run skips repositories already restored from the same last backup, so a nightly run
762+ only does what changed. `--into <root>` writes on the host; `--gitstore <url>` (with `GITSTORE_SECRET`)
763+ goes through the store's API and git, for a store that is not read-only. `--bundles <dir>` reads a
764+ local copy of the bucket; `--offline` takes the repositories from the manifests in it, so the host
765+ needs no database access (a repository moved between namespaces since its last backup is restored
766+ under its old namespace until the next backup records the new one).
767+- The git store (`deploy/self-host/gitstore/server.mjs`) now takes namespaced keys
768+ (`g1t-us-1/acme--rocket`, served at `/git/g1t-us-1/acme--rocket.git`, the shape Artifacts gives
769+ remotes) beside plain ones, and `GITSTORE_READ_ONLY=1` refuses pushes, write tokens, and making,
770+ forking or deleting repositories.
771+- `services/repos/src/fallback.rs` and `store.rs`: for each namespace in `GIT_FALLBACK_NAMESPACES`
772+ (`*` for all), the `GitStore` sends every call it would make on the Artifacts binding (`create`,
773+ `get`, `delete`, `info`, `createToken`, `log`, `readCommit`, `readTree`, `readBlob`, `readFile`,
774+ `fork`) to the store's API at `GIT_FALLBACK_URL` with `GIT_FALLBACK_SECRET`, and git's smart HTTP to
775+ its remotes. Calls keep their retries and breaker, counted as `<namespace>@fallback` in
776+ `store_health`, never as Artifacts' own health, and are not metered as binding calls (git requests
777+ still are). Credentials are kept apart from Artifacts' (`fallback:<key>`). Answers kept under a refs
778+ version (ref listings, packs, logs and files by branch) are neither used nor kept, since the
779+ fallback may be behind them; objects named by their hash are.
780+- Read-only, the default (`GIT_FALLBACK_WRITES` unset or `refuse`): a write is refused before it is
781+ asked. Git hears 503 with `Retry-After: 300` and "g1t's git storage is read-only while it
782+ recovers: clones, fetches and pages work, and pushes, merges and new repositories wait until it is
783+ back."; the site's reads work, and its writes fail with the same words. New repositories are placed
784+ in a namespace that still takes writes, if `ARTIFACTS_NEW_REPOS` has one. Backups are not cut from a
785+ switched namespace (they would record an older state). status.g1t.sh shows **Git storage**
786+ degraded: "Served from the backup store: reads work, pushes and merges wait".
787+
788+**What does not work while switched.** Working copies of open pull requests are not backed up, so
789+their changes and branches do not read; working copies already removed read from their repository's
790+`refs/pull/<id>/head` as usual. Anything pushed after the last backup is not there until Artifacts is
791+back. Agent runs that only read work (their sandboxes clone from the fallback through handed-out
792+credentials); runs that push wait. Mirror catch-ups wait; pushes out to mirrors work.
793+
794+#### The host
795+
796+| Need | Why | What |
797+| --- | --- | --- |
798+| Outside Cloudflare | It is the exit if Artifacts, or the account, is the problem | A VM with a provider of your choice |
799+| Persistent disk | Repositories and a copy of the bucket live there; Containers' disk is ephemeral and at most 20 GB | Block storage or a local SSD that survives reboots |
800+| Disk size | Restored repositories about equal the newest full bundles; the bucket's copy holds up to two chains | 2.5 times the bucket's size; today 100 GB is ample, at 3,000 workspaces (about 250 GB stored, section 5) 1 TB |
801+| Near the repos Worker | Every read crosses to it; `g1t-repos` runs near D1 in WNAM | US West, for example Oregon. The EU namespace's fallback on a second, EU host, so EU data stays in the EU |
802+| HTTPS on a public name | Workers reach it with `fetch` | Caddy in front of port 8080, a name such as `fallback-git.g1t.sh` |
803+| Software | | Docker (the git store's image), or Node 24 and git; `rclone` |
804+| CPU and memory | `git upload-pack` for clones, restores 4 at a time | 4 vCPU, 8 GB |
805+
806+**What it costs**, at list prices for such a host (check before buying): a 4 vCPU, 8 GB VM is about
807+$15 to $50 a month depending on the provider; block storage $0.04 to $0.10 per GB-month, so $4 to $10
808+a month for 100 GB today and $40 to $100 for 1 TB at launch scale. Reading the bucket costs nothing in
809+egress (R2 charges none) and a few cents a month in R2 operations for a nightly `rclone sync`. In all,
810+about $20 to $60 a month now and $60 to $150 at 3,000 workspaces, per host; the EU host only once there
811+is an EU namespace.
812+
813+#### What you must create (R12)
814+
815+1. The host above, with a DNS name and TLS.
816+2. An R2 API token that can only read `g1t-backups` (Cloudflare dashboard, R2, Manage API tokens:
817+ Object Read, that bucket), for `rclone` on the host:
818+
819+ ```ini
820+ # ~/.config/rclone/rclone.conf on the host
821+ [r2]
822+ type = s3
823+ provider = Cloudflare
824+ access_key_id = <the token's access key id>
825+ secret_access_key = <its secret>
826+ endpoint = https://1e6f2cffa3f445920836e8ebe446bb58.r2.cloudflarestorage.com
827+ ```
828+
829+3. The git store and its secret, read-only from the start:
830+
831+ ```sh
832+ git clone https://g1t.sh/flagon-io/g1t.git /opt/g1t # node, git and rclone installed
833+ openssl rand -hex 32 > /srv/gitstore.secret
834+ GITSTORE_ROOT=/srv/gitstore GITSTORE_PORT=8080 GITSTORE_URL=https://fallback-git.g1t.sh \
835+ GITSTORE_SECRET="$(cat /srv/gitstore.secret)" GITSTORE_READ_ONLY=1 \
836+ node /opt/g1t/deploy/self-host/gitstore/server.mjs # as a systemd unit, or the image in deploy/self-host/gitstore
837+ # Caddyfile: fallback-git.g1t.sh { reverse_proxy 127.0.0.1:8080 }
838+ ```
839+
840+4. The first restore, then every night after the backups (02:53 UTC) have run, say at 07:00 UTC:
841+
842+ ```sh
843+ rclone sync r2:g1t-backups /srv/backups
844+ node /opt/g1t/scripts/ops/restore-to-gitstore.mjs restore --into /srv/gitstore --bundles /srv/backups --offline --jobs 4
845+ ```
846+
847+5. Tell `g1t-repos` where it is, ahead of time. These change nothing until a namespace is named:
848+
849+ ```sh
850+ cd services/repos
851+ echo https://fallback-git.g1t.sh | npx wrangler secret put GIT_FALLBACK_URL
852+ npx wrangler secret put GIT_FALLBACK_SECRET # paste /srv/gitstore.secret
853+ ```
854+
855+6. Drill it once a quarter on a namespace that holds only test repositories (make `g1t-drill` as in
856+ R7, bind it, move a test repository there), with the runbook below.
857+
858+#### Runbook: an Artifacts outage
859+
860+**Detect.**
861+
862+1. status.g1t.sh shows **Git storage** down, or `node scripts/ops/artifacts-namespaces.mjs` shows a
863+ namespace failing (errors, `rejected` calls from an open breaker). `npx wrangler tail g1t-repos`
864+ shows `git store <namespace>: ... failed`.
865+2. Check Cloudflare's status page and the Artifacts metrics (`serverError`, `rateLimited`).
866+3. Switch when it has lasted 15 minutes with no sign of ending, or at once if Cloudflare says it will
867+ be long. A short blip needs nothing: retries and the breaker already answer git with 503 and
868+ `Retry-After`.
869+
870+**Switch.**
871+
872+1. On the host: `curl -s https://fallback-git.g1t.sh/healthz` answers `ok read-only`. If the nightly
873+ restore did not run today, run step 4 of the setup; a run over a restored store only does what
874+ changed.
875+2. Switch the failing namespace (or `*`). A secret takes effect in seconds, with no build:
876+
877+ ```sh
878+ cd services/repos
879+ echo g1t | npx wrangler secret put GIT_FALLBACK_NAMESPACES
880+ ```
881+
882+3. Check: `git ls-remote https://g1t.sh/flagon-io/hello.git` answers; a repository page loads;
883+ status.g1t.sh shows Git storage degraded; `artifacts-namespaces.mjs` lists `@fallback` calls.
884+4. Open an incident on status.g1t.sh: reads work from last night's backup; pushes, merges and new
885+ repositories wait.
886+
887+**Serve reads.** Nothing more to do. Watch the host's disk and load; `upload-pack` is the work.
888+
889+**Take writes, only if the outage will be long.** Everything pushed then must be sent back
890+afterwards, and anything pushed to Artifacts after the last backup will conflict with it.
891+
892+```sh
893+# On the host: restart the git store without GITSTORE_READ_ONLY. Then:
894+echo allow | npx wrangler secret put GIT_FALLBACK_WRITES
895+```
896+
897+**Switch back**, once Artifacts answers again (`artifacts-namespaces.mjs` shows no errors from it;
898+the namespace's own calls are none while switched, so check Cloudflare's status and the metrics):
899+
900+1. If writes were taken: stop them first (`echo refuse | npx wrangler secret put GIT_FALLBACK_WRITES`,
901+ and restart the git store with `GITSTORE_READ_ONLY=1`), then list and send back what came in, while
902+ the namespace is still switched, so nothing else writes to Artifacts meanwhile:
903+
904+ ```sh
905+ node scripts/ops/restore-to-gitstore.mjs changed --into /srv/gitstore
906+ CLOUDFLARE_API_TOKEN=<Artifacts edit, D1 edit> node scripts/ops/restore-to-gitstore.mjs reconcile --into /srv/gitstore
907+ ```
908+
909+ A ref Artifacts still has as it was backed up takes the fallback's value (leased on that value, so
910+ nothing newer is overwritten). One that moved on both sides keeps Artifacts' value, and the
911+ fallback's goes beside it as `refs/fallback/<rest of the name>` (exit 3 says some did): tell the
912+ repository's owners to merge it. Each reconciled repository's `refs_version` is moved. To push,
913+ `reconcile` mints a write token with Cloudflare's REST API
914+ (`POST /accounts/<id>/artifacts/namespaces/<ns>/repos/<name>/tokens`, taken to mirror the binding's
915+ `createToken`); that endpoint is not verified yet, so run `reconcile` on one repository in the
916+ drill before relying on it.
917+2. Switch back: `npx wrangler secret delete GIT_FALLBACK_NAMESPACES` (and `GIT_FALLBACK_WRITES`).
918+3. Check as in "Switch", step 3: Git storage is no longer degraded once the fallback's calls age out
919+ of the five-minute window.
920+4. Backups resume the next night. Close the incident.
921+
922+Tested by `npm run test:ops` (`scripts/ops/restore-to-gitstore.test.mjs`): bundles cut as the runner
923+cuts them are restored into a store's root, served read-only by the git store itself (a namespaced
924+remote, a read token, a clone; write tokens and new repositories refused), pushed to, listed by
925+`changed`, reconciled into a live copy, and reconciled again after the live copy moved too; and a
926+restore through the store's API. `cargo test` covers the routing, answers, read-only refusals and
927+kept credentials (`fallback.rs`, `store.rs`, `resilience.rs`). Not yet run against production: the
928+switch itself, which wants the host.
929+
627930 ### Deploy order and what to watch
628931
932+R7 and R12 (2026-10-07): migrations `repos/0014` and `identity/0026` first (the registry reads
933+`writes_paused_until`, and `claim_store_key` reads `repo_move_copies`, so the new repos code must not
934+run before 0014); then `g1t-repos`, `g1t-identity`, `g1t-web` and `g1t-status`. No new bindings or
935+variables: with today's configuration nothing is placed, moved, offered or switched. Watch that
936+repository creation still answers as fast (it reads nothing new), and that `repository_moves` stays
937+empty.
938+
939+For 2026-10-06's work:
940+
629941 1. Migration 0011 (the deploy tool applies migrations first). `forks_of` reads `retired_at`, so
630942 the new code must not run before it.
631943 2. `g1t-repos` (new vars in `wrangler.jsonc`; no new bindings).
+14−4
260260 Hosted is untouched: `ArtifactsStore` is still the only adapter compiled
261261 into `services/repos`. Self-hosted, the `ARTIFACTS` binding is a service
262262 binding to `deploy/self-host/workers/artifacts`, which offers Artifacts'
263−methods and calls the git store. Long term, a `LocalGitStore` adapter in
264−Rust should call the git store's API directly, which removes the shim. The
265−git store can later gain `git gc` scheduling and object-store-backed packs
266−for large installations.
263+methods and calls the git store. The git store can later gain `git gc`
264+scheduling and object-store-backed packs for large installations.
265+
266+The same git store is hosted g1t's cold fallback for an Artifacts outage
267+(docs/ARTIFACTS.md, R12). For that it takes namespaced keys as well as
268+plain ones (`g1t-us-1/acme--rocket`, kept at
269+`<GITSTORE_ROOT>/g1t-us-1/acme--rocket.git` and served at
270+`/git/g1t-us-1/acme--rocket.git`, the shape Artifacts gives remotes), and
271+`GITSTORE_READ_ONLY=1` refuses pushes, write tokens and making, forking or
272+deleting repositories. `services/repos/src/fallback.rs` calls its API from
273+Rust, the start of the `LocalGitStore` adapter: a namespace named in
274+`GIT_FALLBACK_NAMESPACES` is served from it, with the shim out of the path.
275+Self-hosted installations keep using the shim, with plain keys; nothing
276+changes for them.
267277
268278 ### Search and context
269279
+3−0
6565 call("update_oauth_grant", { user, id, scopes: grant.scopes }),
6666 createWorkspace: (user, slug, name) => call("create_workspace", { user, slug, name }),
6767 getWorkspace: (slug) => call("get_workspace", { slug }),
68+ workspaceResidency: (slug) => call("workspace_residency", { slug }),
69+ setWorkspaceResidency: (actor, slug, residency) => call("set_workspace_residency", { actor, slug, residency }),
6870 listMembers: (slug, viewer) => call("list_members", { slug, viewer }),
6971 addMember: (actor, slug, username) => call("add_member", { actor, slug, username }),
7072 removeMember: (actor, slug, username) =>
273275 renameBranch: (actor, path, from, to) => call("rename_branch", { actor, path, from, to }),
274276 resolveBranch: (repoId, branch) => call("resolve_branch", { repoId, branch }),
275277 statusById: (id) => call("status_by_id", { id }),
278+ storageOptions: () => call("storage_options", {}),
276279 resolvePath: (path) => call("resolve_path", { path }),
277280 tree: (path, viewer, ref, treePath) =>
278281 call("tree", { path, viewer, ref, treePath }),
+14−0
7575 export const MAX_AVATAR_BYTES = 1024 * 1024;
7676
7777 /**
78+ * Where a workspace keeps its repositories' git data: anywhere g1t stores
79+ * it (the default), or in the EU only. It applies to repositories made
80+ * after it is set.
81+ */
82+export type DataResidency = "anywhere" | "eu";
83+
84+/**
7885 * A workspace: the owner of repositories, and the first segment of their
7986 * URLs. A person's own space and a team's are the same thing.
8087 */
511518 createWorkspace(user: User, slug: string, name: string): Promise<Result<Workspace>>;
512519 /** Public details of a workspace, or null. */
513520 getWorkspace(slug: string): Promise<Workspace | null>;
521+ /** Where a workspace keeps its repositories' git data; null when there is no such workspace. */
522+ workspaceResidency(slug: string): Promise<DataResidency | null>;
523+ /**
524+ * Owners only. Applies to repositories made from then on. Offer `eu`
525+ * only when the repos service's `storageOptions()` says it is available.
526+ */
527+ setWorkspaceResidency(actor: User, slug: string, residency: DataResidency): Promise<Result<DataResidency>>;
514528 /** Members only. */
515529 listMembers(slug: string, viewer: Viewer): Promise<Result<Member[]>>;
516530 /** Owners only. */
+8−0
114114 importUrl?: string;
115115 };
116116
117+/**
118+ * Where repositories may be kept. `euAvailable`: an EU namespace takes new
119+ * repositories, so a workspace may keep its data in the EU.
120+ */
121+export type StorageOptions = { euAvailable: boolean };
122+
117123 /** Repositories: metadata, contents and git access. */
118124 export interface ReposApi {
119125 get(path: RepoPath, viewer: Viewer): Promise<Result<Repo>>;
197203 resolveBranch(repoId: string, branch: string): Promise<string | null>;
198204 /** Whether a repository is archived or deleted; an unknown id answers as deleted. */
199205 statusById(id: string): Promise<RepoStatus>;
206+ /** What a workspace may choose about where its repositories are kept. */
207+ storageOptions(): Promise<StorageOptions>;
200208 /**
201209 * Moves the repository to the workspace `to`, keeping its name, id and
202210 * everything under it. The actor must own both workspaces. The old path
+261−0
1+#!/usr/bin/env node
2+// How each git store namespace stands (docs/ARTIFACTS.md, R7): what it
3+// holds, how busy its busiest minute was against Cloudflare's limit of
4+// 2,000 control-plane requests per 10 seconds, how it has been failing,
5+// whether it takes new repositories, and its limits. Also queues and lists
6+// moves of repositories between namespaces (services/repos/src/moves.rs).
7+//
8+// node scripts/ops/artifacts-namespaces.mjs # the report, as a table
9+// node scripts/ops/artifacts-namespaces.mjs --json # the same, as JSON
10+// node scripts/ops/artifacts-namespaces.mjs --cloudflare # with Cloudflare's own event counts per namespace
11+// node scripts/ops/artifacts-namespaces.mjs moves # moves asked for, newest first
12+// node scripts/ops/artifacts-namespaces.mjs move acme/rocket g1t-us-1 # queue one; the hourly sweep runs it
13+//
14+// The report and `moves` are read-only: SELECTs against the g1t-repos
15+// database through Wrangler (as you are logged in, or CLOUDFLARE_D1_TOKEN),
16+// and with --cloudflare one GraphQL query (CLOUDFLARE_API_TOKEN with Account
17+// Analytics: Read). `move` inserts one row into repo_moves, nothing else.
18+// What is configured is read from services/repos/wrangler.jsonc, so the
19+// report says what the next deploy will do.
20+
21+import { readFileSync } from "node:fs";
22+import { join } from "node:path";
23+
24+import { ACCOUNT_ID, cloudflareAuth, exec, jsonFrom, wranglerEnv } from "../deploy/cloudflare.mjs";
25+import { ROOT, parseJsonc } from "../deploy/stack.mjs";
26+
27+const WRANGLER = join(ROOT, "node_modules/wrangler/bin/wrangler.js");
28+const DATABASE = "g1t-repos";
29+/** Cloudflare's control-plane limit for one namespace, per minute (shards.rs). */
30+export const LIMIT_PER_MINUTE = 12_000;
31+/** Past this share of it, the repos service stops placing new repositories there. */
32+export const HOT_SHARE = 0.7;
33+
34+/** What services/repos/wrangler.jsonc configures: each namespace, its binding and jurisdiction, and the placement variables. */
35+export function configured(wrangler) {
36+ const vars = wrangler.vars ?? {};
37+ let named = {};
38+ try {
39+ named = JSON.parse(vars.ARTIFACTS_NAMESPACES ?? "{}");
40+ } catch {}
41+ const bindings = new Map((wrangler.artifacts ?? []).map((one) => [one.binding, one]));
42+ if (!named.ARTIFACTS) named.ARTIFACTS = "g1t";
43+ let limits = {};
44+ try {
45+ limits = JSON.parse(vars.ARTIFACTS_NAMESPACE_LIMITS ?? "{}");
46+ } catch {}
47+ const newRepos = String(vars.ARTIFACTS_NEW_REPOS ?? "")
48+ .split(",")
49+ .map((name) => name.trim())
50+ .filter(Boolean);
51+ const eu = vars.ARTIFACTS_EU_NAMESPACE?.trim() || null;
52+ return Object.entries(named).map(([binding, namespace]) => ({
53+ namespace,
54+ binding,
55+ bound: bindings.has(binding) && bindings.get(binding).namespace === namespace,
56+ // Set when the namespace is made, not in the binding: known with --cloudflare.
57+ jurisdiction: null,
58+ default: binding === "ARTIFACTS",
59+ eu: eu === namespace,
60+ takes_new_repos: newRepos.includes(namespace),
61+ max_repos: limits[namespace]?.max_repos ?? null,
62+ }));
63+}
64+
65+/** A store key's namespace, as the registry keeps it: none means the default. */
66+export function namespaceOf(store, defaultNamespace = "g1t") {
67+ const at = (store ?? "").indexOf("/");
68+ return at > 0 ? store.slice(0, at) : defaultNamespace;
69+}
70+
71+/**
72+ * Every namespace's standing, from what is configured, what the registry
73+ * holds (`held`: ns, repos, forks, stored_bytes), how it answered
74+ * (`health`: store, peak, calls, errors, rate_limited, rejected, the last
75+ * hour and the last day), Cloudflare's own counts (`events`) and the
76+ * namespaces Cloudflare has (`made`: namespace, jurisdiction), when asked.
77+ */
78+export function standings(config, held, health, events = [], made = null) {
79+ const defaultNamespace = config.find((one) => one.default)?.namespace ?? "g1t";
80+ const names = [...new Set([...config.map((one) => one.namespace), ...held.map((row) => row.ns || defaultNamespace)])];
81+ return names.map((namespace) => {
82+ const known = made?.find((one) => one.namespace === namespace);
83+ const set = { ...(config.find((one) => one.namespace === namespace) ?? { namespace, binding: null, bound: false }) };
84+ if (known) set.jurisdiction = known.jurisdiction ?? "any";
85+ const holds = held.filter((row) => (row.ns || defaultNamespace) === namespace);
86+ const sum = (rows, field) => rows.reduce((total, row) => total + Number(row[field] ?? 0), 0);
87+ const hour = health.find((row) => row.store === namespace && row.window === "hour") ?? {};
88+ const day = health.find((row) => row.store === namespace && row.window === "day") ?? {};
89+ const fallback = health.find((row) => row.store === `${namespace}@fallback` && row.window === "hour");
90+ const peak = Number(day.peak ?? 0);
91+ const repos = sum(holds, "repos");
92+ const warnings = [];
93+ if (!set.bound && repos > 0) warnings.push("holds repositories but is not bound");
94+ if (set.takes_new_repos && !set.bound) warnings.push("named in ARTIFACTS_NEW_REPOS but not bound: passed over");
95+ if (peak >= LIMIT_PER_MINUTE * HOT_SHARE) warnings.push(`busiest minute at ${Math.round((peak / LIMIT_PER_MINUTE) * 100)}% of the limit`);
96+ if (set.max_repos && repos >= set.max_repos) warnings.push("at its max_repos: takes no new repositories while another can");
97+ if (Number(hour.rate_limited ?? 0) > 0) warnings.push(`${hour.rate_limited} calls rate limited in the last hour`);
98+ if (made && set.binding && !known) warnings.push("named in ARTIFACTS_NAMESPACES, but Cloudflare has no namespace of this name: make it before deploying");
99+ if (set.eu && known && known.jurisdiction !== "eu") warnings.push(`named as the EU namespace, but Cloudflare says its jurisdiction is ${known.jurisdiction ?? "unrestricted"}`);
100+ if (fallback) warnings.push(`served from the fallback store lately (${fallback.calls} calls in the last hour)`);
101+ return {
102+ ...set,
103+ repos,
104+ forks: sum(holds, "forks"),
105+ stored_bytes: sum(holds, "stored_bytes"),
106+ peak_per_minute_day: peak,
107+ peak_per_minute_hour: Number(hour.peak ?? 0),
108+ peak_share: peak / LIMIT_PER_MINUTE,
109+ calls_day: Number(day.calls ?? 0),
110+ errors_day: Number(day.errors ?? 0),
111+ rate_limited_day: Number(day.rate_limited ?? 0),
112+ rejected_day: Number(day.rejected ?? 0),
113+ cloudflare_events: events.filter((event) => event.namespace === namespace).reduce((total, event) => total + event.count, 0),
114+ warnings,
115+ };
116+ });
117+}
118+
119+const gb = (bytes) => `${(bytes / 1e9).toFixed(2)} GB`;
120+const pct = (share) => `${(share * 100).toFixed(1)}%`;
121+
122+export function table(rows) {
123+ const header = ["namespace", "binding", "where", "new", "repos", "forks", "stored", "peak/min (24h)", "of limit", "calls 24h", "errors", "429s"];
124+ const lines = rows.map((row) => [
125+ row.namespace + (row.default ? " *" : ""),
126+ row.bound ? row.binding : `${row.binding ?? "-"} (not bound)`,
127+ row.jurisdiction ?? "?",
128+ row.takes_new_repos ? "yes" : row.eu ? "eu" : "no",
129+ String(row.repos),
130+ String(row.forks),
131+ gb(row.stored_bytes),
132+ String(row.peak_per_minute_day),
133+ pct(row.peak_share),
134+ String(row.calls_day),
135+ String(row.errors_day),
136+ String(row.rate_limited_day),
137+ ]);
138+ const widths = header.map((title, at) => Math.max(title.length, ...lines.map((line) => line[at].length)));
139+ const format = (line) => line.map((cell, at) => cell.padEnd(widths[at])).join(" ");
140+ const out = [format(header), format(widths.map((width) => "-".repeat(width))), ...lines.map(format)];
141+ for (const row of rows) for (const warning of row.warnings) out.push(`! ${row.namespace}: ${warning}`);
142+ out.push("* the default namespace: keys without a namespace are in it. Limit: 12,000 control-plane requests a minute per namespace.");
143+ return out.join("\n");
144+}
145+
146+// ---------------------------------------------------------------------
147+
148+async function d1(sql) {
149+ const env = { ...wranglerEnv({ ...process.env, CI: "true" }) };
150+ if (process.env.CLOUDFLARE_D1_TOKEN) env.CLOUDFLARE_API_TOKEN = process.env.CLOUDFLARE_D1_TOKEN;
151+ const { code, out } = await exec(process.execPath, [WRANGLER, "d1", "execute", DATABASE, "--remote", "--json", "--command", sql], {
152+ cwd: join(ROOT, "services/repos"),
153+ env,
154+ });
155+ if (code !== 0) throw new Error(out.slice(-600));
156+ return jsonFrom(out)[0]?.results ?? [];
157+}
158+
159+const quoted = (text) => `'${String(text).replaceAll("'", "''")}'`;
160+const minuteAgo = (minutes) => new Date(Date.now() - minutes * 60_000).toISOString().slice(0, 16);
161+
162+async function readHeld() {
163+ return d1(`SELECT CASE WHEN instr(coalesce(store, ''), '/') > 0 THEN substr(store, 1, instr(store, '/') - 1) ELSE '' END AS ns,
164+ count(*) AS repos, sum(CASE WHEN fork_of IS NULL THEN 0 ELSE 1 END) AS forks, sum(coalesce(stored_bytes, 0)) AS stored_bytes
165+ FROM repos WHERE deleted_at IS NULL AND retired_at IS NULL GROUP BY ns`);
166+}
167+
168+async function readHealth() {
169+ const window = (name, minutes) =>
170+ `SELECT '${name}' AS window, store, max(calls) AS peak, sum(calls) AS calls, sum(errors) AS errors,
171+ sum(rate_limited) AS rate_limited, sum(rejected) AS rejected
172+ FROM store_health WHERE minute >= '${minuteAgo(minutes)}' GROUP BY store`;
173+ return d1(`${window("hour", 60)} UNION ALL ${window("day", 24 * 60)}`);
174+}
175+
176+async function readEvents() {
177+ const auth = cloudflareAuth();
178+ if (!auth) throw new Error("--cloudflare needs CLOUDFLARE_API_TOKEN with Account Analytics: Read");
179+ const end = new Date();
180+ const start = new Date(end.getTime() - 24 * 3600 * 1000);
181+ const query = `query Q($accountTag: String!, $start: Time!, $end: Time!) { viewer { accounts(filter: { accountTag: $accountTag }) {
182+ artifactsEventsAdaptiveGroups(limit: 10000, filter: { datetime_geq: $start, datetime_leq: $end }) { count dimensions { repositoryNamespace } } } } }`;
183+ const response = await fetch("https://api.cloudflare.com/client/v4/graphql", {
184+ method: "POST",
185+ headers: { ...auth, "content-type": "application/json" },
186+ body: JSON.stringify({ query, variables: { accountTag: ACCOUNT_ID, start: start.toISOString(), end: end.toISOString() } }),
187+ });
188+ const body = await response.json();
189+ if (body.errors?.length) throw new Error(`GraphQL: ${JSON.stringify(body.errors).slice(0, 400)}`);
190+ return (body.data?.viewer?.accounts?.[0]?.artifactsEventsAdaptiveGroups ?? []).map((group) => ({
191+ namespace: group.dimensions.repositoryNamespace,
192+ count: group.count,
193+ }));
194+}
195+
196+/** The namespaces Cloudflare has, with their jurisdictions (CLOUDFLARE_API_TOKEN with Artifacts: Read). */
197+async function readNamespaces() {
198+ const auth = cloudflareAuth();
199+ if (!auth) throw new Error("--cloudflare needs CLOUDFLARE_API_TOKEN");
200+ const response = await fetch(`https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/artifacts/namespaces`, { headers: auth });
201+ const body = await response.json().catch(() => ({}));
202+ if (!response.ok || body.success === false) throw new Error(`listing namespaces: ${response.status} ${JSON.stringify(body.errors ?? body).slice(0, 300)}`);
203+ const list = Array.isArray(body.result) ? body.result : (body.result?.namespaces ?? []);
204+ return list.map((one) => ({ namespace: one.namespace ?? one.name, jurisdiction: one.jurisdiction ?? null }));
205+}
206+
207+function wranglerConfig() {
208+ return parseJsonc(readFileSync(join(ROOT, "services/repos/wrangler.jsonc"), "utf8"));
209+}
210+
211+async function main() {
212+ const args = process.argv.slice(2);
213+ const command = args[0] && !args[0].startsWith("--") ? args[0] : "report";
214+
215+ if (command === "moves") {
216+ const rows = await d1(`SELECT m.*, r.namespace AS workspace, r.name FROM repo_moves m LEFT JOIN repos r ON r.id = m.repo_id
217+ ORDER BY m.queued_ms DESC LIMIT 50`);
218+ if (args.includes("--json")) console.log(JSON.stringify(rows, null, 2));
219+ else for (const row of rows) console.log(`${row.id} ${row.status.padEnd(8)} ${row.workspace}/${row.name} -> ${row.to_namespace}${row.note ? ` (${row.note})` : ""}`);
220+ return 0;
221+ }
222+
223+ if (command === "move") {
224+ const [path, namespace] = args.slice(1);
225+ const [workspace, name] = String(path ?? "").toLowerCase().split("/");
226+ if (!workspace || !name || !namespace) throw new Error("usage: move <workspace/name> <namespace>");
227+ const config = configured(wranglerConfig());
228+ if (!config.some((one) => one.namespace === namespace && one.bound)) throw new Error(`${namespace} is not a bound namespace in services/repos/wrangler.jsonc`);
229+ const [repo] = await d1(`SELECT id, store FROM repos WHERE namespace = ${quoted(workspace)} AND name = ${quoted(name)} AND deleted_at IS NULL AND fork_of IS NULL`);
230+ if (!repo) throw new Error(`no repository ${workspace}/${name}`);
231+ if (namespaceOf(repo.store, config.find((one) => one.default)?.namespace) === namespace) throw new Error(`${workspace}/${name} is in ${namespace} already`);
232+ const id = `mov_${Date.now().toString(36)}${Math.random().toString(36).slice(2, 8)}`;
233+ await d1(`INSERT INTO repo_moves (id, repo_id, to_namespace, status, requested_by, queued_ms)
234+ VALUES (${quoted(id)}, ${quoted(repo.id)}, ${quoted(namespace)}, 'queued', 'scripts/ops/artifacts-namespaces.mjs', ${Date.now()})`);
235+ console.log(`queued ${id}: ${workspace}/${name} (${repo.store}) -> ${namespace}. The hourly sweep (:23) moves it; watch with \`moves\`.`);
236+ return 0;
237+ }
238+
239+ const config = configured(wranglerConfig());
240+ const cloudflare = args.includes("--cloudflare");
241+ const [held, health, events, made] = await Promise.all([
242+ readHeld(),
243+ readHealth(),
244+ cloudflare ? readEvents() : [],
245+ cloudflare ? readNamespaces() : null,
246+ ]);
247+ const rows = standings(config, held, health, events, made);
248+ if (args.includes("--json")) console.log(JSON.stringify(rows, null, 2));
249+ else console.log(table(rows));
250+ return rows.some((row) => row.warnings.length) ? 1 : 0;
251+}
252+
253+if (process.argv[1]?.replaceAll("\\", "/").endsWith("scripts/ops/artifacts-namespaces.mjs")) {
254+ main().then(
255+ (code) => process.exit(code),
256+ (error) => {
257+ console.error(`namespaces: ${error.message}`);
258+ process.exit(2);
259+ },
260+ );
261+}
+95−0
1+// The namespace report, from configuration and table rows as the repos
2+// database holds them.
3+
4+import assert from "node:assert/strict";
5+import { readFileSync } from "node:fs";
6+import { test } from "node:test";
7+import { fileURLToPath } from "node:url";
8+
9+import { parseJsonc } from "../deploy/stack.mjs";
10+import { LIMIT_PER_MINUTE, configured, namespaceOf, standings, table } from "./artifacts-namespaces.mjs";
11+
12+const sharded = {
13+ artifacts: [
14+ { binding: "ARTIFACTS", namespace: "g1t" },
15+ { binding: "ARTIFACTS_1", namespace: "g1t-us-1" },
16+ { binding: "ARTIFACTS_EU", namespace: "g1t-eu" },
17+ ],
18+ vars: {
19+ ARTIFACTS_NAMESPACES: JSON.stringify({ ARTIFACTS: "g1t", ARTIFACTS_1: "g1t-us-1", ARTIFACTS_EU: "g1t-eu", ARTIFACTS_2: "g1t-us-2" }),
20+ ARTIFACTS_NEW_REPOS: "g1t,g1t-us-1,g1t-us-2",
21+ ARTIFACTS_EU_NAMESPACE: "g1t-eu",
22+ ARTIFACTS_NAMESPACE_LIMITS: JSON.stringify({ "g1t": { max_repos: 100 } }),
23+ },
24+};
25+
26+test("today's configuration is one namespace that takes nothing new by name", () => {
27+ const today = parseJsonc(readFileSync(fileURLToPath(new URL("../../services/repos/wrangler.jsonc", import.meta.url)), "utf8"));
28+ const config = configured(today);
29+ assert.equal(config.length, 1);
30+ assert.deepEqual(
31+ { namespace: config[0].namespace, bound: config[0].bound, default: config[0].default, takes: config[0].takes_new_repos, eu: config[0].eu },
32+ { namespace: "g1t", bound: true, default: true, takes: false, eu: false },
33+ );
34+});
35+
36+test("each namespace's binding, jurisdiction and part in placing are read from the configuration", () => {
37+ const config = configured(sharded);
38+ const by = Object.fromEntries(config.map((one) => [one.namespace, one]));
39+ // A binding never says its namespace's jurisdiction; Cloudflare does.
40+ assert.equal(by["g1t-eu"].jurisdiction, null);
41+ assert.ok(by["g1t-eu"].eu && !by["g1t-eu"].takes_new_repos);
42+ assert.ok(by["g1t-us-1"].bound && by["g1t-us-1"].takes_new_repos);
43+ // Named, but no binding for it: not bound.
44+ assert.equal(by["g1t-us-2"].bound, false);
45+ assert.equal(by.g1t.max_repos, 100);
46+ assert.equal(namespaceOf("acme--rocket"), "g1t");
47+ assert.equal(namespaceOf("g1t-us-1/acme--rocket"), "g1t-us-1");
48+ assert.equal(namespaceOf(null, "main"), "main");
49+});
50+
51+test("standings add up holdings and health, and warn about what needs doing", () => {
52+ const held = [
53+ { ns: "", repos: 90, forks: 10, stored_bytes: 2e9 },
54+ { ns: "g1t", repos: 10, forks: 0, stored_bytes: 1e9 },
55+ { ns: "g1t-us-1", repos: 3, forks: 1, stored_bytes: 5e8 },
56+ ];
57+ const health = [
58+ { window: "day", store: "g1t", peak: LIMIT_PER_MINUTE * 0.8, calls: 900_000, errors: 4, rate_limited: 2, rejected: 0 },
59+ { window: "hour", store: "g1t", peak: 100, calls: 5_000, errors: 0, rate_limited: 2, rejected: 0 },
60+ { window: "hour", store: "g1t-us-1@fallback", peak: 3, calls: 40, errors: 0, rate_limited: 0, rejected: 0 },
61+ ];
62+ const rows = standings(configured(sharded), held, health, [{ namespace: "g1t", count: 7 }, { namespace: "g1t-eu", count: 1 }]);
63+ const by = Object.fromEntries(rows.map((row) => [row.namespace, row]));
64+ assert.equal(by.g1t.repos, 100);
65+ assert.equal(by.g1t.stored_bytes, 3e9);
66+ assert.equal(by.g1t.cloudflare_events, 7);
67+ assert.ok(by.g1t.warnings.some((w) => /80% of the limit/.test(w)));
68+ assert.ok(by.g1t.warnings.some((w) => /max_repos/.test(w)));
69+ assert.ok(by.g1t.warnings.some((w) => /rate limited/.test(w)));
70+ assert.ok(by["g1t-us-1"].warnings.some((w) => /fallback/.test(w)));
71+ assert.ok(by["g1t-us-2"].warnings.some((w) => /not bound/.test(w)));
72+ assert.deepEqual(by["g1t-eu"].warnings, []);
73+ const text = table(rows);
74+ assert.match(text, /^namespace/);
75+ assert.match(text, /g1t \*/);
76+ assert.match(text, /g1t-us-2 .*\(not bound\)/);
77+});
78+
79+test("what Cloudflare has is checked against what is bound", () => {
80+ const made = [
81+ { namespace: "g1t", jurisdiction: null },
82+ { namespace: "g1t-us-1", jurisdiction: null },
83+ { namespace: "g1t-eu", jurisdiction: null },
84+ ];
85+ const rows = standings(configured(sharded), [], [], [], made);
86+ const by = Object.fromEntries(rows.map((row) => [row.namespace, row]));
87+ // Made without the EU jurisdiction: it cannot be changed, so it is said.
88+ assert.ok(by["g1t-eu"].warnings.some((w) => /jurisdiction is unrestricted/.test(w)));
89+ assert.equal(by.g1t.jurisdiction, "any");
90+ // Named and bound nowhere, and not made: both said.
91+ assert.ok(by["g1t-us-2"].warnings.some((w) => /no namespace of this name/.test(w)));
92+ const right = standings(configured(sharded), [], [], [], made.map((one) => (one.namespace === "g1t-eu" ? { ...one, jurisdiction: "eu" } : one)));
93+ assert.deepEqual(right.find((row) => row.namespace === "g1t-eu").warnings, []);
94+ assert.equal(right.find((row) => row.namespace === "g1t-eu").jurisdiction, "eu");
95+});
+451−0
1+#!/usr/bin/env node
2+// Rebuilds repositories from the nightly backups into a git store, the cold
3+// fallback for an Artifacts outage (docs/ARTIFACTS.md, R12), and afterwards
4+// sends back what was pushed to it while it served.
5+//
6+// The store is deploy/self-host/gitstore: bare repositories under a root,
7+// one directory per Artifacts namespace, `<root>/<namespace>/<name>.git`.
8+// The repos service reads them through GIT_FALLBACK_URL once a namespace
9+// is switched to it (services/repos/src/fallback.rs).
10+//
11+// node scripts/ops/restore-to-gitstore.mjs index > index.json
12+// node scripts/ops/restore-to-gitstore.mjs restore --into /srv/gitstore --bundles /srv/backups
13+// node scripts/ops/restore-to-gitstore.mjs restore --gitstore https://gitstore.example # GITSTORE_SECRET
14+// node scripts/ops/restore-to-gitstore.mjs changed --into /srv/gitstore
15+// node scripts/ops/restore-to-gitstore.mjs reconcile --into /srv/gitstore
16+//
17+// Commands:
18+// index Every repository with a backup, its id and store key, from
19+// the g1t-repos database (read-only), as JSON. Keep a recent
20+// one on the fallback host: `restore --index` needs no database.
21+// restore Each repository's chain, verified as the restore drill
22+// verifies it (scripts/ops/backup-restore-drill.mjs), into the
23+// store. One already restored from the same last backup is
24+// left alone, so a second run only does what changed.
25+// changed The restored repositories whose refs moved since they were
26+// restored: pushes the fallback took (GIT_FALLBACK_WRITES=allow).
27+// reconcile Sends those back to Artifacts. A ref that Artifacts still has
28+// as it was backed up is moved to what the fallback has; one
29+// that moved on both sides is kept beside it, as
30+// refs/fallback/<the rest of its name>, for its owners to merge.
31+// Then each one's refs_version is moved in the database, so
32+// nothing kept from before is served.
33+//
34+// Options:
35+// --into <root> the git store's root on this machine (GITSTORE_ROOT).
36+// --gitstore <url> a git store reached over HTTP instead, with
37+// GITSTORE_SECRET; it must not be read-only while
38+// restoring. `changed` and `reconcile` need --into.
39+// --bundles <dir> read the bucket from a local copy (`backups/<id>/...`,
40+// as `rclone copy r2:g1t-backups <dir>` leaves it);
41+// without it each object is read through Wrangler.
42+// --index <file> the repositories from `index`'s output, not the database.
43+// --namespace <name> only repositories in this Artifacts namespace.
44+// --repo <id> only this repository (repeatable).
45+// --default-namespace the namespace keys without one are in (default g1t).
46+// --jobs <n> repositories at once (default 4).
47+// --live-root <dir> reconcile into bare repositories here
48+// (`<dir>/<namespace>/<name>.git`), for drills and tests,
49+// instead of Artifacts.
50+// --no-bump reconcile without moving refs_version.
51+//
52+// Artifacts is reached for `reconcile` with CLOUDFLARE_API_TOKEN (Artifacts
53+// edit), or CLOUDFLARE_API_KEY with CLOUDFLARE_EMAIL; the database and the
54+// bucket through Wrangler, as the restore drill does.
55+
56+import { randomUUID } from "node:crypto";
57+import { existsSync, mkdirSync, readFileSync, readdirSync, renameSync, rmSync, writeFileSync } from "node:fs";
58+import { mkdtemp } from "node:fs/promises";
59+import { tmpdir } from "node:os";
60+import { dirname, join } from "node:path";
61+
62+import { cloudflareAuth, exec, jsonFrom, wranglerEnv } from "../deploy/cloudflare.mjs";
63+import { ROOT } from "../deploy/stack.mjs";
64+import { compareRefs, parseRefs, readManifest, restore } from "./backup-restore-drill.mjs";
65+
66+const WRANGLER = join(ROOT, "node_modules/wrangler/bin/wrangler.js");
67+const DATABASE = "g1t-repos";
68+const BUCKET = process.env.BACKUP_BUCKET || "g1t-backups";
69+const ACCOUNT_ID = process.env.CLOUDFLARE_ACCOUNT_ID || "1e6f2cffa3f445920836e8ebe446bb58";
70+const NAME = /^[A-Za-z0-9_][A-Za-z0-9._-]{0,199}$/;
71+/** Where refs that moved on both sides are kept. */
72+export const CONFLICT_PREFIX = "refs/fallback/";
73+
74+async function git(args, { cwd, input } = {}) {
75+ const { code, out } = await exec("git", args, { cwd, input });
76+ if (code !== 0) throw new Error(`git ${args.find((arg) => !arg.startsWith("-")) ?? ""} failed: ${out.trim().slice(-600)}`);
77+ return out.trim();
78+}
79+
80+/** A store key's Artifacts namespace and name: `g1t-us-1/acme--rocket`, or the default's. */
81+export function locate(store, defaultNamespace = "g1t") {
82+ const at = store.indexOf("/");
83+ const [namespace, name] = at >= 0 ? [store.slice(0, at), store.slice(at + 1)] : [defaultNamespace, store];
84+ if (!NAME.test(namespace) || !NAME.test(name)) throw new Error(`not a store key: ${store}`);
85+ return { namespace, name };
86+}
87+
88+/** Where a repository lives under the git store's root. */
89+export function repoDir(root, namespace, name) {
90+ return join(root, namespace, `${name}.git`);
91+}
92+
93+/** What the git store keeps beside a repository (gitstore/server.mjs `g1t.json`), with what it was restored from. */
94+export function metaFor(target, manifest, now = new Date().toISOString(), existing = {}) {
95+ const last = manifest.chain.at(-1);
96+ return {
97+ id: existing.id ?? randomUUID(),
98+ description: existing.description ?? null,
99+ createdAt: existing.createdAt ?? now,
100+ readOnly: false,
101+ source: null,
102+ restored: {
103+ repo_id: target.id,
104+ entry: last.id,
105+ backed_up_at: last.created_at,
106+ restored_at: now,
107+ refs: Object.fromEntries(Object.entries(last.refs).filter(([ref]) => ref !== "HEAD")),
108+ },
109+ };
110+}
111+
112+function readMeta(dir) {
113+ try {
114+ return JSON.parse(readFileSync(join(dir, "g1t.json"), "utf8"));
115+ } catch {
116+ return {};
117+ }
118+}
119+
120+/** As the git store configures a repository it makes. */
121+async function configure(dir) {
122+ await git(["config", "http.receivepack", "true"], { cwd: dir });
123+ await git(["config", "receive.denyNonFastForwards", "false"], { cwd: dir });
124+ await git(["config", "uploadpack.allowAnySHA1InWant", "true"], { cwd: dir });
125+}
126+
127+/** Every ref of a bare repository but HEAD. */
128+async function refsIn(dir) {
129+ return parseRefs(await git(["for-each-ref", "--format=%(objectname) %(refname)"], { cwd: dir }));
130+}
131+
132+/**
133+ * What changed in a restored repository since it was restored: the refs
134+ * pushed to or deleted from it, each with the restored value (`base`) and
135+ * what it has now (`now`), `null` for absent.
136+ */
137+export function changedSince(restoredRefs, current) {
138+ return compareRefs(restoredRefs, current).map(({ ref, want, have }) => ({ ref, base: want, now: have }));
139+}
140+
141+/**
142+ * How each changed ref goes back to the live repository, which has `live`
143+ * now: `move` (the live ref is still what was backed up, so it takes the
144+ * fallback's value, or is deleted), `same` (it has it already), or
145+ * `conflict` (it moved too: the fallback's value goes beside it, under
146+ * CONFLICT_PREFIX).
147+ */
148+export function reconcilePlan(changes, live) {
149+ return changes.map(({ ref, base, now }) => {
150+ const current = live[ref] ?? null;
151+ if (current === now) return { ref, action: "same", from: current, to: now };
152+ if (current === base) return { ref, action: "move", from: current, to: now };
153+ return { ref, action: "conflict", from: current, to: now, kept: now ? CONFLICT_PREFIX + ref.replace(/^refs\//, "") : null };
154+ });
155+}
156+
157+// ---------------------------------------------------------------------
158+
159+async function d1(sql) {
160+ const { code, out } = await exec(process.execPath, [WRANGLER, "d1", "execute", DATABASE, "--remote", "--json", "--command", sql], {
161+ cwd: join(ROOT, "services/repos"),
162+ env: wranglerEnv(),
163+ });
164+ if (code !== 0) throw new Error(out.slice(-600));
165+ return jsonFrom(out)[0]?.results ?? [];
166+}
167+
168+const quoted = (text) => `'${String(text).replaceAll("'", "''")}'`;
169+
170+/** Every live repository with a backup: id, store key, path. */
171+async function indexFromDatabase() {
172+ const rows = await d1(`SELECT r.id, coalesce(r.store, r.namespace || '--' || r.name) AS store, r.namespace AS workspace, r.name,
173+ r.default_branch, b.last_entry
174+ FROM repos r JOIN repo_backups b ON b.repo_id = r.id
175+ WHERE r.deleted_at IS NULL AND r.fork_of IS NULL AND b.last_entry IS NOT NULL
176+ ORDER BY r.id`);
177+ return rows.map((row) => ({ id: row.id, store: row.store, path: `${row.workspace}/${row.name}`, default_branch: row.default_branch }));
178+}
179+
180+/** Without a database: what each manifest in a local copy of the bucket says. */
181+function indexFromBundles(bundles) {
182+ const base = join(bundles, "backups");
183+ if (!existsSync(base)) return [];
184+ return readdirSync(base)
185+ .filter((id) => existsSync(join(base, id, "manifest.json")))
186+ .map((id) => {
187+ const manifest = JSON.parse(readFileSync(join(base, id, "manifest.json"), "utf8"));
188+ const path = manifest.path ? `${manifest.path.namespace}/${manifest.path.name}` : null;
189+ return { id, store: manifest.store_key, path };
190+ });
191+}
192+
193+function bucketReader(localCopy) {
194+ if (localCopy) return async (key) => join(localCopy, key);
195+ return async (key, file) => {
196+ const { code, out } = await exec(process.execPath, [WRANGLER, "r2", "object", "get", `${BUCKET}/${key}`, "--remote", "--file", file], {
197+ env: wranglerEnv(),
198+ });
199+ if (code !== 0) throw new Error(`${key} could not be read: ${out.slice(-400)}`);
200+ return file;
201+ };
202+}
203+
204+/** Runs `work` over `items`, `jobs` at a time. */
205+async function pool(items, jobs, work) {
206+ const results = [];
207+ let next = 0;
208+ const lanes = Array.from({ length: Math.max(1, Math.min(jobs, items.length)) }, async () => {
209+ while (next < items.length) {
210+ const at = next++;
211+ results[at] = await work(items[at]);
212+ }
213+ });
214+ await Promise.all(lanes);
215+ return results;
216+}
217+
218+/** The git store's API, over HTTP. */
219+function gitstoreApi(url, secret) {
220+ const base = url.replace(/\/$/, "");
221+ return async (method, path, body) => {
222+ const response = await fetch(`${base}/api/repos${path}`, {
223+ method,
224+ headers: { "x-gitstore-secret": secret, ...(body ? { "content-type": "application/json" } : {}) },
225+ body: body ? JSON.stringify(body) : undefined,
226+ });
227+ const json = await response.json().catch(() => ({}));
228+ return { status: response.status, json };
229+ };
230+}
231+
232+/**
233+ * Restores one repository. Returns what happened: `restored`, `current`
234+ * (already restored from the same last backup), or `missing` (no backup).
235+ */
236+export async function restoreOne(target, { read, into, gitstore, defaultNamespace = "g1t", work }) {
237+ const { namespace, name } = locate(target.store, defaultNamespace);
238+ const scratch = await mkdtemp(join(work ?? tmpdir(), "g1t-restore-"));
239+ try {
240+ let manifestFile;
241+ try {
242+ manifestFile = await read(`backups/${target.id}/manifest.json`, join(scratch, "manifest.json"));
243+ } catch {
244+ return { id: target.id, namespace, name, result: "missing" };
245+ }
246+ if (!existsSync(manifestFile)) return { id: target.id, namespace, name, result: "missing" };
247+ const manifest = readManifest(readFileSync(manifestFile, "utf8"));
248+ const last = manifest.chain.at(-1);
249+ if (into) {
250+ const dir = repoDir(into, namespace, name);
251+ const existing = readMeta(dir);
252+ if (existing.restored?.entry === last.id && existing.restored?.repo_id === target.id) {
253+ return { id: target.id, namespace, name, result: "current" };
254+ }
255+ // Built beside it, then put in place, so a reader never sees half.
256+ const building = `${dir}.restoring-${process.pid}`;
257+ rmSync(building, { recursive: true, force: true });
258+ mkdirSync(dirname(dir), { recursive: true });
259+ const refs = await restore(manifest, read, building, scratch);
260+ await configure(building);
261+ writeFileSync(join(building, "g1t.json"), JSON.stringify(metaFor(target, manifest, undefined, existing), null, 2));
262+ rmSync(dir, { recursive: true, force: true });
263+ renameSync(building, dir);
264+ return { id: target.id, namespace, name, result: "restored", refs: Object.keys(refs).length };
265+ }
266+ // Over HTTP: rebuilt here, then pushed as a mirror.
267+ const local = join(scratch, "restored.git");
268+ const refs = await restore(manifest, read, local, scratch);
269+ const key = `${namespace}/${name}`;
270+ const made = await gitstore.api("POST", "", { name: key, defaultBranch: target.default_branch ?? "main" });
271+ if (made.status !== 200 && made.json.code !== "ALREADY_EXISTS") throw new Error(`${key}: ${made.json.message ?? made.status}`);
272+ const token = await gitstore.api("POST", `/${encodeURIComponent(key)}/tokens`, { scope: "write", ttl: 3600 });
273+ if (token.status !== 200) throw new Error(`${key}: ${token.json.message ?? token.status}`);
274+ const remote = `${gitstore.url.replace(/\/$/, "")}/git/${key}.git`;
275+ await git(["-c", `http.extraHeader=Authorization: Bearer ${token.json.plaintext}`, "push", "--quiet", "--mirror", remote], { cwd: local });
276+ return { id: target.id, namespace, name, result: "restored", refs: Object.keys(refs).length };
277+ } finally {
278+ rmSync(scratch, { recursive: true, force: true });
279+ }
280+}
281+
282+/** The restored repositories under `root`, with what each was restored from. */
283+function restoredUnder(root) {
284+ const out = [];
285+ if (!existsSync(root)) return out;
286+ for (const namespace of readdirSync(root)) {
287+ const dir = join(root, namespace);
288+ if (!NAME.test(namespace) || !existsSync(dir)) continue;
289+ for (const entry of readdirSync(dir)) {
290+ if (!entry.endsWith(".git")) continue;
291+ const meta = readMeta(join(dir, entry));
292+ if (meta.restored) out.push({ namespace, name: entry.slice(0, -4), dir: join(dir, entry), restored: meta.restored });
293+ }
294+ }
295+ return out;
296+}
297+
298+/** Repositories that took pushes since they were restored. */
299+export async function changedRepos(root) {
300+ const out = [];
301+ for (const repo of restoredUnder(root)) {
302+ const changes = changedSince(repo.restored.refs, await refsIn(repo.dir));
303+ if (changes.length) out.push({ ...repo, changes });
304+ }
305+ return out;
306+}
307+
308+/** Where to push a repository back to: Artifacts, or a bare repository under `--live-root`. */
309+function liveRemote(liveRoot) {
310+ if (liveRoot) return async (namespace, name) => ({ url: repoDir(liveRoot, namespace, name), args: [] });
311+ const auth = cloudflareAuth();
312+ if (!auth) throw new Error("set CLOUDFLARE_API_TOKEN (Artifacts edit), or CLOUDFLARE_API_KEY and CLOUDFLARE_EMAIL, or --live-root");
313+ const api = `https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/artifacts/namespaces`;
314+ return async (namespace, name) => {
315+ const at = `${api}/${namespace}/repos/${encodeURIComponent(name)}`;
316+ const info = await (await fetch(at, { headers: auth })).json().catch(() => ({}));
317+ const minted = await (
318+ await fetch(`${at}/tokens`, { method: "POST", headers: { ...auth, "content-type": "application/json" }, body: JSON.stringify({ scope: "write", ttl: 3600 }) })
319+ )
320+ .json()
321+ .catch(() => ({}));
322+ const token = minted.result?.plaintext ?? minted.result?.token;
323+ if (!token) throw new Error(`${namespace}/${name}: Artifacts gave no write token (${JSON.stringify(minted.errors ?? minted).slice(0, 300)})`);
324+ const url = info.result?.remote ?? `https://${ACCOUNT_ID}.artifacts.cloudflare.net/git/${namespace}/${name}.git`;
325+ // The token as Artifacts gives it, `?expires=` and all, as g1t sends it.
326+ return { url, args: ["-c", `http.extraHeader=Authorization: Bearer ${token}`] };
327+ };
328+}
329+
330+/** Sends one repository's changes back; what was done with each ref. */
331+export async function reconcileOne(repo, remoteFor) {
332+ const { url, args } = await remoteFor(repo.namespace, repo.name);
333+ const live = parseRefs(await git([...args, "ls-remote", url], { cwd: repo.dir }));
334+ const plan = reconcilePlan(repo.changes, live);
335+ const specs = [];
336+ for (const step of plan) {
337+ if (step.action === "move") {
338+ const lease = `--force-with-lease=${step.ref}:${step.from ?? ""}`;
339+ specs.push({ lease, spec: step.to ? `+${step.to}:${step.ref}` : `:${step.ref}` });
340+ } else if (step.action === "conflict" && step.kept) {
341+ specs.push({ lease: null, spec: `+${step.to}:${step.kept}` });
342+ }
343+ }
344+ if (specs.length) {
345+ // Not --atomic: whether Artifacts takes it is not documented (docs/ARTIFACTS.md, Q6).
346+ // Each move is leased on the value read above, so one that moved since is refused, not overwritten.
347+ const leases = specs.map((one) => one.lease).filter(Boolean);
348+ await git([...args, "push", "--quiet", ...leases, url, ...specs.map((one) => one.spec)], { cwd: repo.dir });
349+ }
350+ return plan;
351+}
352+
353+async function main() {
354+ const argv = process.argv.slice(2);
355+ const command = argv[0];
356+ const option = (name) => {
357+ const at = argv.indexOf(name);
358+ return at >= 0 ? argv[at + 1] : undefined;
359+ };
360+ const many = (name) => argv.flatMap((arg, at) => (arg === name && argv[at + 1] ? [argv[at + 1]] : []));
361+ const defaultNamespace = option("--default-namespace") ?? "g1t";
362+ const into = option("--into");
363+
364+ if (command === "index") {
365+ console.log(JSON.stringify(await indexFromDatabase(), null, 2));
366+ return 0;
367+ }
368+
369+ if (command === "restore") {
370+ const url = option("--gitstore");
371+ if (!into && !url) throw new Error("say where to restore to: --into <root> or --gitstore <url>");
372+ const gitstore = url ? { url, api: gitstoreApi(url, process.env.GITSTORE_SECRET ?? "") } : null;
373+ const bundles = option("--bundles");
374+ let targets = option("--index")
375+ ? JSON.parse(readFileSync(option("--index"), "utf8"))
376+ : bundles && argv.includes("--offline")
377+ ? indexFromBundles(bundles)
378+ : await indexFromDatabase();
379+ const only = many("--repo");
380+ if (only.length) targets = targets.filter((target) => only.includes(target.id));
381+ const namespace = option("--namespace");
382+ if (namespace) targets = targets.filter((target) => locate(target.store, defaultNamespace).namespace === namespace);
383+ const read = bucketReader(bundles);
384+ const started = Date.now();
385+ const counts = { restored: 0, current: 0, missing: 0, failed: 0 };
386+ await pool(targets, Number(option("--jobs") ?? 4), async (target) => {
387+ try {
388+ const done = await restoreOne(target, { read, into, gitstore, defaultNamespace });
389+ counts[done.result] += 1;
390+ if (done.result !== "current") console.log(`${done.result.padEnd(8)} ${done.namespace}/${done.name} (${target.path ?? target.id})`);
391+ } catch (error) {
392+ counts.failed += 1;
393+ console.error(`failed ${target.store} (${target.id}): ${error.message}`);
394+ }
395+ });
396+ const seconds = ((Date.now() - started) / 1000).toFixed(0);
397+ console.log(
398+ `${targets.length} repositories in ${seconds}s: ${counts.restored} restored, ${counts.current} already current, ${counts.missing} without a backup, ${counts.failed} failed`,
399+ );
400+ return counts.failed ? 1 : 0;
401+ }
402+
403+ if (command === "changed" || command === "reconcile") {
404+ if (!into) throw new Error(`${command} reads the git store's root: --into <root>`);
405+ const changed = await changedRepos(into);
406+ if (command === "changed") {
407+ for (const repo of changed) {
408+ console.log(`${repo.namespace}/${repo.name} (${repo.restored.repo_id})`);
409+ for (const { ref, base, now } of repo.changes) console.log(` ${ref}: ${base ?? "(none)"} -> ${now ?? "(deleted)"}`);
410+ }
411+ console.log(`${changed.length} repositories took pushes since they were restored`);
412+ return 0;
413+ }
414+ const remoteFor = liveRemote(option("--live-root"));
415+ const bumped = [];
416+ let conflicts = 0;
417+ let failed = 0;
418+ for (const repo of changed) {
419+ try {
420+ const plan = await reconcileOne(repo, remoteFor);
421+ bumped.push(repo.restored.repo_id);
422+ for (const step of plan) {
423+ if (step.action === "conflict") conflicts += 1;
424+ const what = step.action === "conflict" ? `moved on both sides; the fallback's kept as ${step.kept ?? "(it deleted it)"}` : step.action;
425+ console.log(`${repo.namespace}/${repo.name} ${step.ref}: ${what}`);
426+ }
427+ } catch (error) {
428+ failed += 1;
429+ console.error(`${repo.namespace}/${repo.name}: ${error.message}`);
430+ }
431+ }
432+ if (bumped.length && !argv.includes("--no-bump")) {
433+ await d1(`UPDATE repos SET refs_version = coalesce(refs_version, 0) + 1 WHERE id IN (${bumped.map(quoted).join(", ")})`);
434+ }
435+ console.log(`${changed.length} repositories reconciled: ${conflicts} refs kept beside a newer one, ${failed} failed`);
436+ return failed ? 1 : conflicts ? 3 : 0;
437+ }
438+
439+ console.error("usage: restore-to-gitstore.mjs index | restore | changed | reconcile (see the top of the file)");
440+ return 2;
441+}
442+
443+if (process.argv[1]?.replaceAll("\\", "/").endsWith("scripts/ops/restore-to-gitstore.mjs")) {
444+ main().then(
445+ (code) => process.exit(code),
446+ (error) => {
447+ console.error(`restore: ${error.message}`);
448+ process.exit(2);
449+ },
450+ );
451+}
+223−0
1+// The fallback path end to end (docs/ARTIFACTS.md, R12): bundles cut as the
2+// runner cuts them, restored into a git store's root, served by the git
3+// store itself (deploy/self-host/gitstore/server.mjs, read-only), pushed to
4+// while it serves, and reconciled back into the "live" repository.
5+
6+import assert from "node:assert/strict";
7+import { execFileSync, spawn, spawnSync } from "node:child_process";
8+import { createHash } from "node:crypto";
9+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
10+import { tmpdir } from "node:os";
11+import { join } from "node:path";
12+import { test } from "node:test";
13+import { fileURLToPath } from "node:url";
14+
15+import { parseRefs } from "./backup-restore-drill.mjs";
16+import { CONFLICT_PREFIX, changedSince, locate, metaFor, reconcilePlan, repoDir } from "./restore-to-gitstore.mjs";
17+
18+const TOOL = fileURLToPath(new URL("./restore-to-gitstore.mjs", import.meta.url));
19+const SERVER = fileURLToPath(new URL("../../deploy/self-host/gitstore/server.mjs", import.meta.url));
20+const git = (cwd, ...args) => execFileSync("git", args, { cwd, encoding: "utf8" }).trim();
21+const tool = (...args) => spawnSync(process.execPath, [TOOL, ...args], { encoding: "utf8" });
22+
23+function commit(dir, file, text) {
24+ writeFileSync(join(dir, file), text);
25+ git(dir, "add", "--all");
26+ git(dir, "-c", "user.name=t", "-c", "user.email=t@example.com", "commit", "--quiet", "-m", text);
27+}
28+
29+function refsOf(dir) {
30+ const refs = parseRefs(git(dir, "for-each-ref", "--format=%(objectname) %(refname)"));
31+ refs.HEAD = git(dir, "rev-parse", "HEAD");
32+ return refs;
33+}
34+
35+/** A bucket holding one full backup of `origin` as `repo_1`, as the runner and repos service leave it. */
36+function backUp(root, origin, id, storeKey) {
37+ const mirror = join(root, `${id}-mirror.git`);
38+ git(root, "clone", "--mirror", "--quiet", origin, mirror);
39+ const dir = join(root, "bucket", "backups", id);
40+ mkdirSync(dir, { recursive: true });
41+ const bundle = join(dir, "1-full.bundle");
42+ git(mirror, "bundle", "create", "--quiet", bundle, "--all");
43+ const entry = {
44+ id: "20261006T025300Z",
45+ kind: "full",
46+ key: `backups/${id}/1-full.bundle`,
47+ created_at: "2026-10-06T02:53:00.000Z",
48+ refs_version: 1,
49+ refs: refsOf(mirror),
50+ prerequisites: [],
51+ size: statSync(bundle).size,
52+ sha256: createHash("sha256").update(readFileSync(bundle)).digest("hex"),
53+ };
54+ const manifest = { version: 1, repo_id: id, store_key: storeKey, path: { namespace: "acme", name: "rocket" }, updated_at: "", chain: [entry], previous: [] };
55+ writeFileSync(join(dir, "manifest.json"), JSON.stringify(manifest));
56+ return join(root, "bucket");
57+}
58+
59+/** The git store, serving `root`, until `stop()`. */
60+async function serve(root, { readOnly }) {
61+ const port = 41000 + Math.floor(Math.random() * 2000);
62+ const secret = "0123456789abcdef0123456789abcdef";
63+ const child = spawn(process.execPath, [SERVER], {
64+ env: { ...process.env, GITSTORE_ROOT: root, GITSTORE_PORT: String(port), GITSTORE_SECRET: secret, GITSTORE_URL: `http://127.0.0.1:${port}`, GITSTORE_READ_ONLY: readOnly ? "1" : "" },
65+ stdio: "ignore",
66+ });
67+ const url = `http://127.0.0.1:${port}`;
68+ for (let tries = 0; tries < 100; tries++) {
69+ try {
70+ if ((await fetch(`${url}/healthz`)).ok) break;
71+ } catch {}
72+ await new Promise((resolve) => setTimeout(resolve, 100));
73+ }
74+ const api = async (method, path, body) => {
75+ const response = await fetch(`${url}/api/repos${path}`, {
76+ method,
77+ headers: { "x-gitstore-secret": secret, ...(body ? { "content-type": "application/json" } : {}) },
78+ body: body ? JSON.stringify(body) : undefined,
79+ });
80+ return { status: response.status, json: await response.json().catch(() => ({})) };
81+ };
82+ return { url, secret, api, stop: () => child.kill() };
83+}
84+
85+test("keys, plans and what a restore records", () => {
86+ assert.deepEqual(locate("acme--rocket"), { namespace: "g1t", name: "acme--rocket" });
87+ assert.deepEqual(locate("g1t-us-1/pulls--pul_1"), { namespace: "g1t-us-1", name: "pulls--pul_1" });
88+ assert.throws(() => locate("../x"), /not a store key/);
89+ assert.equal(repoDir("/srv", "g1t", "acme--rocket").replaceAll("\\", "/"), "/srv/g1t/acme--rocket.git");
90+ const a = "a".repeat(40);
91+ const b = "b".repeat(40);
92+ const c = "c".repeat(40);
93+ const changes = changedSince({ "refs/heads/main": a, "refs/heads/old": a }, { "refs/heads/main": b, "refs/heads/new": c });
94+ assert.deepEqual(changes, [
95+ { ref: "refs/heads/main", base: a, now: b },
96+ { ref: "refs/heads/new", base: null, now: c },
97+ { ref: "refs/heads/old", base: a, now: null },
98+ ]);
99+ // Live still as backed up: moved. Live has it: nothing. Live moved too: kept beside.
100+ assert.deepEqual(
101+ reconcilePlan(changes, { "refs/heads/main": a, "refs/heads/old": c }).map((step) => [step.ref, step.action, step.kept ?? null]),
102+ [
103+ ["refs/heads/main", "move", null],
104+ ["refs/heads/new", "move", null],
105+ ["refs/heads/old", "conflict", null],
106+ ],
107+ );
108+ assert.equal(reconcilePlan(changes, { "refs/heads/main": c })[0].kept, `${CONFLICT_PREFIX}heads/main`);
109+ assert.equal(reconcilePlan(changes, { "refs/heads/main": b })[0].action, "same");
110+ const manifest = { chain: [{ id: "e1", created_at: "t", refs: { HEAD: a, "refs/heads/main": a } }] };
111+ const meta = metaFor({ id: "repo_1" }, manifest, "now", { id: "kept", createdAt: "then" });
112+ assert.equal(meta.id, "kept");
113+ assert.deepEqual(meta.restored.refs, { "refs/heads/main": a });
114+ assert.equal(meta.restored.entry, "e1");
115+});
116+
117+test("restored repositories are served read-only, and pushes taken later are reconciled", async () => {
118+ const root = mkdtempSync(join(tmpdir(), "g1t-fallback-test-"));
119+ let server = null;
120+ try {
121+ const origin = join(root, "origin");
122+ mkdirSync(origin);
123+ git(origin, "init", "--quiet", "--initial-branch=main");
124+ commit(origin, "a.txt", "one");
125+ git(origin, "tag", "-a", "v1", "-m", "v1");
126+ const bucket = backUp(root, origin, "repo_1", "g1t-us-1/acme--rocket");
127+ const index = join(root, "index.json");
128+ writeFileSync(index, JSON.stringify([{ id: "repo_1", store: "g1t-us-1/acme--rocket", path: "acme/rocket" }, { id: "repo_2", store: "acme--gone", path: "acme/gone" }]));
129+ const store = join(root, "store");
130+
131+ const first = tool("restore", "--into", store, "--bundles", bucket, "--index", index);
132+ assert.equal(first.status, 0, first.stdout + first.stderr);
133+ assert.match(first.stdout, /1 restored, 0 already current, 1 without a backup/);
134+ const dir = repoDir(store, "g1t-us-1", "acme--rocket");
135+ assert.ok(existsSync(join(dir, "g1t.json")));
136+ assert.equal(git(dir, "rev-parse", "refs/heads/main"), git(origin, "rev-parse", "main"));
137+ // Again: nothing to do.
138+ const again = tool("restore", "--into", store, "--bundles", bucket, "--index", index);
139+ assert.match(again.stdout, /0 restored, 1 already current/);
140+ // Only a namespace asked for.
141+ assert.match(tool("restore", "--into", store, "--bundles", bucket, "--index", index, "--namespace", "g1t-eu").stdout, /^0 repositories/m);
142+
143+ // Served as the repos service reads it: a namespaced key, a token, git.
144+ server = await serve(store, { readOnly: true });
145+ const key = encodeURIComponent("g1t-us-1/acme--rocket");
146+ const info = await server.api("GET", `/${key}`);
147+ assert.equal(info.status, 200);
148+ assert.equal(info.json.remote, `${server.url}/git/g1t-us-1/acme--rocket.git`);
149+ const read = await server.api("POST", `/${key}/tokens`, { scope: "read", ttl: 600 });
150+ assert.equal(read.status, 200);
151+ const header = `http.extraHeader=Authorization: Bearer ${read.json.plaintext}`;
152+ const clone = join(root, "clone");
153+ git(root, "-c", header, "clone", "--quiet", info.json.remote, clone);
154+ assert.equal(git(clone, "rev-parse", "HEAD"), git(origin, "rev-parse", "main"));
155+ // Read-only: no write token, no new repository.
156+ assert.equal((await server.api("POST", `/${key}/tokens`, { scope: "write" })).json.code, "READ_ONLY");
157+ assert.equal((await server.api("POST", "", { name: "g1t-us-1/new" })).json.code, "READ_ONLY");
158+ assert.equal((await server.api("GET", `/${encodeURIComponent("../etc")}`)).json.code, "INVALID_REPO_NAME");
159+ server.stop();
160+ server = null;
161+
162+ // While it served with writes allowed, a push came in.
163+ assert.match(tool("changed", "--into", store).stdout, /^0 repositories/m);
164+ commit(clone, "b.txt", "pushed to the fallback");
165+ git(clone, "push", "--quiet", dir, "HEAD:refs/heads/main", "HEAD:refs/heads/during");
166+ const changed = tool("changed", "--into", store);
167+ assert.match(changed.stdout, /g1t-us-1\/acme--rocket \(repo_1\)/);
168+ assert.match(changed.stdout, /refs\/heads\/during/);
169+
170+ // The live repository still as backed up: the push goes back as it is.
171+ const live = join(root, "live");
172+ mkdirSync(join(live, "g1t-us-1"), { recursive: true });
173+ git(root, "clone", "--bare", "--quiet", origin, repoDir(live, "g1t-us-1", "acme--rocket"));
174+ const reconciled = tool("reconcile", "--into", store, "--live-root", live, "--no-bump");
175+ assert.equal(reconciled.status, 0, reconciled.stdout + reconciled.stderr);
176+ const liveDir = repoDir(live, "g1t-us-1", "acme--rocket");
177+ assert.equal(git(liveDir, "rev-parse", "refs/heads/main"), git(clone, "rev-parse", "HEAD"));
178+ assert.equal(git(liveDir, "rev-parse", "refs/heads/during"), git(clone, "rev-parse", "HEAD"));
179+
180+ // Moved on both sides: the live one stays, the fallback's goes beside it.
181+ rmSync(liveDir, { recursive: true, force: true });
182+ commit(origin, "c.txt", "pushed to Artifacts before the outage, after the backup");
183+ git(root, "clone", "--bare", "--quiet", origin, liveDir);
184+ const conflicted = tool("reconcile", "--into", store, "--live-root", live, "--no-bump");
185+ assert.equal(conflicted.status, 3, conflicted.stdout + conflicted.stderr);
186+ assert.match(conflicted.stdout, /moved on both sides/);
187+ assert.equal(git(liveDir, "rev-parse", "refs/heads/main"), git(origin, "rev-parse", "main"));
188+ assert.equal(git(liveDir, "rev-parse", "refs/fallback/heads/main"), git(clone, "rev-parse", "HEAD"));
189+ } finally {
190+ server?.stop();
191+ rmSync(root, { recursive: true, force: true });
192+ }
193+});
194+
195+test("a git store over HTTP is restored into through its API", async () => {
196+ const root = mkdtempSync(join(tmpdir(), "g1t-fallback-http-"));
197+ let server = null;
198+ try {
199+ const origin = join(root, "origin");
200+ mkdirSync(origin);
201+ git(origin, "init", "--quiet", "--initial-branch=main");
202+ commit(origin, "a.txt", "one");
203+ git(origin, "branch", "dev");
204+ const bucket = backUp(root, origin, "repo_1", "acme--rocket");
205+ const index = join(root, "index.json");
206+ writeFileSync(index, JSON.stringify([{ id: "repo_1", store: "acme--rocket", default_branch: "main" }]));
207+ server = await serve(join(root, "store"), { readOnly: false });
208+ const run = spawn(process.execPath, [TOOL, "restore", "--gitstore", server.url, "--bundles", bucket, "--index", index], {
209+ env: { ...process.env, GITSTORE_SECRET: server.secret },
210+ });
211+ let out = "";
212+ run.stdout.on("data", (chunk) => (out += chunk));
213+ run.stderr.on("data", (chunk) => (out += chunk));
214+ const status = await new Promise((resolve) => run.on("close", resolve));
215+ assert.equal(status, 0, out);
216+ // The default namespace's repositories are kept under it.
217+ const dir = repoDir(join(root, "store"), "g1t", "acme--rocket");
218+ assert.equal(git(dir, "rev-parse", "refs/heads/dev"), git(origin, "rev-parse", "dev"));
219+ } finally {
220+ server?.stop();
221+ rmSync(root, { recursive: true, force: true });
222+ }
223+});
+8−0
1+-- Where a workspace keeps its repositories' git data (docs/ARTIFACTS.md,
2+-- R7): NULL for anywhere g1t stores it, 'eu' for the EU only. Set by an
3+-- owner in the workspace's settings (`set_workspace_residency`), offered
4+-- only once the repos service has an EU namespace (`storage_options`). The
5+-- repos service reads it as it places a new repository; repositories made
6+-- before a change stay where they are. Additive; every workspace starts
7+-- as anywhere, as today.
8+ALTER TABLE workspaces ADD COLUMN data_residency TEXT;
+65−0
10591059 Ok(Outcome::Ok(a.base_permission))
10601060 }
10611061
1062+ /// `workspace_residency`: where a workspace keeps its repositories'
1063+ /// git data, for the repos service as it places a new one, and for its
1064+ /// settings page. Null when there is no such workspace.
1065+ pub async fn workspace_residency(&self, a: g1t_contracts::identity::SlugArgs) -> Result<Option<g1t_contracts::identity::DataResidency>> {
1066+ #[derive(Deserialize)]
1067+ struct Row {
1068+ #[serde(default)]
1069+ data_residency: Option<String>,
1070+ }
1071+ let row = self
1072+ .db
1073+ .prepare("SELECT data_residency FROM workspaces WHERE slug = ? AND deleted_at IS NULL")
1074+ .bind(&[a.slug.trim().to_lowercase().into()])?
1075+ .first::<Row>(None)
1076+ .await?;
1077+ Ok(row.map(|row| {
1078+ row.data_residency
1079+ .as_deref()
1080+ .and_then(g1t_contracts::identity::DataResidency::parse)
1081+ .unwrap_or_default()
1082+ }))
1083+ }
1084+
1085+ /// `set_workspace_residency`: owners only. Applies to repositories
1086+ /// made from then on; those it has stay where they are. Whether the EU
1087+ /// can be chosen is the repos service's to say (`storage_options`);
1088+ /// the site offers it only then, and the repos service refuses to
1089+ /// place an EU workspace's repository anywhere else.
1090+ pub async fn set_workspace_residency(
1091+ &self,
1092+ a: g1t_contracts::identity::SetResidencyArgs,
1093+ ) -> Result<Outcome<g1t_contracts::identity::DataResidency>> {
1094+ let slug = a.slug.trim().to_lowercase();
1095+ if !crate::security::is_person(&a.actor) || a.actor.role_in(&slug) != Some(Role::Owner) {
1096+ return Ok(Outcome::fail(FailureCode::Forbidden, "Only an owner can change where a workspace keeps its data."));
1097+ }
1098+ if !a.actor.verified {
1099+ return Ok(Outcome::fail(FailureCode::Forbidden, "Confirm your email address before changing where the workspace keeps its data."));
1100+ }
1101+ let Some(previous) = self.workspace_residency(g1t_contracts::identity::SlugArgs { slug: slug.clone() }).await? else {
1102+ return Ok(Outcome::fail(FailureCode::NotFound, "Workspace not found."));
1103+ };
1104+ if previous == a.residency {
1105+ return Ok(Outcome::Ok(previous));
1106+ }
1107+ let stored = match a.residency {
1108+ g1t_contracts::identity::DataResidency::Anywhere => JsValue::NULL,
1109+ other => other.as_str().into(),
1110+ };
1111+ self.db
1112+ .prepare("UPDATE workspaces SET data_residency = ? WHERE slug = ? AND deleted_at IS NULL")
1113+ .bind(&[stored, slug.as_str().into()])?
1114+ .run()
1115+ .await?;
1116+ self.audit_workspace(
1117+ &a.actor,
1118+ "workspace.residency_changed",
1119+ &slug,
1120+ Surface::Web,
1121+ format!("Changed where new repositories keep their data from {} to {}", previous.as_str(), a.residency.as_str()),
1122+ )
1123+ .await;
1124+ Ok(Outcome::Ok(a.residency))
1125+ }
1126+
10621127 pub async fn outside_collaborators(&self, a: OutsideCollaboratorsArgs) -> Result<Outcome<Vec<OutsideCollaborator>>> {
10631128 let slug = a.slug.trim().to_lowercase();
10641129 if !a.viewer.as_ref().is_some_and(|viewer| viewer.role_in(&slug) == Some(Role::Owner)) {
+3−0
791791 "respond_repo_invitation" => reply(&identity.respond_repo_invitation(args(body)?).await?),
792792 "revoke_repo_invitation" => reply(&identity.revoke_repo_invitation(args(body)?).await?),
793793 "set_base_permission" => reply(&identity.set_base_permission(args(body)?).await?),
794+ // Where a workspace keeps its repositories' git data (EU residency).
795+ "workspace_residency" => reply(&identity.workspace_residency(args(body)?).await?),
796+ "set_workspace_residency" => reply(&identity.set_workspace_residency(args(body)?).await?),
794797 "outside_collaborators" => reply(&identity.outside_collaborators(args(body)?).await?),
795798 "forget_repo_access" => reply(&identity.forget_repo_access(args(body)?).await?),
796799 // Staff only: sudo.g1t.sh, over its service binding. See admin.rs.
+62−0
1+-- Moving a repository from one git store namespace to another
2+-- (src/moves.rs; docs/ARTIFACTS.md, R7). Additive; nothing is backfilled,
3+-- and nothing moves until an operator asks.
4+--
5+-- writes_paused_until: milliseconds since the epoch. Until then nothing
6+-- writes to the repository: pushes wait up to 20 seconds for it to pass,
7+-- then are told to try again; merges, commits from the web and handed-out
8+-- push credentials the same. Set while a move copies the repository and
9+-- its pull requests' working copies, cleared when it ends either way, and
10+-- it passes on its own should a move die half way.
11+-- writes_paused_for: why, in words (`moving to g1t-us-1`).
12+ALTER TABLE repos ADD COLUMN writes_paused_until INTEGER;
13+ALTER TABLE repos ADD COLUMN writes_paused_for TEXT;
14+
15+-- One row per move asked for. The hourly sweep (`23 * * * *`) runs the
16+-- queued ones, oldest first, one at a time.
17+--
18+-- status: queued | moving | moved | failed | cleaned | diverged.
19+-- moved: the repository reads and writes in its new namespace; the copy
20+-- in the old one is kept MOVE_KEEP_DAYS (7) in case of a rollback, then
21+-- deleted (cleaned). diverged: the old copy's refs changed after the
22+-- move (a push that slipped past the pause), so it is kept and an
23+-- operator reconciles it.
24+-- to_namespace: where it goes.
25+-- requested_by: who asked, in words.
26+-- note: the last thing that happened (why it waits, why it failed).
27+CREATE TABLE repo_moves (
28+ id TEXT PRIMARY KEY,
29+ repo_id TEXT NOT NULL,
30+ to_namespace TEXT NOT NULL,
31+ status TEXT NOT NULL DEFAULT 'queued',
32+ requested_by TEXT,
33+ queued_ms INTEGER NOT NULL,
34+ started_ms INTEGER,
35+ finished_ms INTEGER,
36+ cleaned_ms INTEGER,
37+ attempts INTEGER NOT NULL DEFAULT 0,
38+ note TEXT
39+);
40+CREATE INDEX repo_moves_queue ON repo_moves (status, queued_ms);
41+-- At most one move in hand per repository.
42+CREATE UNIQUE INDEX repo_moves_active ON repo_moves (repo_id) WHERE status IN ('queued', 'moving');
43+
44+-- What each move copied: the repository and each of its pull requests'
45+-- working copies, from one key to another, with every ref as copied.
46+--
47+-- name: the key's name without its namespace. A name with a copy not yet
48+-- cleaned stays taken (registry.rs `claim_store_key`), so a new
49+-- repository never adopts an old copy left behind.
50+-- refs: JSON object, ref name to object, as copied; the old copy is
51+-- deleted only while it still says the same.
52+CREATE TABLE repo_move_copies (
53+ move_id TEXT NOT NULL,
54+ repo_id TEXT NOT NULL,
55+ from_key TEXT NOT NULL,
56+ to_key TEXT NOT NULL,
57+ name TEXT NOT NULL,
58+ refs TEXT NOT NULL DEFAULT '{}',
59+ cleaned_ms INTEGER,
60+ PRIMARY KEY (move_id, repo_id)
61+);
62+CREATE INDEX repo_move_copies_name ON repo_move_copies (name) WHERE cleaned_ms IS NULL;
+6−0
601601 return Ok(refused("The repository was deleted."));
602602 };
603603 let key = store_key(&repo);
604+ // Served from the fallback store (fallback.rs), which holds what the
605+ // backups hold: backing that up would only record an older state.
606+ if store.on_fallback(&key) {
607+ settle_failure(db, &row, "The git store is on its fallback; backups wait until it is back.").await?;
608+ return Ok(refused("The git store is on its fallback; backups wait until it is back."));
609+ }
604610 let access = store.handout(&key, Scope::Read).await?;
605611 // Asked before: the same bundle, so parts already sent still fit.
606612 if let (Some(kind), Some(_)) = (row.upload_kind.as_deref(), row.upload_id.as_deref()) {
+6−0
403403 if let Some((code, message)) = crate::lifecycle::archived_refusal(&target) {
404404 return Ok(Outcome::fail(code, message));
405405 }
406+ // Moving between namespaces: wait for it (moves.rs). Both are read
407+ // again once it is done, for their new keys.
408+ let (source, target) = match (self.unpaused(source).await?, self.unpaused(target).await?) {
409+ (Ok(source), Ok(target)) => (source, target),
410+ (Err((code, message)), _) | (_, Err((code, message))) => return Ok(Outcome::fail(code, message)),
411+ };
406412 let from_fork = source.id != target.id;
407413 let base_branch = target.default_branch.clone();
408414 let branch = a.branch.clone().unwrap_or_else(|| base_branch.clone());
+5−0
5555 if let Some((code, message)) = crate::lifecycle::archived_refusal(&repo) {
5656 return Ok(Outcome::fail(code, message));
5757 }
58+ // Moving between namespaces: wait for it (moves.rs).
59+ let repo = match self.unpaused(repo).await? {
60+ Ok(repo) => repo,
61+ Err((code, message)) => return Ok(Outcome::fail(code, message)),
62+ };
5863 if !is_valid_branch_name(&a.branch) || a.branch == repo.default_branch {
5964 return Ok(Outcome::fail(FailureCode::Invalid, format!("{} cannot be the new branch's name.", a.branch)));
6065 }
+358−0
1+//! The cold fallback for the git store (docs/ARTIFACTS.md, R12).
2+//!
3+//! When Artifacts is down for a namespace, the repos service can serve that
4+//! namespace from a self-hosted git store instead
5+//! (`deploy/self-host/gitstore`), rebuilt from the nightly backups
6+//! (`scripts/ops/restore-to-gitstore.mjs`). It is switched by
7+//! configuration, not code:
8+//!
9+//! - `GIT_FALLBACK_URL`: where the git store answers, `https://...`.
10+//! - `GIT_FALLBACK_SECRET` (a secret): the store's API secret.
11+//! - `GIT_FALLBACK_NAMESPACES`: the namespaces served from it, comma
12+//! separated, or `*` for all. Unset or empty: none, and nothing changes.
13+//! - `GIT_FALLBACK_WRITES`: `refuse` (the default) or `allow`. While
14+//! refused, a switched namespace is read-only: clones, fetches and pages
15+//! work; pushes, merges and new repositories wait, told so in words.
16+//!
17+//! The store keeps each namespace's repositories under its own directory,
18+//! so the key `g1t-us-1/acme--rocket` is `g1t-us-1/acme--rocket` there and
19+//! `acme--rocket` (the default namespace) is `g1t/acme--rocket`. Its remotes
20+//! read `<url>/git/<namespace>/<name>.git`, the shape Artifacts uses.
21+//!
22+//! This module is the plain part: the settings, and how each call the
23+//! service makes on an Artifacts binding becomes a request to the store's
24+//! API and back. store.rs makes the requests.
25+
26+use serde_json::{Value, json};
27+
28+/// The fallback store's settings, when it is configured at all.
29+#[derive(Clone, Debug, PartialEq, Eq)]
30+pub struct Settings {
31+ /// Where the store answers, without a trailing slash.
32+ pub url: String,
33+ pub secret: String,
34+ pub switched: Switched,
35+ /// Whether a switched namespace takes writes.
36+ pub writes: bool,
37+}
38+
39+/// Which namespaces are served from the fallback store.
40+#[derive(Clone, Debug, PartialEq, Eq)]
41+pub enum Switched {
42+ All,
43+ Some(Vec<String>),
44+}
45+
46+impl Settings {
47+ /// The settings from the variables, or `None` when there is no store
48+ /// to fall back to (no address, or no secret).
49+ pub fn from_vars(url: Option<&str>, secret: Option<&str>, namespaces: Option<&str>, writes: Option<&str>) -> Option<Self> {
50+ let url = url.map(str::trim).filter(|url| url.starts_with("https://") || url.starts_with("http://"))?;
51+ let secret = secret.map(str::trim).filter(|secret| secret.len() >= 16)?;
52+ let namespaces = namespaces.unwrap_or_default().trim();
53+ let switched = if namespaces == "*" {
54+ Switched::All
55+ } else {
56+ Switched::Some(
57+ namespaces
58+ .split(',')
59+ .map(str::trim)
60+ .filter(|name| crate::shards::valid_namespace(name))
61+ .map(str::to_owned)
62+ .collect(),
63+ )
64+ };
65+ Some(Settings {
66+ url: url.trim_end_matches('/').to_owned(),
67+ secret: secret.to_owned(),
68+ switched,
69+ writes: writes.is_some_and(|value| value.trim().eq_ignore_ascii_case("allow")),
70+ })
71+ }
72+
73+ /// Whether `namespace` is served from the fallback store now.
74+ pub fn serves(&self, namespace: &str) -> bool {
75+ match &self.switched {
76+ Switched::All => true,
77+ Switched::Some(names) => names.iter().any(|name| name == namespace),
78+ }
79+ }
80+
81+ /// The remote git uses for `name` in `namespace`.
82+ pub fn remote(&self, namespace: &str, name: &str) -> String {
83+ format!("{}/git/{}.git", self.url, store_key(namespace, name))
84+ }
85+}
86+
87+/// A repository's key in the fallback store: always with its namespace.
88+pub fn store_key(namespace: &str, name: &str) -> String {
89+ format!("{namespace}/{name}")
90+}
91+
92+/// Percent-encodes one path segment.
93+fn segment(text: &str) -> String {
94+ text.bytes()
95+ .map(|b| if b.is_ascii_alphanumeric() || b"-._~".contains(&b) { (b as char).to_string() } else { format!("%{b:02X}") })
96+ .collect()
97+}
98+
99+/// Percent-encodes a query value.
100+fn query(pairs: &[(&str, String)]) -> String {
101+ pairs
102+ .iter()
103+ .map(|(key, value)| format!("{key}={}", segment(value)))
104+ .collect::<Vec<_>>()
105+ .join("&")
106+}
107+
108+/// How a call on the binding is asked of the store's API.
109+#[derive(Debug, PartialEq)]
110+pub struct Route {
111+ pub method: &'static str,
112+ /// Below the store's address: `/api/repos/...`.
113+ pub path: String,
114+ pub body: Option<Value>,
115+ /// What the answer is: JSON, or bytes (a blob or a file).
116+ pub bytes: bool,
117+}
118+
119+/// A call the fallback cannot make, as the binding would have thrown it.
120+#[derive(Debug, PartialEq)]
121+pub struct Refused {
122+ pub code: &'static str,
123+ pub message: String,
124+}
125+
126+fn arg<'a>(args: &'a [Value], at: usize) -> &'a Value {
127+ args.get(at).unwrap_or(&Value::Null)
128+}
129+
130+fn text(args: &[Value], at: usize) -> Result<String, Refused> {
131+ arg(args, at).as_str().map(str::to_owned).ok_or_else(|| Refused {
132+ code: "INVALID_ARGUMENT",
133+ message: format!("argument {at} should be text"),
134+ })
135+}
136+
137+/// The request for `method(args)`: on the namespace when `repo` is `None`
138+/// (`create`, `get`, `delete`), else on the repository named `repo` there.
139+pub fn route(namespace: &str, repo: Option<&str>, method: &str, args: &[Value]) -> Result<Route, Refused> {
140+ let json_route = |method: &'static str, path: String, body: Option<Value>| Route { method, path, body, bytes: false };
141+ let Some(repo) = repo else {
142+ let name = text(args, 0)?;
143+ let key = store_key(namespace, &name);
144+ return match method {
145+ "create" => {
146+ let options = arg(args, 1);
147+ Ok(json_route(
148+ "POST",
149+ "/api/repos".to_owned(),
150+ Some(json!({
151+ "name": key,
152+ "description": options.get("description").cloned().unwrap_or(Value::Null),
153+ "defaultBranch": options.get("setDefaultBranch").cloned().unwrap_or(Value::Null),
154+ })),
155+ ))
156+ }
157+ "get" => Ok(json_route("GET", format!("/api/repos/{}", segment(&key)), None)),
158+ "delete" => Ok(json_route("DELETE", format!("/api/repos/{}", segment(&key)), None)),
159+ other => Err(Refused { code: "NOT_SUPPORTED", message: format!("{other} is not offered by the fallback store") }),
160+ };
161+ };
162+ let base = format!("/api/repos/{}", segment(&store_key(namespace, repo)));
163+ match method {
164+ "info" => Ok(json_route("GET", base, None)),
165+ "createToken" => Ok(json_route(
166+ "POST",
167+ format!("{base}/tokens"),
168+ Some(json!({ "scope": arg(args, 0).as_str().unwrap_or("write"), "ttl": arg(args, 1).as_u64().unwrap_or(3_600) })),
169+ )),
170+ "log" => {
171+ let options = arg(args, 0);
172+ let mut pairs = Vec::new();
173+ for name in ["ref", "limit", "offset"] {
174+ match options.get(name) {
175+ Some(Value::String(value)) => pairs.push((name, value.clone())),
176+ Some(Value::Number(value)) => pairs.push((name, value.to_string())),
177+ _ => {}
178+ }
179+ }
180+ Ok(json_route("GET", format!("{base}/log?{}", query(&pairs)), None))
181+ }
182+ "readCommit" => Ok(json_route("GET", format!("{base}/commits/{}", segment(&text(args, 0)?)), None)),
183+ "readTree" => Ok(json_route("GET", format!("{base}/trees/{}", segment(&text(args, 0)?)), None)),
184+ "readBlob" => Ok(Route { method: "GET", path: format!("{base}/blobs/{}", segment(&text(args, 0)?)), body: None, bytes: true }),
185+ "readFile" => {
186+ let options = arg(args, 0);
187+ let field = |name: &str| options.get(name).and_then(Value::as_str).unwrap_or_default().to_owned();
188+ Ok(Route {
189+ method: "GET",
190+ path: format!("{base}/file?{}", query(&[("ref", field("ref")), ("path", field("path"))])),
191+ body: None,
192+ bytes: true,
193+ })
194+ }
195+ "fork" => {
196+ let target = text(args, 0)?;
197+ let options = arg(args, 1);
198+ Ok(json_route(
199+ "POST",
200+ format!("{base}/fork"),
201+ Some(json!({
202+ "name": store_key(namespace, &target),
203+ "defaultBranchOnly": options.get("defaultBranchOnly").and_then(Value::as_bool).unwrap_or(true),
204+ })),
205+ ))
206+ }
207+ other => Err(Refused { code: "NOT_SUPPORTED", message: format!("{other} is not offered by the fallback store") }),
208+ }
209+}
210+
211+/// What an answer from the store means for the call that asked.
212+#[derive(Debug, PartialEq)]
213+pub enum Answer {
214+ /// JSON, as the binding would have returned it.
215+ Json(Value),
216+ /// A blob or a file's bytes.
217+ Bytes(Vec<u8>),
218+ /// Nothing there: a blob or file not found, as the binding's `null`.
219+ Null,
220+ /// An error, with the binding's code; `None` for one that may pass.
221+ Error { code: Option<String>, message: String },
222+}
223+
224+/// Reads an answer: `status` and `body` from the store, for `route`.
225+pub fn answer(route: &Route, status: u16, body: Vec<u8>) -> Answer {
226+ if (200..300).contains(&status) {
227+ if route.bytes {
228+ return Answer::Bytes(body);
229+ }
230+ return match serde_json::from_slice::<Value>(&body) {
231+ Ok(Value::Null) => Answer::Null,
232+ Ok(value) => Answer::Json(value),
233+ Err(_) => Answer::Error { code: None, message: "the fallback store sent something that is not JSON".to_owned() },
234+ };
235+ }
236+ if status == 404 && route.bytes {
237+ return Answer::Null;
238+ }
239+ let said: Value = serde_json::from_slice(&body).unwrap_or(Value::Null);
240+ let message = said.get("message").and_then(Value::as_str).map_or_else(|| format!("the fallback store answered {status}"), str::to_owned);
241+ // 5xx may pass: like the binding's INTERNAL_ERROR.
242+ let code = if status >= 500 {
243+ Some("INTERNAL_ERROR".to_owned())
244+ } else {
245+ Some(said.get("code").and_then(Value::as_str).unwrap_or(if status == 404 { "NOT_FOUND" } else { "INVALID_ARGUMENT" }).to_owned())
246+ };
247+ Answer::Error { code, message }
248+}
249+
250+/// Whether a call writes: refused while a switched namespace is read-only.
251+pub fn writes(repo: Option<&str>, method: &str, args: &[Value]) -> bool {
252+ match (repo, method) {
253+ (None, "create" | "delete") => true,
254+ (Some(_), "fork") => true,
255+ (Some(_), "createToken") => arg(args, 0).as_str().unwrap_or("write") == "write",
256+ _ => false,
257+ }
258+}
259+
260+#[cfg(test)]
261+mod tests {
262+ use super::*;
263+
264+ fn settings(namespaces: &str) -> Settings {
265+ Settings::from_vars(Some("https://gitstore.example/"), Some("0123456789abcdef"), Some(namespaces), None).unwrap()
266+ }
267+
268+ #[test]
269+ fn nothing_is_switched_unless_the_store_and_namespaces_are_named() {
270+ assert_eq!(Settings::from_vars(None, Some("0123456789abcdef"), Some("*"), None), None);
271+ assert_eq!(Settings::from_vars(Some("https://x"), None, Some("*"), None), None);
272+ // A secret too short to be one.
273+ assert_eq!(Settings::from_vars(Some("https://x"), Some("short"), Some("*"), None), None);
274+ assert_eq!(Settings::from_vars(Some("ftp://x"), Some("0123456789abcdef"), Some("*"), None), None);
275+ let none = settings("");
276+ assert!(!none.serves("g1t"));
277+ let some = settings("g1t, g1t-us-1,bad name");
278+ assert!(some.serves("g1t") && some.serves("g1t-us-1") && !some.serves("g1t-eu"));
279+ assert!(settings("*").serves("anything"));
280+ // Read-only unless told otherwise.
281+ assert!(!some.writes);
282+ let writable = Settings::from_vars(Some("https://x"), Some("0123456789abcdef"), Some("*"), Some("Allow")).unwrap();
283+ assert!(writable.writes);
284+ assert_eq!(some.url, "https://gitstore.example");
285+ assert_eq!(some.remote("g1t", "acme--rocket"), "https://gitstore.example/git/g1t/acme--rocket.git");
286+ }
287+
288+ #[test]
289+ fn a_remote_names_its_key_as_an_artifacts_remote_does() {
290+ let remote = settings("*").remote("g1t-us-1", "acme--rocket");
291+ // store.rs reads keys back from remotes by their last two segments.
292+ assert!(remote.ends_with("/g1t-us-1/acme--rocket.git"));
293+ }
294+
295+ #[test]
296+ fn calls_on_the_namespace_become_requests_for_its_key() {
297+ let create = route("g1t", None, "create", &[json!("acme--rocket"), json!({ "description": "d", "setDefaultBranch": "trunk" })]).unwrap();
298+ assert_eq!(create.method, "POST");
299+ assert_eq!(create.path, "/api/repos");
300+ assert_eq!(create.body.unwrap(), json!({ "name": "g1t/acme--rocket", "description": "d", "defaultBranch": "trunk" }));
301+ let get = route("g1t", None, "get", &[json!("acme--rocket")]).unwrap();
302+ assert_eq!((get.method, get.path.as_str()), ("GET", "/api/repos/g1t%2Facme--rocket"));
303+ assert_eq!(route("g1t", None, "delete", &[json!("x1")]).unwrap().method, "DELETE");
304+ assert_eq!(route("g1t", None, "list", &[json!("x1")]).unwrap_err().code, "NOT_SUPPORTED");
305+ assert_eq!(route("g1t", None, "get", &[]).unwrap_err().code, "INVALID_ARGUMENT");
306+ }
307+
308+ #[test]
309+ fn calls_on_a_repository_become_requests_below_it() {
310+ let base = "/api/repos/g1t-us-1%2Fpulls--pul_1";
311+ let at = |method: &str, args: &[Value]| route("g1t-us-1", Some("pulls--pul_1"), method, args).unwrap();
312+ assert_eq!(at("info", &[]).path, base);
313+ let token = at("createToken", &[json!("read"), json!(300)]);
314+ assert_eq!(token.body.unwrap(), json!({ "scope": "read", "ttl": 300 }));
315+ assert_eq!(at("log", &[json!({ "ref": "fix/#1", "limit": 20 })]).path, format!("{base}/log?ref=fix%2F%231&limit=20"));
316+ assert_eq!(at("readCommit", &[json!("abc")]).path, format!("{base}/commits/abc"));
317+ assert_eq!(at("readTree", &[json!("abc")]).path, format!("{base}/trees/abc"));
318+ let blob = at("readBlob", &[json!("abc")]);
319+ assert!(blob.bytes);
320+ let file = at("readFile", &[json!({ "ref": "main", "path": "src/a b.rs" })]);
321+ assert_eq!(file.path, format!("{base}/file?ref=main&path=src%2Fa%20b.rs"));
322+ assert!(file.bytes);
323+ let fork = at("fork", &[json!("pulls--pul_2"), json!({ "defaultBranchOnly": true })]);
324+ assert_eq!(fork.body.unwrap(), json!({ "name": "g1t-us-1/pulls--pul_2", "defaultBranchOnly": true }));
325+ }
326+
327+ #[test]
328+ fn answers_read_as_the_binding_would_have_returned_them() {
329+ let json_route = route("g1t", Some("r"), "readTree", &[json!("a")]).unwrap();
330+ let blob = route("g1t", Some("r"), "readBlob", &[json!("a")]).unwrap();
331+ assert_eq!(answer(&json_route, 200, b"[]".to_vec()), Answer::Json(json!([])));
332+ assert_eq!(answer(&json_route, 200, b"null".to_vec()), Answer::Null);
333+ assert_eq!(answer(&blob, 200, b"hi".to_vec()), Answer::Bytes(b"hi".to_vec()));
334+ assert_eq!(answer(&blob, 404, b"{}".to_vec()), Answer::Null);
335+ assert_eq!(
336+ answer(&json_route, 404, br#"{"code":"NOT_FOUND","message":"no repository g1t/r"}"#.to_vec()),
337+ Answer::Error { code: Some("NOT_FOUND".into()), message: "no repository g1t/r".into() }
338+ );
339+ assert_eq!(
340+ answer(&json_route, 409, br#"{"code":"ALREADY_EXISTS","message":"x"}"#.to_vec()),
341+ Answer::Error { code: Some("ALREADY_EXISTS".into()), message: "x".into() }
342+ );
343+ // A 5xx may pass, as the binding's INTERNAL_ERROR.
344+ assert!(matches!(answer(&json_route, 502, Vec::new()), Answer::Error { code: Some(code), .. } if code == "INTERNAL_ERROR"));
345+ assert!(matches!(answer(&json_route, 200, b"<html>".to_vec()), Answer::Error { code: None, .. }));
346+ }
347+
348+ #[test]
349+ fn writes_are_told_apart_from_reads() {
350+ assert!(writes(None, "create", &[json!("x")]));
351+ assert!(writes(None, "delete", &[json!("x")]));
352+ assert!(!writes(None, "get", &[json!("x")]));
353+ assert!(writes(Some("r"), "fork", &[json!("y")]));
354+ assert!(writes(Some("r"), "createToken", &[json!("write"), json!(60)]));
355+ assert!(!writes(Some("r"), "createToken", &[json!("read"), json!(60)]));
356+ assert!(!writes(Some("r"), "readBlob", &[json!("a")]));
357+ }
358+}
+4−0
279279 /// Keeps a working copy's head in its repository, then removes its git
280280 /// data. A failure before the removal leaves everything as it was.
281281 async fn retire(&self, fork: &Repo) -> Result<()> {
282+ // Being copied to another namespace (moves.rs): the next sweep.
283+ if let Some(reason) = crate::registry::paused(&fork.id, now_ms()) {
284+ return Err(worker::Error::RustError(format!("paused: {reason}")));
285+ }
282286 let key = store_key(fork);
283287 let parent = match &fork.fork_of {
284288 Some(id) => self.registry.by_id(id).await?,
+5−4
824824 let query = request.url()?.query().map(|query| format!("?{query}")).unwrap_or_default();
825825 let url = format!("{}/{}{query}", access.remote, git.endpoint);
826826 let method = request.method();
827− let (namespace, _) = crate::store::locate(&crate::store::key_from_remote(&access.remote).unwrap_or_default());
827+ // Its own health and breaker: the fallback store's apart from Artifacts'.
828+ let namespace = crate::store::health_namespace(&access.remote);
828829
829830 if method == Method::Post && git.endpoint == "git-receive-pack" {
830831 return push(request, &url, headers, protected, limits, scan, &namespace).await;
870871 attempt += 1;
871872 }
872873 (_, Some(failure)) => {
873− let busy = Busy { rate_limited: failure == Failure::RateLimited, retry_after: 5 };
874+ let busy = Busy { rate_limited: failure == Failure::RateLimited, retry_after: 5, read_only: false };
874875 return Ok(Push::Forwarded(Forwarded {
875876 response: busy_response(busy)?,
876877 pushed: Vec::new(),
10291030 (Ok(response), None) => response,
10301031 (Ok(response), Some(Failure::RateLimited)) => {
10311032 drop(response);
1032− busy_response(Busy { rate_limited: true, retry_after: 5 })?
1033+ busy_response(Busy { rate_limited: true, retry_after: 5, read_only: false })?
10331034 }
10341035 (Ok(response), Some(_)) => response,
10351036 (Err(error), _) => {
10361037 worker::console_error!("a push did not reach the store: {error}");
1037− busy_response(Busy { rate_limited: false, retry_after: 5 })?
1038+ busy_response(Busy { rate_limited: false, retry_after: 5, read_only: false })?
10381039 }
10391040 };
10401041 Ok(Push::Forwarded(Forwarded {
+103−0
259259 Ok(reported(&report, &reference, true))
260260 }
261261
262+/// The commands at the start of a receive-pack request for many refs:
263+/// `(name, old, new)`, `new` the zero id to delete.
264+fn commands_block(commands: &[crate::mirror::Command]) -> Vec<u8> {
265+ let deletes = commands.iter().any(|(_, _, new)| new == ZERO_ID);
266+ let capabilities = if deletes { "report-status delete-refs" } else { "report-status" };
267+ let mut body = Vec::new();
268+ for (index, (name, old, new)) in commands.iter().enumerate() {
269+ let old = old.as_deref().unwrap_or(ZERO_ID);
270+ let line = if index == 0 { format!("{old} {new} {name}\0 {capabilities}\n") } else { format!("{old} {new} {name}\n") };
271+ body.extend(pkt_line(&line));
272+ }
273+ body.extend_from_slice(FLUSH);
274+ body
275+}
276+
277+/// Makes `target`'s refs match `commands`, fetching from `source` the
278+/// objects `wants` names that `haves` (commits the target holds) do not
279+/// reach. The pack streams from one into the other, never held whole, so
280+/// a whole repository can be copied this way (moves.rs). `Err` in the
281+/// inner result says which refs the target refused, and why.
282+pub(crate) async fn copy_refs(
283+ source: &GitAccess,
284+ target: &GitAccess,
285+ commands: &[crate::mirror::Command],
286+ wants: &[String],
287+ haves: &[String],
288+) -> Result<std::result::Result<(), String>> {
289+ if commands.is_empty() {
290+ return Ok(Ok(()));
291+ }
292+ let head = commands_block(commands);
293+ let sends_pack = commands.iter().any(|(_, _, new)| new != ZERO_ID);
294+ let report = if wants.is_empty() {
295+ // Every object is there already: only refs move.
296+ let mut body = head;
297+ if sends_pack {
298+ body.extend_from_slice(EMPTY_PACK);
299+ }
300+ post(target, "git-receive-pack", body).await?
301+ } else {
302+ let request = crate::mirror::upload_request(wants, haves);
303+ let sent = request.len() as u64;
304+ let mut fetched = send(source, "git-upload-pack", Uint8Array::from(request.as_slice()).into(), sent).await?;
305+ let failure: Rc<RefCell<Option<String>>> = Rc::default();
306+ let demux = Rc::new(RefCell::new(Sideband::default()));
307+ let pack = {
308+ let failure = failure.clone();
309+ let demux = demux.clone();
310+ fetched.stream()?.map(move |chunk| {
311+ let chunk = chunk?;
312+ demux.borrow_mut().feed(&chunk).map_err(|why| {
313+ *failure.borrow_mut() = Some(why.clone());
314+ Error::RustError(why)
315+ })
316+ })
317+ };
318+ let end = {
319+ let failure = failure.clone();
320+ let demux = demux.clone();
321+ futures_util::stream::once(async move {
322+ demux.borrow().finish().map(|()| Vec::new()).map_err(|why| {
323+ *failure.borrow_mut() = Some(why.clone());
324+ Error::RustError(why)
325+ })
326+ })
327+ };
328+ let body = futures_util::stream::once(async move { Ok::<Vec<u8>, Error>(head) })
329+ .chain(pack)
330+ .chain(end)
331+ .filter(|chunk| futures_util::future::ready(!matches!(chunk, Ok(bytes) if bytes.is_empty())));
332+ let pushed = send(target, "git-receive-pack", crate::git_http::stream_body(body)?, 0).await;
333+ if let Some(why) = failure.borrow_mut().take() {
334+ return Err(Error::RustError(why));
335+ }
336+ let report = pushed?.bytes().await?;
337+ let moved = demux.borrow().pack_bytes;
338+ if let (Some(from), Some(to)) = (crate::store::key_from_remote(&source.remote), crate::store::key_from_remote(&target.remote)) {
339+ meters::record_bytes("internal.git.fetch", &from, 0, moved);
340+ meters::record_bytes("internal.git.receive_pack", &to, moved, 0);
341+ }
342+ report
343+ };
344+ let problems = crate::mirror::refused(&report, commands);
345+ Ok(if problems.is_empty() { Ok(()) } else { Err(problems.join("; ")) })
346+}
347+
262348 #[cfg(test)]
263349 mod tests {
264350 use super::*;
265351
266352 #[test]
353+ fn many_refs_are_moved_by_one_block_of_commands() {
354+ let a = "c71546fcd893ef8b0f57388b65e620d759705dda".to_owned();
355+ let b = "4807077b296e6edbf410d55e72749d3e1170c291".to_owned();
356+ let block = String::from_utf8(commands_block(&[
357+ ("refs/heads/main".to_owned(), None, a.clone()),
358+ ("refs/tags/v1".to_owned(), Some(a.clone()), b.clone()),
359+ ]))
360+ .unwrap();
361+ // Capabilities ride on the first command only.
362+ assert!(block.contains(&format!("{ZERO_ID} {a} refs/heads/main\0 report-status\n")));
363+ assert!(block.contains(&format!("{a} {b} refs/tags/v1\n")));
364+ assert!(block.ends_with("0000"));
365+ let deleting = String::from_utf8(commands_block(&[("refs/heads/old".to_owned(), Some(a), ZERO_ID.to_owned())])).unwrap();
366+ assert!(deleting.contains("delete-refs"));
367+ }
368+
369+ #[test]
267370 fn one_ref_is_moved_by_one_command() {
268371 let new = "4807077b296e6edbf410d55e72749d3e1170c291";
269372 let body = String::from_utf8(command("refs/pull/pul_1/head", None, new, true)).unwrap();
+136−2
1111 mod coalesce;
1212 mod commit_file;
1313 mod diff;
14+mod fallback;
1415 mod forks;
1516 mod git_http;
1617 mod git_ops;
2122 mod listing;
2223 mod meters;
2324 mod mirror;
25+mod moves;
26+mod namespaces;
2427 mod pack_cache;
2528 mod pack_limits;
2629 mod refs;
192195 /// to a push too large to scan.
193196 repo_limit: u64,
194197 large_pushes: git_http::LargePushes,
195− /// Which git store namespace new repositories go in (shards.rs).
198+ /// Which git store namespace new repositories go in (shards.rs), and
199+ /// the most each should hold (`ARTIFACTS_NAMESPACE_LIMITS`).
196200 placement: shards::Placement,
201+ limits: HashMap<String, u64>,
197202 /// What isolates share: answers that list refs (refs_cache.rs).
198203 shared: Option<Rc<shared::Shared>>,
199204 /// Packs for fresh clones (pack_cache.rs); `None` without the bucket.
551556 website: None,
552557 archived_at: None,
553558 };
554− let namespace = self.placement.place(&repo.id, shards::Residency::Anywhere, &self.store.namespaces());
559+ let namespace = match self.place(&repo).await? {
560+ Ok(namespace) => namespace,
561+ Err(unplaced) => return Ok(Outcome::fail(FailureCode::Conflict, unplaced.message())),
562+ };
555563 self.registry
556564 .claim_store_key(&repo, namespace.as_deref(), &self.store.default_namespace())
557565 .await?;
636644 Ok(Outcome::Ok(repo))
637645 }
638646
647+ /// Where a workspace keeps its data, asked of identity only when an EU
648+ /// namespace is configured: without one, every workspace's
649+ /// repositories go anywhere and identity is never asked.
650+ async fn residency_of(&self, workspace: &str) -> Result<shards::Residency> {
651+ if self.placement.eu.is_none() {
652+ return Ok(shards::Residency::Anywhere);
653+ }
654+ let Some(identity) = &self.identity else {
655+ return Ok(shards::Residency::Anywhere);
656+ };
657+ let residency: Option<g1t_contracts::identity::DataResidency> = g1t_kit::call(
658+ identity,
659+ "workspace_residency",
660+ &g1t_contracts::identity::SlugArgs { slug: workspace.to_owned() },
661+ )
662+ .await?;
663+ Ok(match residency {
664+ Some(g1t_contracts::identity::DataResidency::Eu) => shards::Residency::Eu,
665+ _ => shards::Residency::Anywhere,
666+ })
667+ }
668+
669+ /// How each bound namespace stands (namespaces.rs).
670+ async fn standings(&self) -> Result<Vec<namespaces::Standing>> {
671+ let bound = self.store.namespaces();
672+ let default = self.store.default_namespace();
673+ let now = now_ms();
674+ let config = namespaces::Configured {
675+ bound: &bound,
676+ default: &default,
677+ placement: &self.placement,
678+ limits: &self.limits,
679+ on_fallback: &|namespace| self.store.on_fallback(&shards::compose(Some(namespace), "x", &default)),
680+ writable: &|namespace| self.store.writable(namespace),
681+ breaker_open: &|namespace| resilience::open_now(namespace, now),
682+ };
683+ let (held, recent) = futures_util::future::join(namespaces::held(&self.registry.db), namespaces::recent(&self.registry.db, now)).await;
684+ Ok(namespaces::standings(&config, &held?, &recent?))
685+ }
686+
687+ /// The namespace a new repository goes in (shards.rs): its workspace's
688+ /// residency, then how each namespace stands, read only when there is
689+ /// a choice to make. `Ok(None)` for the default.
690+ async fn place(&self, repo: &Repo) -> Result<std::result::Result<Option<String>, shards::Unplaced>> {
691+ let residency = self.residency_of(&repo.namespace).await?;
692+ let bound = self.store.namespaces();
693+ let loads = if residency == shards::Residency::Anywhere && !self.placement.needs_loads(&bound) {
694+ // One namespace to choose from at most: nothing to read.
695+ bound
696+ .iter()
697+ .map(|namespace| shards::Load {
698+ namespace: namespace.clone(),
699+ bound: true,
700+ writable: self.store.writable(namespace),
701+ ..shards::Load::default()
702+ })
703+ .collect()
704+ } else {
705+ let default = self.store.default_namespace();
706+ let now = now_ms();
707+ let config = namespaces::Configured {
708+ bound: &bound,
709+ default: &default,
710+ placement: &self.placement,
711+ limits: &self.limits,
712+ on_fallback: &|namespace| self.store.on_fallback(&shards::compose(Some(namespace), "x", &default)),
713+ writable: &|namespace| self.store.writable(namespace),
714+ breaker_open: &|namespace| resilience::open_now(namespace, now),
715+ };
716+ namespaces::loads(&self.registry.db, &config, now).await?
717+ };
718+ Ok(self.placement.choose(&repo.id, residency, &loads))
719+ }
720+
721+ /// `storage_options`: what a workspace may choose about where its
722+ /// repositories are kept.
723+ fn storage_options(&self) -> StorageOptions {
724+ let bound = self.store.namespaces();
725+ StorageOptions { eu_available: self.placement.eu_available(&bound, |namespace| self.store.writable(namespace)) }
726+ }
727+
639728 async fn tree(&self, a: TreeArgs) -> Result<Outcome<TreeView>> {
640729 let Some(repo) = self.readable(&a.path, &a.viewer).await? else {
641730 return Ok(not_found());
9781067 let Some(repo) = self.registry.by_id(&a.repo_id).await? else {
9791068 return Ok(not_found());
9801069 };
1070+ let repo = match self.unpaused(repo).await? {
1071+ Ok(repo) => repo,
1072+ Err((code, message)) => return Ok(Outcome::fail(code, message)),
1073+ };
9811074 self.live(&repo).await?;
9821075 let git = self.store.open(&store_key(&repo)).await?;
9831076 let Some(old) = git
10141107 if let Some((code, message)) = lifecycle::archived_refusal(&source) {
10151108 return Ok(Outcome::fail(code, message));
10161109 }
1110+ // Its working copy is made in its namespace: not while it moves.
1111+ let source = match self.unpaused(source).await? {
1112+ Ok(source) => source,
1113+ Err((code, message)) => return Ok(Outcome::fail(code, message)),
1114+ };
10171115 let now = now_ms();
10181116 let fork = Repo {
10191117 id: new_id("rep", now),
11821280 }
11831281 }
11841282 };
1283+ // A push, or a credential to push with, waits while the repository
1284+ // moves between namespaces (moves.rs), and goes to where it is now.
1285+ if write {
1286+ return Ok(match self.unpaused(repo).await? {
1287+ Ok(repo) => Outcome::Ok(repo),
1288+ Err((code, message)) => Outcome::fail(code, format!("{message}\n")),
1289+ });
1290+ }
11851291 Ok(Outcome::Ok(repo))
11861292 }
11871293
12101316 if let Some((code, message)) = lifecycle::archived_refusal(&target) {
12111317 return Ok(Outcome::fail(code, message));
12121318 }
1319+ // Moving between namespaces: wait for it (moves.rs). Both are read
1320+ // again once it is done, for their new keys.
1321+ let (source, target) = match (self.unpaused(source).await?, self.unpaused(target).await?) {
1322+ (Ok(source), Ok(target)) => (source, target),
1323+ (Err((code, message)), _) | (_, Err((code, message))) => return Ok(Outcome::fail(code, message)),
1324+ };
12131325
12141326 let branch = &target.default_branch;
12151327 let from_fork = source.id != target.id;
15111623 let body = if !write && !get { Some(request.bytes().await?) } else { None };
15121624 // What it asks the store, for the meters (meters.rs).
15131625 let call = git_ops::classify(git.service, git.endpoint, get, body.as_deref());
1626+ // Answers kept from the usual store may name refs the fallback
1627+ // store does not have (fallback.rs): none are used, or kept.
1628+ let fallback = self.store.on_fallback(&key);
15141629 // An answer that lists refs may have been kept: see refs_cache.rs.
15151630 let kept_key = refs_cache::kind(git, get, protocol, body.as_deref())
1631+ .filter(|_| !fallback)
15161632 .zip(refs_cache::usable(registry::refs_state(&repo.id), now_ms()))
15171633 .map(|(kind, version)| {
15181634 refs_cache::Key::new(&repo.id, version, default_branch.as_deref(), protocol, &kind)
15221638 let pack_key = self
15231639 .packs
15241640 .as_ref()
1641+ .filter(|_| !fallback)
15251642 .and_then(|_| {
15261643 let encoding = request.headers().get("content-encoding").ok().flatten();
15271644 pack_cache::cacheable(git, get, protocol, encoding.as_deref(), body.as_deref())
19502067 env.var("ARTIFACTS_NEW_REPOS").ok().map(|value| value.to_string()).as_deref(),
19512068 env.var("ARTIFACTS_EU_NAMESPACE").ok().map(|value| value.to_string()).as_deref(),
19522069 ),
2070+ limits: shards::limits(env.var("ARTIFACTS_NAMESPACE_LIMITS").ok().map(|value| value.to_string()).as_deref()),
19532071 })
19542072 }
19552073
21752293 let a: meters::HealthArgs = args(body)?;
21762294 reply(&meters::health(&repos.registry.db, &a).await?)
21772295 }
2296+ // Where repositories may be kept, for a workspace's settings.
2297+ "storage_options" => reply(&repos.storage_options()),
2298+ // Services and operators only: how each namespace stands, and
2299+ // moving a repository between them (namespaces.rs, moves.rs).
2300+ "namespaces" => reply(&repos.standings().await?),
2301+ "move_repository" => reply(&repos.move_repository(args(body)?).await?),
2302+ "repository_moves" => {
2303+ let a: moves::ListMovesArgs = args(body)?;
2304+ reply(&repos.registry.moves(a.limit.unwrap_or(50)).await?)
2305+ }
21782306 _ => Response::error("Unknown method", 404),
21792307 } }
21802308 .await;
22312359 Ok(count) => worker::console_log!("repos: removed {count} pull request working copies"),
22322360 Err(error) => worker::console_error!("repos: the working copy sweep failed: {error}"),
22332361 }
2362+ // Repositories moving between namespaces, and old copies (moves.rs).
2363+ match repos.run_moves().await {
2364+ Ok(0) => {}
2365+ Ok(count) => worker::console_log!("repos: moved {count} repositories between namespaces"),
2366+ Err(error) => worker::console_error!("repos: the move sweep failed: {error}"),
2367+ }
22342368 meters::flush(&repos.registry.db).await;
22352369 }
22362370
+2−1
12731273 if let Some(refusal) = archived_refusal(&repo) {
12741274 return Ok(Err(refusal));
12751275 }
1276− Ok(Ok(repo))
1276+ // Moving between namespaces: wait for it (moves.rs).
1277+ self.unpaused(repo).await
12771278 }
12781279
12791280 /// `set_default_branch`: see `g1t_contracts::repos::SetDefaultBranchArgs`.
+9−0
416416 let Some(url) = import::clean_url(&a.url) else {
417417 return Ok(Outcome::fail(FailureCode::Invalid, "That is not an https repository address."));
418418 };
419+ // Catching up writes: it waits for a move between namespaces (moves.rs).
420+ let repo = if a.direction == MirrorDirection::Pull {
421+ match self.unpaused(repo).await? {
422+ Ok(repo) => repo,
423+ Err((code, message)) => return Ok(Outcome::fail(code, message)),
424+ }
425+ } else {
426+ repo
427+ };
419428 let scope = match a.direction {
420429 MirrorDirection::Pull => Scope::Write,
421430 MirrorDirection::Push => Scope::Read,
+0−0

Binary or large file; its contents are not shown.

+0−0

Binary or large file; its contents are not shown.

+0−0

Binary or large file; its contents are not shown.

+0−0

Binary or large file; its contents are not shown.

+0−0

Binary or large file; its contents are not shown.

+0−0

Binary or large file; its contents are not shown.

+0−0

Binary or large file; its contents are not shown.

This change is too large to show in full.