Skip to content
Request access

flagon-io/g1t

public
00

Where people and agents ship software together. The open-source git platform for the whole job: issues, agents, checks and deploys to the edge.

gitagentscloudflarerustforgemcp
128 linesCodeBlameRaw
1# Storage holds
2
3Owner: whoever answers a workspace whose public pushes are refused for room
4it does not use, or a fork whose source refuses pushes. The page is sudo →
5Platform → **Storage holds** (`/storage`); the design is in
6`services/repos/src/storage_holds.rs` and the docs' architecture page
7[Git](https://docs.g1t.sh/architecture/git/#public-storage-reservations).
8
9A write reserves public storage before it reaches the git store, and a
10native fork locks its source while it copies. One that is interrupted keeps
11its hold: a reservation counts against the workspace's 5 GB public cap, and
12a copy lock refuses every write to the source, private or public (each
13told "This repository is being copied into a fork. Try again in a few
14minutes; if it lasts more than an hour, g1t staff release it.", never that
15storage is full; a push is told the same in git's `remote:` lines; once the
16copy has taken longer than it should or did not finish cleanly, that it can
17take a few hours, below). A pull
18request g1t merges by itself stalls with that message, and a person's
19automatic merge turns off with it: neither is tried again by itself once the lock
20is released. Nothing expires a hold on a timer, on
21purpose: releasing one that is still in flight would let two writes take the
22same room.
23
24## A source that takes no writes
25
26A fork whose call to the store timed out or met a server error may still be
27copying, so it keeps its source's lock, and the source refuses every write
28until staff release it: at the soonest 75 minutes after the copy began (the
29lease, an hour, and 15 minutes' grace). The repos service reads a lock as
30stalled when its fork's reservation ended, when its fork holds no live
31reservation a minute after it locked, or once its lease has passed. Its
32writers are then told the copy has taken longer than it should, or did not
33finish cleanly, and that it can take a few hours, so do not leave it.
34
35- **Where it shows.** sudo's Overview has **Writes blocked** under Needs
36 attention: the repositories a copy lock has held longer than a copy
37 ordinarily takes (15 minutes), red once one did not finish cleanly. The
38 Storage holds page lists them under **Writes blocked**, with how long each
39 has been blocked, how the fork's call ended, and when its lock becomes
40 releasable. The repos service also logs `copy_lock_kept` (an error, with
41 the source, the fork and the store's error) when it keeps a lock this
42 way.
43- **What to do.** Inspect the lock: the fork's refs in the store say whether
44 the copy landed. Once it is releasable, release the **lock first**, with
45 what you saw; the source takes writes again at once. Then release the
46 fork's ended reservation the same way. In that order because the ended
47 reservation is releasable at once while the lock waits for its lease and
48 grace, and the reservation is what tells sudo how the fork's call ended:
49 released first, the Writes blocked row loses it (the lock still reads as
50 stalled, as one whose fork holds no live reservation). Whether the fork
51 kept the copy, its `stored_bytes` stays as it was: note it in the reason,
52 and use Drift.
53- **A fork that outlived its lock.** If the store's call returns after staff
54 released the lock, the fork is counted at its source's measured size, a
55 lower bound that never overcharges: it is the source's `stored_bytes`
56 read right after the lock, which only grows, and a fork counts in full by
57 policy, even of its default branch only. The fork is counted before the
58 lock is let go, so an unlock that fails (logged as `copy_unlock_failed`,
59 the lock left for staff, reading as stalled) never leaves it uncounted.
60 The repos service logs
61 `fork_outlived_copy_lock` (a warning, with the fork and the bytes
62 counted): the source may have taken writes the fork holds and the count
63 misses. Drift measures how far it is out.
64
65## Provably abandoned
66
67The repos service decides, from what the write recorded:
68
69| Hold | Abandoned when |
70| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
71| A reservation marked `ended` | At once: the write stopped without knowing whether the store kept it, and sends nothing after |
72| A reservation still `held` | 15 minutes after its lease, an hour from when it was taken. A push or a branch copied chunk by chunk is cut short at its lease; a write of one pack, and a native fork's one call to the store, are not, and are taken as answered by then |
73| Unledgered bytes (reserved before migration 0023, with no row) | 24 hours after the ledger began (`storage_ledger_epoch`): older code had no lease |
74| A copy lock with a lease | 15 minutes after its lease. A fork whose call timed out or met a server error keeps its lock until then: the store may still be copying |
75| A copy lock without one (older code) | 24 hours after the ledger began, and at least its lease and grace after it was taken |
76
77Release is offered only then, and the repos service checks again, in the
78same batch that releases.
79
80## Checklist
81
821. **Find the hold.** On Storage holds, the workspace's repository, oldest
83 first. A copy lock names the fork it was copying into.
842. **Inspect it.** Read what the store lists for each repository (refs, the
85 default branch's commit) and what g1t counts (stored, reserved, how much
86 of it the ledger accounts for). For a push that ended, each ref it asked
87 for says **landed** when the store has it: the push may have been stored
88 though its answer was an error (a 500, or a 200 with a fatal error,
89 whose outcome quotes it), or a later push may have set the same ref.
903. **Release it**, with a reason that says what you saw ("ended with a 500;
91 main is not at the commit it asked for"). The reason and the inspection
92 are kept in the audit row, listed under **Released**. The release is
93 also in sudo's audit log (**Storage hold released**, under the
94 repository's workspace, or `storage` if its row is gone), naming the
95 row. The repos service keeps that line in the release's own batch
96 (`storage_recovery_audits`) and sends it to billing right after, keyed by
97 the row, then every hour until billing confirms it, so it is written
98 once even if an answer is lost or the page stops waiting. If billing has
99 not confirmed it yet, the page and the row under **Released** say so,
100 with billing's last answer: the release still stands, so do not release
101 it again. A line that stays unconfirmed past a few hours means billing
102 keeps refusing it: the repos logs have `storage_release_unaudited` with
103 the error.
1044. **If it is not provably abandoned yet**, the page says when it will be.
105 Wait: a write that may still be sending is never released by hand. A
106 workspace that needs room sooner can delete what it does not need.
1075. **If a push ended but landed**, its bytes are not in `stored_bytes`,
108 which is a lower bound by design and what private storage is billed
109 from. A release never changes it, and neither does anything here: note
110 it in the reason, and use Drift to see how far a repository is out.
111
112## Drift
113
114**Measure** streams a small random sample of repositories (1 to 20, those
115over 200 MB passed over) from the store: the pack it sends for every ref,
116plus what each pull request's working copy holds beyond it, beside
117`stored_bytes`. It reads only. Above zero, g1t counts less than the store
118holds: pushes made with a credential straight to the store (sandboxes'
119`git_access`), ended pushes that landed, g1t's own refs (a pull
120request's test merge, `refs/pull/N/merge`), and a pull request's head
121copied in from someone's fork are not counted. Below zero,
122g1t counted objects that a later push made unreachable (a force push), or
123packs that sent objects the store already had: g1t asks for whole packs.
124Before #307, g1t also counted writes the store refused (a merge or update
125whose branch had moved, an `ng`, a failed unpack); since, a refused write
126gives its reservation back and counts nothing. Large or growing
127drift is for the owner of billing to decide on; nothing corrects it
128automatically.