Skip to content
Sign up

flagon-io/g1t

public
0
1,576 linesCodeBlameRaw

Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.

Agents as a team: lifecycle, merge queue, billing and a new shell1//! The storage that actually holds git repositories.
2//!
3//! The service depends on the [`GitStore`] and [`GitRepo`] ports;
4//! [`ArtifactsStore`] is the adapter for Cloudflare Artifacts.
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily5//!
6//! Every call on the binding goes through [`invoke`]: it is counted
7//! (meters.rs), timed for the store's health, refused at once while its
8//! namespace's breaker is open, and tried again after a failure that may
9//! pass when it only reads (resilience.rs). Repositories may live in
10//! several namespaces (shards.rs).
Agents as a team: lifecycle, merge queue, billing and a new shell11
12use g1t_contracts::repos::{Branch, Commit, EntryKind, GitAccess, Signature, TreeEntry};
13use g1t_contracts::time::rfc3339;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily14use g1t_kit::js::{self, Thrown};
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms15use serde::{Deserialize, Serialize};
16use std::cell::RefCell;
17use std::collections::HashMap;
18use std::rc::Rc;
Agents as a team: lifecycle, merge queue, billing and a new shell19use worker::js_sys::{Reflect, Uint8Array};
20use worker::wasm_bindgen::{JsCast, JsValue};
21use worker::{Env, Result};
22
Merge branch 'worktree-agent-a2013627e5ea4ab13'23use crate::fallback;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily24use crate::meters::{self, Outcome};
25use crate::resilience::{self, Admit, Busy, Failure};
26use crate::shards;
27
28/// Credentials that never leave this service: g1t has already decided who
29/// may do what before one is used. They live an hour and are used for 50
30/// minutes, so each one used has at least ten minutes left.
31const INTERNAL_TTL_SECONDS: u32 = 3_600;
32const INTERNAL_REUSE_MS: u64 = 50 * 60 * 1000;
Merge branch 'worktree-agent-a57ff9fecefa1eaf7'33/// Credentials handed out: to a nightly backup's sandbox (backups.rs), and
34/// by `git_access`, which nothing deployed asks yet (git over SSH will).
35/// Other sandboxes never get one: they use g1t's git endpoints. Five
36/// minutes, used for three, so whoever gets one has at least two.
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily37const HANDOUT_TTL_SECONDS: u32 = 300;
38const HANDOUT_REUSE_MS: u64 = 180_000;
39/// How long a handed-out credential stays valid, in milliseconds.
40pub const CREDENTIAL_LIFE_MS: u64 = HANDOUT_TTL_SECONDS as u64 * 1000;
Agents as a team: lifecycle, merge queue, billing and a new shell41
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms42#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
Agents as a team: lifecycle, merge queue, billing and a new shell43pub enum Scope {
44 Read,
45 Write,
46}
47
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms48impl Scope {
49 fn as_str(self) -> &'static str {
50 match self {
51 Scope::Read => "read",
52 Scope::Write => "write",
53 }
54 }
55}
56
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily57/// Who a credential is for.
58#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
59pub enum Use {
60 /// This service, talking to the store itself.
61 Internal,
62 /// Someone outside it, through `git_access`.
63 Handout,
64}
65
66impl Use {
67 pub fn ttl_seconds(self) -> u32 {
68 match self {
69 Use::Internal => INTERNAL_TTL_SECONDS,
70 Use::Handout => HANDOUT_TTL_SECONDS,
71 }
72 }
73
74 pub fn reuse_ms(self) -> u64 {
75 match self {
76 Use::Internal => INTERNAL_REUSE_MS,
77 Use::Handout => HANDOUT_REUSE_MS,
78 }
79 }
80
81 fn as_str(self) -> &'static str {
82 match self {
83 Use::Internal => "internal",
84 Use::Handout => "handout",
85 }
86 }
87}
88
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms89/// Where a credential handed out came from, for `Server-Timing`.
90#[derive(Clone, Copy, PartialEq, Eq, Debug)]
91pub enum Kept {
92 /// This isolate made it, or had it from another, a moment ago.
93 Isolate,
94 /// Another isolate made it and shared it.
95 Shared,
96}
97
98impl Kept {
99 pub fn as_str(self) -> &'static str {
100 match self {
101 Kept::Isolate => "isolate",
102 Kept::Shared => "shared",
103 }
104 }
105}
106
Agents as a team: lifecycle, merge queue, billing and a new shell107/// A place repositories live. `key` is the store's own name for a repo.
108#[allow(async_fn_in_trait)]
109pub trait GitStore {
110 type Repo: GitRepo;
111
112 /// Creates an empty repository. Succeeds if it already exists.
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily113 async fn create(&self, key: &str, description: Option<&str>, default_branch: &str) -> Result<()>;
Agents as a team: lifecycle, merge queue, billing and a new shell114 async fn open(&self, key: &str) -> Result<Self::Repo>;
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms115 /// A credential for `key` made a moment ago, if the store keeps one.
116 async fn kept_access(&self, _key: &str, _scope: Scope) -> Option<(GitAccess, Kept)> {
117 None
118 }
119 /// A new credential for `key`, which the store may keep for next time.
120 async fn mint_access(&self, key: &str, scope: Scope) -> Result<GitAccess> {
121 self.open(key).await?.access(scope).await
122 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily123 /// A remote URL and credential for this service's own use. A store may
124 /// hand out one it made a moment ago.
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms125 async fn access(&self, key: &str, scope: Scope) -> Result<GitAccess> {
126 match self.kept_access(key, scope).await {
127 Some((access, _)) => Ok(access),
128 None => self.mint_access(key, scope).await,
129 }
130 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily131 /// A short-lived credential for someone outside this service.
132 async fn handout(&self, key: &str, scope: Scope) -> Result<GitAccess> {
133 self.access(key, scope).await
134 }
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms135 /// Stops handing out the credentials it keeps for `key`: the store
136 /// turned one down, or the repository is gone.
137 async fn forget_access(&self, _key: &str) {}
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look138 /// Removes a repository and everything in it, for good. Succeeds if it
139 /// is already gone.
140 async fn delete(&self, key: &str) -> Result<()>;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily141 /// The namespaces new repositories may be placed in (shards.rs).
142 fn namespaces(&self) -> Vec<String> {
143 vec![shards::DEFAULT_NAMESPACE.to_owned()]
144 }
145 /// The namespace bound as the default.
146 fn default_namespace(&self) -> String {
147 shards::DEFAULT_NAMESPACE.to_owned()
148 }
Merge branch 'worktree-agent-a2013627e5ea4ab13'149 /// Whether the repository at `key` is served from the fallback store
150 /// now (fallback.rs): answers kept from the usual store may name refs
151 /// it does not have, so none are used.
152 fn on_fallback(&self, _key: &str) -> bool {
153 false
154 }
155 /// Whether `namespace` takes writes now: not while it is served from a
156 /// read-only fallback.
157 fn writable(&self, _namespace: &str) -> bool {
158 true
159 }
Agents as a team: lifecycle, merge queue, billing and a new shell160}
161
162/// One open repository.
163#[allow(async_fn_in_trait)]
164pub trait GitRepo {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily165 /// A remote URL and credential for git itself, for this service.
Agents as a team: lifecycle, merge queue, billing and a new shell166 async fn access(&self, scope: Scope) -> Result<GitAccess>;
167 /// Every branch and the commit it points to.
168 async fn branches(&self) -> Result<Vec<Branch>>;
169 /// Newest first along the first-parent chain; empty for an unknown ref.
170 async fn log(&self, git_ref: &str, limit: u32) -> Result<Vec<Commit>>;
171 /// The parents of a commit, or `None` if the commit does not exist.
172 async fn parents(&self, commit_hash: &str) -> Result<Option<Vec<String>>>;
173 async fn read_tree(&self, tree_hash: &str) -> Result<Option<Vec<TreeEntry>>>;
174 async fn read_blob(&self, blob_hash: &str) -> Result<Option<Vec<u8>>>;
Merge branch 'main' into worktree-agent-a69aeabc4b0deeb97175 /// A blob's size in bytes, `None` when it is missing. By default its
176 /// bytes are read; a store that can say less does.
177 async fn blob_size(&self, blob_hash: &str) -> Result<Option<u64>> {
178 Ok(self.read_blob(blob_hash).await?.map(|bytes| bytes.len() as u64))
179 }
Agents as a team: lifecycle, merge queue, billing and a new shell180 /// `None` when the ref or path does not resolve to a file.
181 async fn read_file(&self, git_ref: &str, path: &str) -> Result<Option<Vec<u8>>>;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily182 /// Makes a copy of this repository under `target_key`, in the same
183 /// namespace.
Agents as a team: lifecycle, merge queue, billing and a new shell184 async fn fork(&self, target_key: &str) -> Result<()>;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily185 /// The version of the repository's refs (registry.rs `RefsState`),
186 /// when answers about branches may be kept under it (R9). Reads by
187 /// branch name are then kept until the version moves.
188 fn at_refs_version(&mut self, _version: Option<u64>) {}
189}
190
191thread_local! {
192 /// The namespace bound to `ARTIFACTS`, for keys that name none.
193 static DEFAULT_NS: RefCell<String> = RefCell::new(shards::DEFAULT_NAMESPACE.to_owned());
194 /// Where each namespace's remotes start: `https://<account>.artifacts.cloudflare.net/git/<namespace>/`.
195 static REMOTE_PREFIX: RefCell<HashMap<String, String>> = RefCell::new(HashMap::new());
196}
197
198/// The namespace and name a store key stands for.
199pub fn locate(key: &str) -> (String, String) {
200 let (namespace, name) = shards::split(key);
201 let namespace = namespace.map_or_else(|| DEFAULT_NS.with(|ns| ns.borrow().clone()), str::to_owned);
202 (namespace, name.to_owned())
203}
204
Merge branch 'worktree-agent-a2013627e5ea4ab13'205thread_local! {
206 /// Where the fallback store's remotes start, when one is configured.
207 static FALLBACK_BASE: RefCell<Option<String>> = const { RefCell::new(None) };
208}
209
210/// Whose breaker and health git requests to `remote` count toward: its
211/// namespace's, or `<namespace>@fallback` for the fallback store's.
212pub fn health_namespace(remote: &str) -> String {
213 let (namespace, _) = locate(&key_from_remote(remote).unwrap_or_default());
214 let on_fallback = FALLBACK_BASE.with(|base| base.borrow().as_deref().is_some_and(|base| remote.starts_with(base)));
215 if on_fallback { format!("{namespace}@fallback") } else { namespace }
216}
217
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily218/// The store key a git remote is for, from its last two path segments.
219pub fn key_from_remote(remote: &str) -> Option<String> {
220 let path = remote.trim_end_matches('/');
221 let path = path.strip_suffix(".git").unwrap_or(path);
222 let (rest, name) = path.rsplit_once('/')?;
223 let namespace = rest.rsplit('/').next()?;
224 let default = DEFAULT_NS.with(|ns| ns.borrow().clone());
225 Some(shards::compose(Some(namespace), name, &default))
226}
227
228/// Where a namespace's remotes start, learned from one remote the store
229/// gave for `name`.
230fn learn_prefix(remote: &str, name: &str) -> Option<String> {
231 remote.strip_suffix(&format!("{name}.git")).filter(|prefix| prefix.ends_with('/')).map(str::to_owned)
Agents as a team: lifecycle, merge queue, billing and a new shell232}
233
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily234/// A repository's remote, from where its namespace's remotes start.
235fn remote_from(prefix: &str, name: &str) -> String {
236 format!("{prefix}{name}.git")
237}
238
239struct Namespace {
240 name: String,
Merge branch 'worktree-agent-a2013627e5ea4ab13'241 /// Its binding, for every call: Artifacts, or the fallback store.
242 target: Target,
243}
244
245/// What a call on the store goes to: an Artifacts binding or one of its
246/// repository handles, or the fallback store (fallback.rs) for a
247/// namespace, or for one repository in it.
248#[derive(Clone)]
249enum Target {
250 Js(JsValue),
251 Fallback {
252 settings: Rc<fallback::Settings>,
253 namespace: String,
254 repo: Option<String>,
255 },
256}
257
258impl Target {
259 fn is_fallback(&self) -> bool {
260 matches!(self, Target::Fallback { .. })
261 }
262
263 /// Whose breaker and health a call counts toward: the fallback store's
264 /// own, so an Artifacts outage never holds it back.
265 fn health_name(&self, namespace: &str) -> String {
266 if self.is_fallback() { format!("{namespace}@fallback") } else { namespace.to_owned() }
267 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily268}
269
270pub struct ArtifactsStore {
271 namespaces: Rc<Vec<Namespace>>,
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms272 /// Where isolates share the credentials they make; see shared.rs.
273 shared: Option<Rc<crate::shared::Shared>>,
Agents as a team: lifecycle, merge queue, billing and a new shell274}
275
276impl ArtifactsStore {
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms277 pub fn new(env: &Env, shared: Option<Rc<crate::shared::Shared>>) -> Result<Self> {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily278 let config = env.var("ARTIFACTS_NAMESPACES").ok().map(|value| value.to_string());
Merge branch 'worktree-agent-a2013627e5ea4ab13'279 let text = |name: &str| env.var(name).ok().map(|value| value.to_string());
280 let secret = env.secret("GIT_FALLBACK_SECRET").ok().map(|value| value.to_string());
281 let fallback = fallback::Settings::from_vars(
282 text("GIT_FALLBACK_URL").as_deref(),
283 secret.as_deref(),
284 text("GIT_FALLBACK_NAMESPACES").as_deref(),
285 text("GIT_FALLBACK_WRITES").as_deref(),
286 )
287 .map(Rc::new);
288 FALLBACK_BASE.with(|base| *base.borrow_mut() = fallback.as_ref().map(|settings| format!("{}/git/", settings.url)));
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily289 let mut namespaces = Vec::new();
290 for (binding, name) in shards::bindings(config.as_deref()) {
Merge branch 'worktree-agent-a2013627e5ea4ab13'291 // Served from the fallback store, by configuration.
292 if let Some(settings) = fallback.as_ref().filter(|settings| settings.serves(&name)) {
293 worker::console_log!("git store {name}: served from the fallback store");
294 let target = Target::Fallback { settings: settings.clone(), namespace: name.clone(), repo: None };
295 namespaces.push(Namespace { name, target });
296 continue;
297 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily298 match js::binding(env, &binding) {
Merge branch 'worktree-agent-a2013627e5ea4ab13'299 Ok(value) => namespaces.push(Namespace { name, target: Target::Js(value) }),
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily300 // The default binding is required; the others are optional.
301 Err(error) if binding == shards::DEFAULT_BINDING => return Err(error),
302 Err(_) => worker::console_error!("ARTIFACTS_NAMESPACES names {binding}, which is not bound"),
303 }
304 }
305 if let Some(default) = namespaces.first() {
306 DEFAULT_NS.with(|ns| ns.borrow_mut().clone_from(&default.name));
307 }
308 // Optional: where remotes start, `https://<account>.artifacts.cloudflare.net/git`,
309 // so the first credential an isolate makes needs no `info()` either.
310 if let Ok(base) = env.var("ARTIFACTS_REMOTE_BASE") {
311 let base = base.to_string().trim_end_matches('/').to_owned();
312 if base.starts_with("https://") {
313 REMOTE_PREFIX.with(|prefixes| {
314 let mut prefixes = prefixes.borrow_mut();
315 for namespace in &namespaces {
316 prefixes.entry(namespace.name.clone()).or_insert_with(|| format!("{base}/{}/", namespace.name));
317 }
318 });
319 }
320 }
321 Ok(Self { namespaces: Rc::new(namespaces), shared })
322 }
323
Merge branch 'worktree-agent-a2013627e5ea4ab13'324 fn binding(&self, namespace: &str) -> Result<&Target> {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily325 self.namespaces
326 .iter()
327 .find(|candidate| candidate.name == namespace)
Merge branch 'worktree-agent-a2013627e5ea4ab13'328 .map(|found| &found.target)
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily329 .ok_or_else(|| worker::Error::RustError(format!("git store namespace {namespace} is not bound")))
330 }
Merge branch 'worktree-agent-a2013627e5ea4ab13'331
332 /// The fallback store's settings, when `namespace` is served from it.
333 fn fallback_of(&self, namespace: &str) -> Option<&fallback::Settings> {
334 match self.binding(namespace).ok()? {
335 Target::Fallback { settings, .. } => Some(settings),
336 Target::Js(_) => None,
337 }
338 }
339
340 /// The name credentials for `key` are kept under: those of the
341 /// fallback store never stand in for Artifacts' own, nor the reverse.
342 fn cred_key(&self, key: &str) -> String {
343 let (namespace, _) = locate(key);
344 cred_key(key, self.fallback_of(&namespace).is_some())
345 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily346}
347
Merge branch 'worktree-agent-a2013627e5ea4ab13'348/// See [`ArtifactsStore::cred_key`].
349fn cred_key(key: &str, on_fallback: bool) -> String {
350 if on_fallback { format!("fallback:{key}") } else { key.to_owned() }
351}
352
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily353/// A failed call on the binding.
354pub struct StoreError {
355 pub thrown: Thrown,
356 pub busy: Option<Busy>,
357}
358
359impl StoreError {
360 pub fn is(&self, code: &str) -> bool {
361 self.thrown.is(code)
Agents as a team: lifecycle, merge queue, billing and a new shell362 }
363}
364
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily365impl From<StoreError> for worker::Error {
366 fn from(error: StoreError) -> Self {
367 match error.busy {
368 Some(busy) => busy.error(&error.thrown.to_string()),
369 None => error.thrown.into(),
370 }
371 }
372}
373
374/// The meter for a binding method: `binding.create_token`.
375fn meter_of(method: &str) -> String {
376 let mut out = String::from("binding.");
377 for c in method.chars() {
378 if c.is_ascii_uppercase() {
379 out.push('_');
380 out.push(c.to_ascii_lowercase());
381 } else {
382 out.push(c);
383 }
384 }
385 out
386}
387
388/// Seconds a caller is told to wait when the store is busy.
389const BUSY_RETRY_AFTER: u64 = 5;
390
391/// Calls `target[method](...args)` on namespace `namespace`'s binding, for
392/// the repository at `key`. `retry`: whether a failure that may pass is
393/// tried again (reads and credentials only).
394async fn invoke(
395 namespace: &str,
396 key: &str,
Merge branch 'worktree-agent-a2013627e5ea4ab13'397 target: &Target,
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily398 method: &str,
399 args: &[JsValue],
400 retry: bool,
401) -> std::result::Result<JsValue, StoreError> {
402 let meter = meter_of(method);
Merge branch 'worktree-agent-a2013627e5ea4ab13'403 let health = target.health_name(namespace);
404 let namespace = health.as_str();
405 // A read-only fallback refuses writes before asking (fallback.rs).
406 if let Target::Fallback { settings, repo, .. } = target
407 && !settings.writes
408 {
409 let values: Vec<serde_json::Value> = args.iter().map(|arg| js::from_js(arg).unwrap_or(serde_json::Value::Null)).collect();
410 if fallback::writes(repo.as_deref(), method, &values) {
411 return Err(StoreError {
412 thrown: Thrown { code: Some("READ_ONLY".to_owned()), message: format!("{method} refused: the git store is read-only") },
413 busy: Some(Busy::read_only()),
414 });
415 }
416 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily417 let mut attempt = 0;
418 loop {
419 let now = g1t_kit::now_ms();
420 let admit = resilience::with_breaker(namespace, |breaker| breaker.admit(now));
421 if let Admit::Wait(ms) = admit {
422 meters::record_health(namespace, Outcome::Rejected, 0);
423 return Err(StoreError {
424 thrown: Thrown { code: None, message: format!("{method} not asked: the git store has been failing") },
Merge branch 'worktree-agent-a2013627e5ea4ab13'425 busy: Some(Busy { rate_limited: false, retry_after: resilience::seconds(ms).max(BUSY_RETRY_AFTER), read_only: false }),
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily426 });
427 }
Merge branch 'worktree-agent-a2013627e5ea4ab13'428 // Calls on Artifacts are metered; the fallback store costs nothing
429 // per call.
430 if !target.is_fallback() {
431 meters::record(&meter, key, 0, 0);
432 }
433 let called = dispatch(target, method, args).await;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily434 let ms = g1t_kit::now_ms().saturating_sub(now);
435 match called {
436 Ok(value) => {
437 resilience::with_breaker(namespace, |breaker| breaker.succeeded());
438 meters::record_health(namespace, Outcome::Ok, ms);
439 return Ok(value);
440 }
441 Err(thrown) => {
442 let failure = resilience::classify(thrown.code.as_deref(), &thrown.message);
443 if failure == Failure::Permanent {
444 // An answer (NOT_FOUND, ALREADY_EXISTS): the store is up.
445 resilience::with_breaker(namespace, |breaker| breaker.succeeded());
446 meters::record_health(namespace, Outcome::Ok, ms);
447 return Err(StoreError { thrown, busy: None });
448 }
449 if retry && resilience::retry(failure, attempt) {
450 let wait = resilience::backoff_ms(failure, attempt, worker::js_sys::Math::random());
451 worker::Delay::from(std::time::Duration::from_millis(wait)).await;
452 attempt += 1;
453 continue;
454 }
455 resilience::with_breaker(namespace, |breaker| breaker.failed(failure, g1t_kit::now_ms()));
456 let outcome = if failure == Failure::RateLimited { Outcome::RateLimited } else { Outcome::Failed };
457 meters::record_health(namespace, outcome, ms);
458 worker::console_error!("git store {namespace}: {method} for {key} failed: {thrown}");
459 return Err(StoreError {
460 thrown,
Merge branch 'worktree-agent-a2013627e5ea4ab13'461 busy: Some(Busy { rate_limited: failure == Failure::RateLimited, retry_after: BUSY_RETRY_AFTER, read_only: false }),
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily462 });
463 }
464 }
465 }
466}
467
Merge branch 'worktree-agent-a2013627e5ea4ab13'468/// Makes one call: on the binding, or as a request to the fallback store.
469async fn dispatch(target: &Target, method: &str, args: &[JsValue]) -> std::result::Result<JsValue, Thrown> {
470 let (settings, namespace, repo) = match target {
471 Target::Js(value) => return js::call(value, method, args).await,
472 Target::Fallback { settings, namespace, repo } => (settings, namespace, repo),
473 };
474 let values: Vec<serde_json::Value> = args.iter().map(|arg| js::from_js(arg).unwrap_or(serde_json::Value::Null)).collect();
475 let route = fallback::route(namespace, repo.as_deref(), method, &values)
476 .map_err(|refused| Thrown { code: Some(refused.code.to_owned()), message: refused.message })?;
477 let unreachable = |error: worker::Error| Thrown { code: None, message: format!("the fallback store could not be reached: {error}") };
478 let headers = worker::Headers::new();
479 headers.set("x-gitstore-secret", &settings.secret).map_err(unreachable)?;
480 let mut init = worker::RequestInit::new();
481 init.with_method(match route.method {
482 "POST" => worker::Method::Post,
483 "DELETE" => worker::Method::Delete,
484 _ => worker::Method::Get,
485 });
486 if let Some(body) = &route.body {
487 headers.set("content-type", "application/json").map_err(unreachable)?;
488 init.with_body(Some(JsValue::from_str(&body.to_string())));
489 }
490 init.with_headers(headers);
491 let request = worker::Request::new_with_init(&format!("{}{}", settings.url, route.path), &init).map_err(unreachable)?;
492 let mut response = worker::Fetch::Request(request).send().await.map_err(unreachable)?;
493 let status = response.status_code();
494 let body = response.bytes().await.map_err(unreachable)?;
495 match fallback::answer(&route, status, body) {
496 fallback::Answer::Json(value) => js::to_js(&value).map_err(|error| Thrown { code: None, message: error.to_string() }),
497 fallback::Answer::Bytes(bytes) => Ok(Uint8Array::from(bytes.as_slice()).into()),
498 fallback::Answer::Null => Ok(JsValue::NULL),
499 fallback::Answer::Error { code, message } => Err(Thrown { code, message }),
500 }
501}
502
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms503/// A credential as isolates share it, sealed (see shared.rs): with when it
504/// was made, so that one shared is reused no longer than one kept here.
505#[derive(Serialize, Deserialize)]
506struct SharedCredential {
507 remote: String,
508 token: String,
509 made: u64,
510}
511
512/// The shared cache's key for a credential: the store's key for the
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily513/// repository, the scope, and who it is for. (`cred2`: credentials kept
514/// before their lives differed by use are not read.)
515fn shared_key(key: &str, scope: Scope, using: Use) -> String {
516 format!("cred2:{key}:{}:{}", scope.as_str(), using.as_str())
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms517}
518
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily519/// A shared credential, if it was made less than its reuse window before
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms520/// `now`; with when it was made.
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily521fn shared_credential(bytes: &[u8], now: u64, using: Use) -> Option<(GitAccess, u64)> {
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms522 let kept: SharedCredential = serde_json::from_slice(bytes).ok()?;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily523 (now.saturating_sub(kept.made) < using.reuse_ms()).then_some((
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms524 GitAccess {
525 remote: kept.remote,
526 token: kept.token,
527 },
528 kept.made,
529 ))
530}
531
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily532/// Credentials made a while ago, by repository, scope and use. Making one
533/// is a round trip to the store; reusing it saves that, and the store's
534/// lookup of the repository with it.
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms535#[derive(Default)]
536pub struct Credentials {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily537 kept: HashMap<(String, Scope, Use), (GitAccess, u64)>,
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms538}
539
540impl Credentials {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily541 /// One made for `key`, `scope` and `using` within its reuse window of `now`.
542 pub fn get(&self, key: &str, scope: Scope, using: Use, now: u64) -> Option<GitAccess> {
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms543 self.kept
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily544 .get(&(key.to_owned(), scope, using))
545 .filter(|(_, made)| now.saturating_sub(*made) < using.reuse_ms())
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms546 .map(|(access, _)| access.clone())
547 }
548
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily549 pub fn keep(&mut self, key: &str, scope: Scope, using: Use, access: GitAccess, made: u64, now: u64) {
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms550 // Expired ones go first, so the map stays as small as the isolate's
551 // recent repositories.
552 self.kept
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily553 .retain(|(_, _, kept_use), (_, at)| now.saturating_sub(*at) < kept_use.reuse_ms());
554 self.kept.insert((key.to_owned(), scope, using), (access, made));
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms555 }
556
557 pub fn forget(&mut self, key: &str) {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily558 self.kept.retain(|(kept, _, _), _| kept != key);
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms559 }
560}
561
562thread_local! {
563 static CREDENTIALS: RefCell<Credentials> = RefCell::new(Credentials::default());
564}
565
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily566/// A credential kept in this isolate, else one another isolate shared.
567async fn kept(shared: Option<&crate::shared::Shared>, key: &str, scope: Scope, using: Use) -> Option<(GitAccess, Kept)> {
568 let now = g1t_kit::now_ms();
569 if let Some(access) = CREDENTIALS.with(|kept| kept.borrow().get(key, scope, using, now)) {
570 return Some((access, Kept::Isolate));
571 }
572 let bytes = shared?.get(&shared_key(key, scope, using)).await?;
573 let (access, made) = shared_credential(&bytes, now, using)?;
574 CREDENTIALS.with(|kept| kept.borrow_mut().keep(key, scope, using, access.clone(), made, now));
575 Some((access, Kept::Shared))
576}
577
578/// Keeps a credential just made here, and shares it.
579async fn keep(shared: Option<&crate::shared::Shared>, key: &str, scope: Scope, using: Use, access: &GitAccess) {
580 let now = g1t_kit::now_ms();
581 CREDENTIALS.with(|kept| kept.borrow_mut().keep(key, scope, using, access.clone(), now, now));
582 if let Some(shared) = shared {
583 let value = SharedCredential {
584 remote: access.remote.clone(),
585 token: access.token.clone(),
586 made: now,
587 };
588 if let Ok(bytes) = serde_json::to_vec(&value) {
589 shared.put(&shared_key(key, scope, using), &bytes, using.reuse_ms() / 1000).await;
590 }
591 }
592}
593
Agents as a team: lifecycle, merge queue, billing and a new shell594impl GitStore for ArtifactsStore {
595 type Repo = ArtifactsRepo;
596
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms597 async fn kept_access(&self, key: &str, scope: Scope) -> Option<(GitAccess, Kept)> {
Merge branch 'worktree-agent-a2013627e5ea4ab13'598 kept(self.shared.as_deref(), &self.cred_key(key), scope, Use::Internal).await
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms599 }
600
601 async fn mint_access(&self, key: &str, scope: Scope) -> Result<GitAccess> {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily602 let repo = self.open(key).await?;
603 let access = repo.mint(scope, Use::Internal).await?;
Merge branch 'worktree-agent-a2013627e5ea4ab13'604 keep(self.shared.as_deref(), &self.cred_key(key), scope, Use::Internal, &access).await;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily605 Ok(access)
606 }
607
608 async fn handout(&self, key: &str, scope: Scope) -> Result<GitAccess> {
Merge branch 'worktree-agent-a2013627e5ea4ab13'609 let cred_key = self.cred_key(key);
610 if let Some((access, _)) = kept(self.shared.as_deref(), &cred_key, scope, Use::Handout).await {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily611 return Ok(access);
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms612 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily613 let access = self.open(key).await?.mint(scope, Use::Handout).await?;
Merge branch 'worktree-agent-a2013627e5ea4ab13'614 keep(self.shared.as_deref(), &cred_key, scope, Use::Handout, &access).await;
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms615 Ok(access)
616 }
617
618 async fn forget_access(&self, key: &str) {
Merge branch 'worktree-agent-a2013627e5ea4ab13'619 // Both stores' credentials: whichever serves the key now.
620 let names = [cred_key(key, false), cred_key(key, true)];
621 CREDENTIALS.with(|kept| {
622 let mut kept = kept.borrow_mut();
623 for name in &names {
624 kept.forget(name);
625 }
626 });
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms627 if let Some(shared) = &self.shared {
Merge branch 'worktree-agent-a2013627e5ea4ab13'628 let keys: Vec<String> = names
629 .iter()
630 .flat_map(|name| {
631 [Scope::Read, Scope::Write]
632 .into_iter()
633 .flat_map(move |scope| [Use::Internal, Use::Handout].map(|using| shared_key(name, scope, using)))
634 })
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily635 .collect();
636 futures_util::future::join_all(keys.iter().map(|shared_key| shared.delete(shared_key))).await;
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms637 }
638 }
639
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily640 async fn create(&self, key: &str, description: Option<&str>, default_branch: &str) -> Result<()> {
641 let (namespace, name) = locate(key);
Agents as a team: lifecycle, merge queue, billing and a new shell642 let options = js::to_js(&serde_json::json!({
643 "description": description,
644 "setDefaultBranch": default_branch,
645 }))?;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily646 match invoke(&namespace, key, self.binding(&namespace)?, "create", &[name.as_str().into(), options], true).await {
Agents as a team: lifecycle, merge queue, billing and a new shell647 // Left behind by an earlier failed attempt; adopt it.
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily648 Err(failed) if !failed.is("ALREADY_EXISTS") => Err(failed.into()),
Agents as a team: lifecycle, merge queue, billing and a new shell649 _ => Ok(()),
650 }
651 }
652
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look653 async fn delete(&self, key: &str) -> Result<()> {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily654 let (namespace, name) = locate(key);
Merge branch 'worktree-agent-a2013627e5ea4ab13'655 // Nothing is forgotten while the store refuses to delete.
656 if self.writable(&namespace) {
657 self.forget_access(key).await;
658 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily659 match invoke(&namespace, key, self.binding(&namespace)?, "delete", &[name.as_str().into()], true).await {
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look660 // Gone already: an earlier purge got this far.
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily661 Err(failed) if !failed.is("NOT_FOUND") => Err(failed.into()),
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look662 _ => Ok(()),
663 }
664 }
665
Agents as a team: lifecycle, merge queue, billing and a new shell666 async fn open(&self, key: &str) -> Result<ArtifactsRepo> {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily667 let (namespace, name) = locate(key);
Merge branch 'worktree-agent-a2013627e5ea4ab13'668 let binding = self.binding(&namespace)?.clone();
Agents as a team: lifecycle, merge queue, billing and a new shell669 Ok(ArtifactsRepo {
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for670 handle: RefCell::new(None),
Merge branch 'worktree-agent-a2013627e5ea4ab13'671 cred_key: cred_key(key, binding.is_fallback()),
672 binding,
Agents as a team: lifecycle, merge queue, billing and a new shell673 key: key.to_owned(),
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily674 name,
675 namespace,
676 shared: self.shared.clone(),
677 refs_version: None,
Agents as a team: lifecycle, merge queue, billing and a new shell678 })
679 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily680
681 fn namespaces(&self) -> Vec<String> {
682 self.namespaces.iter().map(|namespace| namespace.name.clone()).collect()
683 }
684
685 fn default_namespace(&self) -> String {
686 DEFAULT_NS.with(|ns| ns.borrow().clone())
687 }
Merge branch 'worktree-agent-a2013627e5ea4ab13'688
689 fn on_fallback(&self, key: &str) -> bool {
690 let (namespace, _) = locate(key);
691 self.fallback_of(&namespace).is_some()
692 }
693
694 fn writable(&self, namespace: &str) -> bool {
695 self.fallback_of(namespace).is_none_or(|settings| settings.writes)
696 }
Agents as a team: lifecycle, merge queue, billing and a new shell697}
698
Merge branch 'worktree-agent-a2013627e5ea4ab13'699/// A handle to one repository in the store. On Artifacts it is an RPC
700/// stub, so it is released when dropped.
Agents as a team: lifecycle, merge queue, billing and a new shell701pub struct ArtifactsRepo {
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for702 /// The store's handle, asked for (`get`) on the first call that needs
703 /// the store: an answer from a cache, or a fetch over git with a kept
704 /// credential, never costs a `get`.
Merge branch 'worktree-agent-a2013627e5ea4ab13'705 handle: RefCell<Option<Target>>,
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for706 /// The namespace's binding, for that `get`.
Merge branch 'worktree-agent-a2013627e5ea4ab13'707 binding: Target,
Agents as a team: lifecycle, merge queue, billing and a new shell708 /// The repository's store key, which scopes its cached objects.
709 key: String,
Merge branch 'worktree-agent-a2013627e5ea4ab13'710 /// What its credentials are kept under (`ArtifactsStore::cred_key`).
711 cred_key: String,
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily712 /// Its name in its namespace.
713 name: String,
714 namespace: String,
715 shared: Option<Rc<crate::shared::Shared>>,
716 /// See [`GitRepo::at_refs_version`].
717 refs_version: Option<u64>,
Agents as a team: lifecycle, merge queue, billing and a new shell718}
719
720/// Where cached git objects live. Trees and blobs are named by their
721/// content, so a cached one is never stale; each is kept under its own
722/// repository's key, so a repository only ever finds its own objects.
723const OBJECT_CACHE: &str = "https://objects.g1t.internal/";
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily724/// Blobs and files larger than this are not cached.
Agents as a team: lifecycle, merge queue, billing and a new shell725const MAX_CACHED_BLOB: usize = 1024 * 1024;
726const OBJECT_MAX_AGE: &str = "public, max-age=31536000, immutable";
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily727/// Answers kept under a refs version: until the version moves, and no
728/// longer than this, which bounds how stale one can be should a change
729/// ever fail to move it.
730const VERSIONED_MAX_AGE: &str = "public, max-age=300";
Repos: a path found not to be a file is remembered for 10 minutes, so repeated misses read nothing731/// How long a path found not to be a file at a ref is remembered. Short:
732/// a miss by commit hash is true for good, but one for a commit not yet in
733/// the store would not be.
734const ABSENT_MAX_AGE: &str = "public, max-age=600";
Agents as a team: lifecycle, merge queue, billing and a new shell735
Merge project overview: one branch_drift call, spliced histories, cached tags, 6 repos calls instead of 25736/// Histories by hash at least this long are put together from a short
737/// read and one kept before, where they can be (`spliced_log`).
738const SPLICE_FROM: u32 = 100;
739/// How many commits that short read takes.
740const SPLICE_PROBE: u32 = 16;
741
742/// `short[..at]` followed by `kept` (the history from `short[at]`), cut
743/// to `limit` commits.
744pub fn splice(short: &[Commit], at: usize, kept: Vec<Commit>, limit: u32) -> Vec<Commit> {
745 let mut out: Vec<Commit> = short[..at.min(short.len())].to_vec();
746 out.extend(kept);
747 out.truncate(limit as usize);
748 out
749}
750
Merge branch drift: count across merges the way git does; v2 cache key751/// The history from `short[0]`, from the newest of `kept` (each the history
752/// kept from the commit of `short` at the same index, if any) that really
753/// is the history from that commit: it starts there and goes on to the
754/// next commit `short` lists, so the join repeats and skips nothing.
755pub fn splice_first(short: &[Commit], kept: Vec<Option<Vec<Commit>>>, limit: u32) -> Option<Vec<Commit>> {
756 kept.into_iter().enumerate().skip(1).find_map(|(at, kept)| {
757 let kept = kept?;
758 let starts = kept.first().is_some_and(|first| short.get(at).is_some_and(|commit| commit.hash == first.hash));
759 let goes_on = match (short.get(at + 1), kept.get(1)) {
760 (Some(next), Some(kept_next)) => next.hash == kept_next.hash,
761 // `short` ends at `at`: it reached the first commit, or its limit.
762 (None, _) => true,
763 // The kept history ends where `short` goes on.
764 (Some(_), None) => false,
765 };
766 (starts && goes_on).then(|| splice(short, at, kept, limit))
767 })
768}
769
Fast pages, required checks on the branch, self-hosted runners, honest incidents770/// Whether a ref is a full commit hash (SHA-1 or SHA-256), whose history
771/// can be kept for good.
772pub fn is_commit_hash(git_ref: &str) -> bool {
773 (git_ref.len() == 40 || git_ref.len() == 64) && git_ref.bytes().all(|b| b.is_ascii_hexdigit() && !b.is_ascii_uppercase())
774}
775
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily776/// Where a read is kept: under the object it names for good, under the
777/// refs version for a name that can move, or nowhere.
778#[derive(Debug, PartialEq, Eq)]
779pub enum CacheKey {
780 Forever(String),
781 Versioned(String),
782}
783
784/// The cache key for `log(git_ref, limit)`.
785pub fn log_key(git_ref: &str, limit: u32, version: Option<u64>) -> Option<CacheKey> {
786 if is_commit_hash(git_ref) {
787 return Some(CacheKey::Forever(format!("log/{git_ref}-{limit}")));
788 }
789 version.map(|version| CacheKey::Versioned(format!("vlog/{version}/{}-{limit}", g1t_secrets::sha256_hex(git_ref))))
790}
791
792/// The cache key for `read_file(git_ref, path)`.
793pub fn file_key(git_ref: &str, path: &str, version: Option<u64>) -> Option<CacheKey> {
794 let path = g1t_secrets::sha256_hex(path);
795 if is_commit_hash(git_ref) {
796 return Some(CacheKey::Forever(format!("file/{git_ref}/{path}")));
797 }
798 version.map(|version| CacheKey::Versioned(format!("vfile/{version}/{}/{path}", g1t_secrets::sha256_hex(git_ref))))
799}
800
Repos: a path found not to be a file is remembered for 10 minutes, so repeated misses read nothing801/// Where `read_file` notes that its key is not a file (see `known_absent`).
802fn absent_path(key: &CacheKey) -> String {
803 match key {
804 CacheKey::Forever(path) | CacheKey::Versioned(path) => format!("absent/{path}"),
805 }
806}
807
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily808/// The cache key for the branch list.
809pub fn branches_key(version: Option<u64>) -> Option<CacheKey> {
810 version.map(|version| CacheKey::Versioned(format!("branches/{version}")))
811}
812
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for813/// Objects named by their content, kept in the isolate ahead of the Cache
814/// API, oldest out first past `MEMORY_CACHE_BYTES`. They never go stale,
815/// and each is under its repository's key.
816const MEMORY_CACHE_BYTES: usize = 16 * 1024 * 1024;
817
818#[derive(Default)]
819struct MemoryCache {
820 entries: HashMap<String, Rc<Vec<u8>>>,
821 order: std::collections::VecDeque<String>,
822 bytes: usize,
823}
824
825impl MemoryCache {
826 fn get(&self, url: &str) -> Option<Vec<u8>> {
827 self.entries.get(url).map(|bytes| bytes.as_ref().clone())
828 }
829
830 fn put(&mut self, url: String, bytes: &[u8]) {
831 if bytes.len() > MEMORY_CACHE_BYTES / 16 || self.entries.contains_key(&url) {
832 return;
833 }
834 self.bytes += bytes.len();
835 self.entries.insert(url.clone(), Rc::new(bytes.to_vec()));
836 self.order.push_back(url);
837 while self.bytes > MEMORY_CACHE_BYTES {
838 let Some(oldest) = self.order.pop_front() else { break };
839 if let Some(gone) = self.entries.remove(&oldest) {
840 self.bytes -= gone.len();
841 }
842 }
843 }
844}
845
846thread_local! {
847 static MEMORY: RefCell<MemoryCache> = RefCell::new(MemoryCache::default());
848}
849
Agents as a team: lifecycle, merge queue, billing and a new shell850impl ArtifactsRepo {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily851 fn cache_url(&self, path: &str) -> String {
852 format!("{OBJECT_CACHE}{}/{path}", self.key)
853 }
854
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for855 /// A kept answer: from the isolate for one kept for good, else the
856 /// Cache API. Each look is metered (`cache.memory_hit`, `cache.edge_hit`,
857 /// `cache.miss`), so the usage check shows which cache answers.
858 async fn cached_at(&self, path: &str, forever: bool) -> Option<Vec<u8>> {
859 let url = self.cache_url(path);
860 if forever && let Some(bytes) = MEMORY.with(|memory| memory.borrow().get(&url)) {
Cache hits are counted, not only their bytes861 meters::record("cache.memory_hit", &self.key, 0, bytes.len() as u64);
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for862 return Some(bytes);
863 }
864 let found = match worker::Cache::default().get(url.clone(), false).await {
865 Ok(Some(mut response)) => response.bytes().await.ok(),
866 _ => None,
867 };
868 match &found {
869 Some(bytes) => {
Cache hits are counted, not only their bytes870 meters::record("cache.edge_hit", &self.key, 0, bytes.len() as u64);
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for871 if forever {
872 MEMORY.with(|memory| memory.borrow_mut().put(url, bytes));
873 }
874 }
875 None => meters::record("cache.miss", &self.key, 0, 0),
876 }
877 found
Agents as a team: lifecycle, merge queue, billing and a new shell878 }
879
Merge project overview: one branch_drift call, spliced histories, cached tags, 6 repos calls instead of 25880 /// A history from the store itself.
881 async fn read_log(&self, git_ref: &str, limit: u32) -> Result<Vec<Commit>> {
882 let options = js::to_js(&serde_json::json!({ "ref": git_ref, "limit": limit }))?;
883 let raw: Vec<RawCommit> = js::from_js(&self.call("log", &[options], true).await?)?;
884 Ok(raw
885 .into_iter()
886 .map(|commit| Commit {
887 hash: commit.hash,
888 tree_hash: commit.tree_hash,
889 message: commit.message,
890 author: commit.author,
891 parents: commit.parents,
892 authored_at: rfc3339(commit.authored_at * 1000),
893 })
894 .collect())
895 }
896
897 /// A long history by commit hash, from a short read and one kept
898 /// before: when a branch moves a few commits, the history from its new
899 /// head is those commits, then the history kept from its old one (the
900 /// first-parent chain from a commit never changes). Only when none of
901 /// the short read's commits has the same history kept is the whole of
902 /// it read. A default branch that moved by a merge then costs a read
903 /// of [`SPLICE_PROBE`] commits instead of a thousand.
904 async fn spliced_log(&self, hash: &str, limit: u32) -> Result<Vec<Commit>> {
905 let short = self.read_log(hash, SPLICE_PROBE).await?;
906 if (short.len() as u32) < SPLICE_PROBE {
907 // The whole history fits in the short read.
908 return Ok(short);
909 }
Merge branch drift: count across merges the way git does; v2 cache key910 let kept = futures_util::future::join_all(short.iter().enumerate().map(|(at, commit)| async move {
911 if at == 0 {
912 return None;
913 }
Merge project overview: one branch_drift call, spliced histories, cached tags, 6 repos calls instead of 25914 let Some(CacheKey::Forever(path)) = log_key(&commit.hash, limit, None) else {
915 return None;
916 };
917 let bytes = self.peek(&path).await?;
Merge branch drift: count across merges the way git does; v2 cache key918 serde_json::from_slice::<Vec<Commit>>(&bytes).ok()
Merge project overview: one branch_drift call, spliced histories, cached tags, 6 repos calls instead of 25919 }))
920 .await;
Merge branch drift: count across merges the way git does; v2 cache key921 match splice_first(&short, kept, limit) {
922 Some(history) => Ok(history),
Merge project overview: one branch_drift call, spliced histories, cached tags, 6 repos calls instead of 25923 None => self.read_log(hash, limit).await,
924 }
925 }
926
927 /// A kept answer, if there is one, without counting a miss: the
928 /// splice looks for many and expects most to be absent.
929 async fn peek(&self, path: &str) -> Option<Vec<u8>> {
930 let url = self.cache_url(path);
931 if let Some(bytes) = MEMORY.with(|memory| memory.borrow().get(&url)) {
932 meters::record("cache.memory_hit", &self.key, 0, bytes.len() as u64);
933 return Some(bytes);
934 }
935 let mut response = worker::Cache::default().get(url.clone(), false).await.ok()??;
936 let bytes = response.bytes().await.ok()?;
937 meters::record("cache.edge_hit", &self.key, 0, bytes.len() as u64);
938 MEMORY.with(|memory| memory.borrow_mut().put(url, &bytes));
939 Some(bytes)
940 }
941
Agents as a team: lifecycle, merge queue, billing and a new shell942 async fn cached(&self, kind: &str, hash: &str) -> Option<Vec<u8>> {
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for943 self.cached_at(&format!("{kind}/{hash}"), true).await
Agents as a team: lifecycle, merge queue, billing and a new shell944 }
945
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily946 async fn keep_at(&self, path: &str, bytes: Vec<u8>, max_age: &str) {
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for947 if max_age == OBJECT_MAX_AGE {
948 MEMORY.with(|memory| memory.borrow_mut().put(self.cache_url(path), &bytes));
949 }
Agents as a team: lifecycle, merge queue, billing and a new shell950 let Ok(mut response) = worker::Response::from_bytes(bytes) else {
951 return;
952 };
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily953 let _ = response.headers_mut().set("cache-control", max_age);
954 let _ = worker::Cache::default().put(self.cache_url(path), response).await;
Agents as a team: lifecycle, merge queue, billing and a new shell955 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily956
Repos: a path found not to be a file is remembered for 10 minutes, so repeated misses read nothing957 /// Whether `read_file`'s key was found not to be a file a little while
958 /// ago. Metered only when it was (`cache.absent_hit`): the lookup runs
959 /// beside the key's own, which already counts the miss.
960 async fn known_absent(&self, key: &CacheKey) -> bool {
961 let url = self.cache_url(&absent_path(key));
962 let found = matches!(worker::Cache::default().get(url, false).await, Ok(Some(_)));
963 if found {
964 meters::record("cache.absent_hit", &self.key, 0, 0);
965 }
966 found
967 }
968
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily969 /// Keeps an object for next time. A failure only costs a later read.
970 async fn keep(&self, kind: &str, hash: &str, bytes: Vec<u8>) {
971 self.keep_at(&format!("{kind}/{hash}"), bytes, OBJECT_MAX_AGE).await;
972 }
973
974 async fn get_key(&self, key: &CacheKey) -> Option<Vec<u8>> {
975 match key {
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for976 CacheKey::Forever(path) => self.cached_at(path, true).await,
977 CacheKey::Versioned(path) => self.cached_at(path, false).await,
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily978 }
979 }
980
981 async fn put_key(&self, key: &CacheKey, bytes: Vec<u8>) {
982 match key {
983 CacheKey::Forever(path) => self.keep_at(path, bytes, OBJECT_MAX_AGE).await,
984 CacheKey::Versioned(path) => self.keep_at(path, bytes, VERSIONED_MAX_AGE).await,
985 }
986 }
987
988 async fn call(&self, method: &str, args: &[JsValue], retry: bool) -> std::result::Result<JsValue, StoreError> {
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for989 let handle = self.handle().await?;
990 invoke(&self.namespace, &self.key, &handle, method, args, retry).await
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily991 }
992
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for993 /// The store's handle, asked for the first time it is needed.
Merge branch 'worktree-agent-a2013627e5ea4ab13'994 async fn handle(&self) -> std::result::Result<Target, StoreError> {
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for995 if let Some(handle) = self.handle.borrow().as_ref() {
996 return Ok(handle.clone());
997 }
Merge branch 'worktree-agent-a2013627e5ea4ab13'998 let found = invoke(&self.namespace, &self.key, &self.binding, "get", &[self.name.as_str().into()], true).await?;
999 let handle = match &self.binding {
1000 Target::Js(_) => Target::Js(found),
1001 // The fallback store said the repository is there.
1002 Target::Fallback { settings, namespace, .. } => Target::Fallback {
1003 settings: settings.clone(),
1004 namespace: namespace.clone(),
1005 repo: Some(self.name.clone()),
1006 },
1007 };
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for1008 *self.handle.borrow_mut() = Some(handle.clone());
1009 Ok(handle)
1010 }
1011
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1012 /// A new credential from the store. Its remote is worked out from the
1013 /// key once this isolate knows where the namespace's remotes start;
Merge branch 'worktree-agent-a2013627e5ea4ab13'1014 /// until then the store is asked (`info()`) alongside the token. The
1015 /// fallback store's remotes are known from its address.
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1016 pub async fn mint(&self, scope: Scope, using: Use) -> Result<GitAccess> {
1017 let args = [scope.as_str().into(), using.ttl_seconds().into()];
Merge branch 'worktree-agent-a2013627e5ea4ab13'1018 if let Target::Fallback { settings, .. } = &self.binding {
1019 let token: RawToken = js::from_js(&self.call("createToken", &args, true).await?)?;
1020 return Ok(GitAccess { remote: settings.remote(&self.namespace, &self.name), token: token.plaintext });
1021 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1022 let prefix = REMOTE_PREFIX.with(|prefixes| prefixes.borrow().get(&self.namespace).cloned());
1023 if let Some(prefix) = prefix {
1024 let token: RawToken = js::from_js(&self.call("createToken", &args, true).await?)?;
1025 return Ok(GitAccess { remote: remote_from(&prefix, &self.name), token: token.plaintext });
1026 }
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for1027 self.handle().await?;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1028 let (info, token) = futures_util::future::join(self.call("info", &[], true), self.call("createToken", &args, true)).await;
1029 let info: RawInfo = js::from_js(&info?)?;
1030 let token: RawToken = js::from_js(&token?)?;
1031 match learn_prefix(&info.remote, &self.name) {
1032 Some(prefix) => REMOTE_PREFIX.with(|prefixes| {
1033 prefixes.borrow_mut().insert(self.namespace.clone(), prefix);
1034 }),
1035 None => worker::console_error!("the git store's remote {} does not end in {}.git", info.remote, self.name),
1036 }
1037 Ok(GitAccess { remote: info.remote, token: token.plaintext })
1038 }
Agents as a team: lifecycle, merge queue, billing and a new shell1039}
1040
1041impl Drop for ArtifactsRepo {
1042 fn drop(&mut self) {
1043 let symbol = js::get(&worker::js_sys::global(), "Symbol");
1044 let dispose = js::get(&symbol, "dispose");
Merge branch 'worktree-agent-a2013627e5ea4ab13'1045 let Some(Target::Js(handle)) = self.handle.borrow_mut().take() else { return };
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for1046 if let Ok(function) = Reflect::get(&handle, &dispose).and_then(|value| value.dyn_into::<worker::js_sys::Function>()) {
1047 let _ = function.call0(&handle);
Agents as a team: lifecycle, merge queue, billing and a new shell1048 }
1049 }
1050}
1051
1052#[derive(Deserialize)]
1053#[serde(rename_all = "camelCase")]
1054struct RawCommit {
1055 hash: String,
1056 tree_hash: String,
1057 message: String,
1058 author: Signature,
1059 parents: Vec<String>,
1060 /// Seconds since the epoch.
1061 authored_at: u64,
1062}
1063
1064#[derive(Deserialize)]
1065struct RawEntry {
1066 name: String,
1067 hash: String,
1068 #[serde(rename = "type")]
1069 kind: EntryKind,
1070}
1071
1072#[derive(Deserialize)]
1073struct RawInfo {
1074 remote: String,
1075}
1076
1077#[derive(Deserialize)]
1078struct RawToken {
1079 plaintext: String,
1080}
1081
Merge branch 'worktree-agent-a2013627e5ea4ab13'1082/// The bytes of a `Blob` (or of the bytes the fallback store sent), or
1083/// `None` for null.
Agents as a team: lifecycle, merge queue, billing and a new shell1084async fn blob_bytes(blob: JsValue) -> Result<Option<Vec<u8>>> {
1085 if blob.is_null() || blob.is_undefined() {
1086 return Ok(None);
1087 }
Merge branch 'worktree-agent-a2013627e5ea4ab13'1088 if let Some(bytes) = blob.dyn_ref::<Uint8Array>() {
1089 return Ok(Some(bytes.to_vec()));
1090 }
Agents as a team: lifecycle, merge queue, billing and a new shell1091 let buffer = js::call(&blob, "arrayBuffer", &[]).await?;
1092 Ok(Some(Uint8Array::new(&buffer).to_vec()))
1093}
1094
1095impl GitRepo for ArtifactsRepo {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1096 /// One made a while ago, here or in another isolate, else a new one.
Agents as a team: lifecycle, merge queue, billing and a new shell1097 async fn access(&self, scope: Scope) -> Result<GitAccess> {
Merge branch 'worktree-agent-a2013627e5ea4ab13'1098 if let Some((access, _)) = kept(self.shared.as_deref(), &self.cred_key, scope, Use::Internal).await {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1099 return Ok(access);
1100 }
1101 let access = self.mint(scope, Use::Internal).await?;
Merge branch 'worktree-agent-a2013627e5ea4ab13'1102 keep(self.shared.as_deref(), &self.cred_key, scope, Use::Internal, &access).await;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1103 Ok(access)
Agents as a team: lifecycle, merge queue, billing and a new shell1104 }
1105
1106 async fn branches(&self) -> Result<Vec<Branch>> {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1107 let key = branches_key(self.refs_version);
1108 if let Some(key) = &key
1109 && let Some(bytes) = self.get_key(key).await
1110 && let Ok(branches) = serde_json::from_slice::<Vec<Branch>>(&bytes)
1111 {
1112 return Ok(branches);
1113 }
1114 let branches = crate::refs::branches(&self.access(Scope::Read).await?).await?;
1115 if let (Some(key), Ok(bytes)) = (&key, serde_json::to_vec(&branches)) {
1116 self.put_key(key, bytes).await;
1117 }
1118 Ok(branches)
Agents as a team: lifecycle, merge queue, billing and a new shell1119 }
1120
1121 async fn log(&self, git_ref: &str, limit: u32) -> Result<Vec<Commit>> {
Fast pages, required checks on the branch, self-hosted runners, honest incidents1122 // History from a commit never changes, so a log asked for by hash is
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1123 // kept like an object: walking it is a read per commit. One asked
1124 // for by a branch's name is kept until the branch can have moved.
1125 let key = log_key(git_ref, limit, self.refs_version);
1126 if let Some(key) = &key
1127 && let Some(bytes) = self.get_key(key).await
Fast pages, required checks on the branch, self-hosted runners, honest incidents1128 && let Ok(commits) = serde_json::from_slice::<Vec<Commit>>(&bytes)
1129 {
1130 return Ok(commits);
1131 }
Merge project overview: one branch_drift call, spliced histories, cached tags, 6 repos calls instead of 251132 let commits = match &key {
1133 Some(CacheKey::Forever(_)) if limit >= SPLICE_FROM => self.spliced_log(git_ref, limit).await?,
1134 _ => self.read_log(git_ref, limit).await?,
1135 };
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1136 // An unknown ref logs nothing; that is not kept, in case it arrives.
1137 if !commits.is_empty()
Fast pages, required checks on the branch, self-hosted runners, honest incidents1138 && let Ok(bytes) = serde_json::to_vec(&commits)
1139 {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1140 if let Some(key) = &key {
1141 self.put_key(key, bytes.clone()).await;
1142 }
1143 // The same history, by the commit the name led to.
1144 if !is_commit_hash(git_ref)
1145 && let Some(CacheKey::Forever(path)) = log_key(&commits[0].hash, limit, None)
1146 {
1147 self.keep_at(&path, bytes, OBJECT_MAX_AGE).await;
1148 }
Fast pages, required checks on the branch, self-hosted runners, honest incidents1149 }
1150 Ok(commits)
Agents as a team: lifecycle, merge queue, billing and a new shell1151 }
1152
1153 async fn parents(&self, commit_hash: &str) -> Result<Option<Vec<String>>> {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1154 // A commit never changes.
1155 if let Some(bytes) = self.cached("commit", commit_hash).await
1156 && let Ok(parents) = serde_json::from_slice::<Vec<String>>(&bytes)
1157 {
1158 return Ok(Some(parents));
1159 }
1160 let commit: Option<RawCommit> = js::from_js(&self.call("readCommit", &[commit_hash.into()], true).await?)?;
1161 let parents = commit.map(|commit| commit.parents);
1162 if let Some(parents) = &parents
1163 && let Ok(bytes) = serde_json::to_vec(parents)
1164 {
1165 self.keep("commit", commit_hash, bytes).await;
1166 }
1167 Ok(parents)
Agents as a team: lifecycle, merge queue, billing and a new shell1168 }
1169
1170 async fn read_tree(&self, tree_hash: &str) -> Result<Option<Vec<TreeEntry>>> {
Polish: phones, copy boxes, the plan page, the landing page, a real glide1171 if let Some(bytes) = self.cached("tree", tree_hash).await
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1172 && let Ok(entries) = serde_json::from_slice::<Vec<TreeEntry>>(&bytes)
1173 {
1174 return Ok(Some(entries));
1175 }
1176 let entries: Option<Vec<RawEntry>> = js::from_js(&self.call("readTree", &[tree_hash.into()], true).await?)?;
Agents as a team: lifecycle, merge queue, billing and a new shell1177 let entries: Option<Vec<TreeEntry>> = entries.map(|entries| {
1178 entries
1179 .into_iter()
1180 .map(|entry| TreeEntry {
1181 name: entry.name,
1182 hash: entry.hash,
1183 kind: entry.kind,
1184 })
1185 .collect()
1186 });
Polish: phones, copy boxes, the plan page, the landing page, a real glide1187 if let Some(entries) = &entries
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1188 && let Ok(bytes) = serde_json::to_vec(entries)
1189 {
1190 self.keep("tree", tree_hash, bytes).await;
1191 }
Agents as a team: lifecycle, merge queue, billing and a new shell1192 Ok(entries)
1193 }
1194
1195 async fn read_blob(&self, blob_hash: &str) -> Result<Option<Vec<u8>>> {
1196 if let Some(bytes) = self.cached("blob", blob_hash).await {
1197 return Ok(Some(bytes));
1198 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1199 let bytes = blob_bytes(self.call("readBlob", &[blob_hash.into()], true).await?).await?;
1200 if let Some(bytes) = &bytes {
1201 meters::record_bytes("binding.read_blob", &self.key, 0, bytes.len() as u64);
1202 }
Agents as a team: lifecycle, merge queue, billing and a new shell1203 if let Some(bytes) = bytes.as_ref().filter(|bytes| bytes.len() <= MAX_CACHED_BLOB) {
1204 self.keep("blob", blob_hash, bytes.clone()).await;
1205 }
1206 Ok(bytes)
1207 }
1208
Merge branch 'main' into worktree-agent-a69aeabc4b0deeb971209 /// Kept for good by hash, as a blob is: its bytes never cross into
1210 /// this isolate's memory, only the size of the store's answer.
1211 async fn blob_size(&self, blob_hash: &str) -> Result<Option<u64>> {
1212 let path = format!("size/{blob_hash}");
1213 if let Some(bytes) = self.cached_at(&path, true).await
1214 && let Some(size) = std::str::from_utf8(&bytes).ok().and_then(|text| text.parse().ok())
1215 {
1216 return Ok(Some(size));
1217 }
1218 let blob = self.call("readBlob", &[blob_hash.into()], true).await?;
1219 let size = if blob.is_null() || blob.is_undefined() {
1220 None
1221 } else if let Some(bytes) = blob.dyn_ref::<Uint8Array>() {
1222 Some(u64::from(bytes.length()))
1223 } else {
1224 js::get(&blob, "size").as_f64().map(|size| size as u64)
1225 };
1226 if let Some(size) = size {
1227 meters::record_bytes("binding.read_blob", &self.key, 0, size);
1228 self.keep_at(&path, size.to_string().into_bytes(), OBJECT_MAX_AGE).await;
1229 }
1230 Ok(size)
1231 }
1232
Agents as a team: lifecycle, merge queue, billing and a new shell1233 async fn read_file(&self, git_ref: &str, path: &str) -> Result<Option<Vec<u8>>> {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1234 let key = file_key(git_ref, path, self.refs_version);
Repos: a path found not to be a file is remembered for 10 minutes, so repeated misses read nothing1235 // A path that is not a file is remembered too, briefly: the store
1236 // answers each such read as a rejected read (crawlers asking for
1237 // old or missing paths make most of them). Never for the fallback
1238 // store, which can be behind.
1239 let remember_absent = !self.binding.is_fallback();
1240 if let Some(key) = &key {
1241 let (found, absent) = futures_util::future::join(self.get_key(key), async {
1242 remember_absent && self.known_absent(key).await
1243 })
1244 .await;
1245 if let Some(bytes) = found {
1246 return Ok(Some(bytes));
1247 }
1248 if absent {
1249 return Ok(None);
1250 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1251 }
Agents as a team: lifecycle, merge queue, billing and a new shell1252 let args = js::to_js(&serde_json::json!({ "ref": git_ref, "path": path }))?;
Teams and CODEOWNERS, labels and milestones, dependency updates, the security suite, and a clearer top bar1253 // The store answers a path that is not a file at the ref with
1254 // NOT_FOUND (its "read rejected"), not with nothing.
1255 let bytes = match self.call("readFile", &[args], true).await {
1256 Ok(blob) => blob_bytes(blob).await?,
1257 Err(failed) if failed.is("NOT_FOUND") => None,
1258 Err(failed) => return Err(failed.into()),
1259 };
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1260 if let Some(bytes) = &bytes {
1261 meters::record_bytes("binding.read_file", &self.key, 0, bytes.len() as u64);
Repos: a path found not to be a file is remembered for 10 minutes, so repeated misses read nothing1262 } else if remember_absent && let Some(key) = &key {
1263 self.keep_at(&absent_path(key), vec![1], ABSENT_MAX_AGE).await;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1264 }
1265 if let (Some(key), Some(bytes)) = (&key, bytes.as_ref().filter(|bytes| bytes.len() <= MAX_CACHED_BLOB)) {
1266 self.put_key(key, bytes.clone()).await;
1267 }
1268 Ok(bytes)
Agents as a team: lifecycle, merge queue, billing and a new shell1269 }
1270
1271 async fn fork(&self, target_key: &str) -> Result<()> {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1272 let (namespace, name) = locate(target_key);
1273 if namespace != self.namespace {
1274 return Err(worker::Error::RustError(format!(
1275 "a fork stays in its repository's namespace: {target_key} is not in {}",
1276 self.namespace
1277 )));
1278 }
Agents as a team: lifecycle, merge queue, billing and a new shell1279 let options = js::to_js(&serde_json::json!({ "defaultBranchOnly": true }))?;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1280 match self.call("fork", &[name.as_str().into(), options], false).await {
1281 Err(failed) if !failed.is("ALREADY_EXISTS") => Err(failed.into()),
Agents as a team: lifecycle, merge queue, billing and a new shell1282 _ => Ok(()),
1283 }
1284 }
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1285
1286 fn at_refs_version(&mut self, version: Option<u64>) {
Merge branch 'worktree-agent-a2013627e5ea4ab13'1287 // The fallback store holds what the last backup held, which may be
1288 // behind what was kept under the version: nothing is kept for it.
1289 self.refs_version = if self.binding.is_fallback() { None } else { version };
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1290 }
Agents as a team: lifecycle, merge queue, billing and a new shell1291}
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1292
1293#[cfg(test)]
1294mod tests {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1295 use super::*;
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1296
Merge project overview: one branch_drift call, spliced histories, cached tags, 6 repos calls instead of 251297 fn chain(names: &[&str]) -> Vec<Commit> {
1298 names
1299 .iter()
1300 .enumerate()
1301 .map(|(at, name)| Commit {
1302 hash: (*name).to_owned(),
1303 tree_hash: String::new(),
1304 message: String::new(),
1305 author: g1t_contracts::repos::Signature { name: "a".into(), email: "a@example.com".into() },
1306 parents: names.get(at + 1).map(|parent| vec![(*parent).to_owned()]).unwrap_or_default(),
1307 authored_at: String::new(),
1308 })
1309 .collect()
1310 }
1311
1312 fn hashes(commits: &[Commit]) -> Vec<&str> {
1313 commits.iter().map(|commit| commit.hash.as_str()).collect()
1314 }
1315
1316 #[test]
1317 fn a_history_is_the_new_commits_then_the_one_kept_from_an_old_head() {
1318 // The branch moved from c3 to c5; c3's history (limit 4) was kept.
1319 let short = chain(&["c5", "c4", "c3", "c2"]);
1320 let kept = chain(&["c3", "c2", "c1", "c0"]);
1321 assert_eq!(hashes(&splice(&short, 2, kept.clone(), 4)), ["c5", "c4", "c3", "c2"]);
1322 assert_eq!(hashes(&splice(&short, 2, kept.clone(), 6)), ["c5", "c4", "c3", "c2", "c1", "c0"]);
1323 // A kept history that reached the first commit ends there.
1324 assert_eq!(hashes(&splice(&short, 2, chain(&["c3", "c2"]), 10)), ["c5", "c4", "c3", "c2"]);
1325 }
1326
1327 #[test]
Merge branch drift: count across merges the way git does; v2 cache key1328 fn a_splice_joins_at_the_newest_kept_history_without_repeats_or_gaps() {
1329 // 16 read from c20; histories of 8 kept from c17 and c12.
1330 let all: Vec<String> = (0..=20).rev().map(|i| format!("c{i}")).collect();
1331 let names: Vec<&str> = all.iter().map(String::as_str).collect();
1332 let short = chain(&names[..16]);
1333 let kept_from = |name: &str| {
1334 let at = names.iter().position(|n| *n == name).unwrap();
1335 chain(&names[at..(at + 8).min(names.len())])
1336 };
1337 let mut kept: Vec<Option<Vec<Commit>>> = vec![None; short.len()];
1338 kept[3] = Some(kept_from("c17"));
1339 kept[8] = Some(kept_from("c12"));
1340 let joined = splice_first(&short, kept.clone(), 8).unwrap();
1341 assert_eq!(hashes(&joined), names[..8]);
1342 // Newest first, every commit once, each the first parent of the one before.
1343 let joined = splice_first(&short, kept, 11).unwrap();
1344 assert_eq!(hashes(&joined), names[..11]);
1345 for pair in joined.windows(2) {
1346 assert_eq!(pair[0].parents.first(), Some(&pair[1].hash));
1347 }
1348 let unique: std::collections::HashSet<_> = joined.iter().map(|commit| &commit.hash).collect();
1349 assert_eq!(unique.len(), joined.len());
1350 }
1351
1352 #[test]
1353 fn a_kept_history_that_does_not_fit_is_not_spliced() {
1354 let short = chain(&["c5", "c4", "c3", "c2"]);
1355 // Kept under c4's key, but from somewhere else.
1356 let wrong = vec![None, Some(chain(&["x4", "x3"])), None, None];
1357 assert!(splice_first(&short, wrong, 10).is_none());
1358 // Starts at c4 but goes on to another commit.
1359 let forked = vec![None, Some(chain(&["c4", "y3"])), None, None];
1360 assert!(splice_first(&short, forked, 10).is_none());
1361 // Ends at c4 where `short` goes on: not the history from c4.
1362 let cut = vec![None, Some(chain(&["c4"])), None, None];
1363 assert!(splice_first(&short, cut, 10).is_none());
1364 // The commit read itself is never spliced onto.
1365 let own = vec![Some(chain(&["c5", "c4"])), None, None, None];
1366 assert!(splice_first(&short, own, 10).is_none());
1367 // Nothing kept.
1368 assert!(splice_first(&short, vec![None; 4], 10).is_none());
1369 // The last commit read: anything kept from it fits.
1370 let last = vec![None, None, None, Some(chain(&["c2", "c1", "c0"]))];
1371 assert_eq!(hashes(&splice_first(&short, last, 10).unwrap()), ["c5", "c4", "c3", "c2", "c1", "c0"]);
1372 }
1373
1374 #[test]
Merge project overview: one branch_drift call, spliced histories, cached tags, 6 repos calls instead of 251375 fn only_long_histories_by_hash_are_spliced() {
1376 assert!(SPLICE_PROBE < SPLICE_FROM);
1377 let hash = "a".repeat(40);
1378 assert!(matches!(log_key(&hash, SPLICE_FROM, None), Some(CacheKey::Forever(_))));
1379 assert!(matches!(log_key("main", SPLICE_FROM, Some(1)), Some(CacheKey::Versioned(_))));
1380 }
1381
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1382 fn access(token: &str) -> GitAccess {
1383 GitAccess {
1384 remote: "https://store.example/acme--rocket.git".to_owned(),
1385 token: token.to_owned(),
1386 }
1387 }
1388
1389 #[test]
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1390 fn internal_credentials_live_an_hour_and_are_reused_for_fifty_minutes() {
1391 assert_eq!(Use::Internal.ttl_seconds(), 3_600);
1392 assert_eq!(Use::Internal.reuse_ms(), 50 * 60 * 1000);
1393 // Handed out: five minutes, reused three, so at least two are left.
1394 assert_eq!(Use::Handout.ttl_seconds(), 300);
1395 assert_eq!(Use::Handout.reuse_ms(), 180_000);
1396 assert_eq!(CREDENTIAL_LIFE_MS, 300_000);
1397 for using in [Use::Internal, Use::Handout] {
1398 assert!(u64::from(using.ttl_seconds()) * 1000 - using.reuse_ms() >= 120_000);
1399 }
1400 }
1401
1402 #[test]
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1403 fn a_credential_is_reused_only_while_it_has_time_left() {
1404 let mut kept = Credentials::default();
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1405 kept.keep("acme--rocket", Scope::Read, Use::Internal, access("r1"), 1_000, 1_000);
1406 assert_eq!(kept.get("acme--rocket", Scope::Read, Use::Internal, 1_000).unwrap().token, "r1");
1407 let last = 1_000 + INTERNAL_REUSE_MS - 1;
1408 assert_eq!(kept.get("acme--rocket", Scope::Read, Use::Internal, last).unwrap().token, "r1");
1409 assert!(kept.get("acme--rocket", Scope::Read, Use::Internal, last + 1).is_none());
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1410 }
1411
1412 #[test]
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1413 fn a_credential_is_kept_for_its_own_repository_scope_and_use() {
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1414 let mut kept = Credentials::default();
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1415 kept.keep("acme--rocket", Scope::Read, Use::Internal, access("r1"), 1_000, 1_000);
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1416 // A read credential never stands in for a write one.
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1417 assert!(kept.get("acme--rocket", Scope::Write, Use::Internal, 1_000).is_none());
1418 assert!(kept.get("acme--booster", Scope::Read, Use::Internal, 1_000).is_none());
1419 // An hour-long credential is never handed out.
1420 assert!(kept.get("acme--rocket", Scope::Read, Use::Handout, 1_000).is_none());
1421 kept.keep("acme--rocket", Scope::Read, Use::Handout, access("h1"), 1_000, 1_000);
1422 assert_eq!(kept.get("acme--rocket", Scope::Read, Use::Handout, 1_000).unwrap().token, "h1");
1423 assert!(kept.get("acme--rocket", Scope::Read, Use::Handout, 1_000 + HANDOUT_REUSE_MS).is_none());
1424 assert!(kept.get("acme--rocket", Scope::Read, Use::Internal, 1_000 + HANDOUT_REUSE_MS).is_some());
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1425 }
1426
1427 #[test]
1428 fn a_shared_credential_is_reused_only_in_its_own_window() {
1429 let value = serde_json::to_vec(&SharedCredential {
1430 remote: "https://store.example/acme--rocket.git".to_owned(),
1431 token: "r1".to_owned(),
1432 made: 10_000,
1433 })
1434 .unwrap();
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1435 let (access, made) = shared_credential(&value, 10_000 + INTERNAL_REUSE_MS - 1, Use::Internal).unwrap();
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1436 assert_eq!(access.token, "r1");
1437 // Kept here only for what is left of its window, not a new one.
1438 assert_eq!(made, 10_000);
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1439 assert!(shared_credential(&value, 10_000 + INTERNAL_REUSE_MS, Use::Internal).is_none());
1440 assert!(shared_credential(&value, 10_000 + HANDOUT_REUSE_MS, Use::Handout).is_none());
1441 assert!(shared_credential(b"not json", 10_000, Use::Internal).is_none());
1442 // Each repository, scope and use has its own key, and none is read
1443 // from before uses differed.
1444 assert_eq!(shared_key("acme--rocket", Scope::Read, Use::Internal), "cred2:acme--rocket:read:internal");
1445 assert_ne!(shared_key("acme--rocket", Scope::Read, Use::Internal), shared_key("acme--rocket", Scope::Write, Use::Internal));
1446 assert_ne!(shared_key("acme--rocket", Scope::Read, Use::Internal), shared_key("acme--rocket", Scope::Read, Use::Handout));
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1447 }
1448
1449 #[test]
1450 fn a_shared_credential_kept_here_expires_with_the_original() {
1451 let mut kept = Credentials::default();
1452 // Made at 1_000 elsewhere, found here at 100_000.
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1453 kept.keep("acme--rocket", Scope::Read, Use::Internal, access("r1"), 1_000, 100_000);
1454 assert!(kept.get("acme--rocket", Scope::Read, Use::Internal, 100_000).is_some());
1455 assert!(kept.get("acme--rocket", Scope::Read, Use::Internal, 1_000 + INTERNAL_REUSE_MS).is_none());
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1456 }
1457
1458 #[test]
1459 fn a_turned_down_credential_is_forgotten_and_old_ones_are_dropped() {
1460 let mut kept = Credentials::default();
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1461 kept.keep("acme--rocket", Scope::Read, Use::Internal, access("r1"), 1_000, 1_000);
1462 kept.keep("acme--rocket", Scope::Write, Use::Handout, access("w1"), 1_000, 1_000);
1463 kept.keep("acme--booster", Scope::Read, Use::Internal, access("b1"), 1_000, 1_000);
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1464 kept.forget("acme--rocket");
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1465 assert!(kept.get("acme--rocket", Scope::Read, Use::Internal, 1_000).is_none());
1466 assert!(kept.get("acme--rocket", Scope::Write, Use::Handout, 1_000).is_none());
1467 assert!(kept.get("acme--booster", Scope::Read, Use::Internal, 1_000).is_some());
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1468 // Keeping another later drops the expired one from the map.
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1469 let later = 1_000 + INTERNAL_REUSE_MS;
1470 kept.keep("acme--other", Scope::Read, Use::Internal, access("o1"), later, later);
Mission control shows where you are needed and what agents landed without you; git answers in about 200ms1471 assert_eq!(kept.kept.len(), 1);
1472 }
Fast pages, required checks on the branch, self-hosted runners, honest incidents1473
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1474 #[test]
1475 fn a_remote_is_worked_out_from_where_its_namespace_starts() {
1476 // As https://developers.cloudflare.com/artifacts/api/git-protocol/ documents.
1477 let remote = "https://1e6f2cffa3f445920836e8ebe446bb58.artifacts.cloudflare.net/git/g1t/acme--rocket.git";
1478 let prefix = learn_prefix(remote, "acme--rocket").unwrap();
1479 assert_eq!(prefix, "https://1e6f2cffa3f445920836e8ebe446bb58.artifacts.cloudflare.net/git/g1t/");
1480 assert_eq!(remote_from(&prefix, "pulls--pul_1"), prefix.clone() + "pulls--pul_1.git");
1481 assert_eq!(remote_from(&prefix, "acme--rocket"), remote);
1482 // A remote that does not end in the name teaches nothing.
1483 assert_eq!(learn_prefix(remote, "rocket"), None);
1484 assert_eq!(learn_prefix("https://x/acme--rocket", "acme--rocket"), None);
1485 // And back: a remote names its key.
1486 assert_eq!(key_from_remote(remote).as_deref(), Some("acme--rocket"));
1487 assert_eq!(
1488 key_from_remote("https://a.artifacts.cloudflare.net/git/g1t-us-1/acme--rocket.git").as_deref(),
1489 Some("g1t-us-1/acme--rocket")
1490 );
1491 assert_eq!(locate("g1t-us-1/acme--rocket"), ("g1t-us-1".to_owned(), "acme--rocket".to_owned()));
1492 assert_eq!(locate("acme--rocket"), ("g1t".to_owned(), "acme--rocket".to_owned()));
1493 }
1494
1495 #[test]
Merge branch 'worktree-agent-a2013627e5ea4ab13'1496 fn the_fallback_stores_credentials_and_remotes_are_its_own() {
1497 assert_eq!(cred_key("acme--rocket", false), "acme--rocket");
1498 assert_eq!(cred_key("g1t-us-1/acme--rocket", true), "fallback:g1t-us-1/acme--rocket");
1499 // A credential Artifacts made is never handed out for the fallback
1500 // store, nor the reverse.
1501 assert_ne!(
1502 shared_key(&cred_key("acme--rocket", true), Scope::Write, Use::Internal),
1503 shared_key(&cred_key("acme--rocket", false), Scope::Write, Use::Internal)
1504 );
1505 // Its remotes name their keys as Artifacts' do.
1506 let settings = fallback::Settings::from_vars(Some("https://gitstore.example"), Some("0123456789abcdef"), Some("*"), None).unwrap();
1507 assert_eq!(key_from_remote(&settings.remote("g1t", "acme--rocket")).as_deref(), Some("acme--rocket"));
1508 assert_eq!(key_from_remote(&settings.remote("g1t-us-1", "pulls--pul_1")).as_deref(), Some("g1t-us-1/pulls--pul_1"));
1509 // Git requests to it count toward its own health, not Artifacts'.
1510 FALLBACK_BASE.with(|base| *base.borrow_mut() = Some(format!("{}/git/", settings.url)));
1511 assert_eq!(health_namespace(&settings.remote("g1t-us-1", "acme--rocket")), "g1t-us-1@fallback");
1512 assert_eq!(health_namespace("https://a.artifacts.cloudflare.net/git/g1t-us-1/acme--rocket.git"), "g1t-us-1");
1513 FALLBACK_BASE.with(|base| *base.borrow_mut() = None);
1514 }
1515
1516 #[test]
Fewer Artifacts reads: the store is asked for a handle only when needed, objects are kept in the isolate, and issue events read no workflows nobody listens for1517 fn the_memory_cache_drops_its_oldest_past_its_budget() {
1518 let mut cache = MemoryCache::default();
1519 let chunk = vec![7u8; MEMORY_CACHE_BYTES / 16];
1520 for n in 0..17 {
1521 cache.put(format!("k{n}"), &chunk);
1522 }
1523 // Sixteen chunks fit; the seventeenth pushed the first out.
1524 assert!(cache.get("k0").is_none());
1525 assert_eq!(cache.get("k16").map(|b| b.len()), Some(chunk.len()));
1526 assert!(cache.bytes <= MEMORY_CACHE_BYTES);
1527 // Too large to keep at all.
1528 cache.put("big".into(), &vec![0u8; MEMORY_CACHE_BYTES / 16 + 1]);
1529 assert!(cache.get("big").is_none());
1530 }
1531
1532 #[test]
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1533 fn binding_methods_have_snake_case_meters() {
1534 assert_eq!(meter_of("createToken"), "binding.create_token");
1535 assert_eq!(meter_of("readBlob"), "binding.read_blob");
1536 assert_eq!(meter_of("get"), "binding.get");
1537 }
1538
1539 #[test]
1540 fn reads_by_name_are_kept_under_the_refs_version_and_by_hash_for_good() {
1541 let hash = "a".repeat(40);
1542 assert_eq!(log_key(&hash, 1, Some(3)), Some(CacheKey::Forever(format!("log/{hash}-1"))));
1543 assert_eq!(log_key(&hash, 1, None), Some(CacheKey::Forever(format!("log/{hash}-1"))));
1544 // A branch is kept only when the version is known.
1545 assert_eq!(log_key("main", 1, None), None);
1546 let v3 = log_key("main", 1, Some(3)).unwrap();
1547 assert!(matches!(&v3, CacheKey::Versioned(path) if path.starts_with("vlog/3/")));
1548 // A push moves the version and leaves the old answer behind.
1549 assert_ne!(Some(v3), log_key("main", 1, Some(4)));
1550 assert_ne!(log_key("main", 1, Some(3)), log_key("main", 50, Some(3)));
1551 assert_ne!(log_key("main", 1, Some(3)), log_key("dev", 1, Some(3)));
1552 // Odd branch names make a usable address.
1553 assert!(matches!(log_key("fix/#1 %20", 1, Some(3)), Some(CacheKey::Versioned(path)) if !path.contains('#') && !path.contains(' ')));
1554 assert_eq!(branches_key(None), None);
1555 assert_ne!(branches_key(Some(1)), branches_key(Some(2)));
1556 assert!(matches!(file_key(&hash, "src/main.rs", None), Some(CacheKey::Forever(_))));
1557 assert_eq!(file_key("main", "src/main.rs", None), None);
1558 assert_ne!(file_key("main", "a", Some(1)), file_key("main", "b", Some(1)));
1559 assert_ne!(file_key("main", "a", Some(1)), file_key("main", "a", Some(2)));
Repos: a path found not to be a file is remembered for 10 minutes, so repeated misses read nothing1560 // A path noted as not a file sits beside the file's own key, and a
1561 // push (a new refs version) leaves the old note behind.
1562 let at = |version| absent_path(&file_key("main", "a", Some(version)).unwrap());
1563 assert!(at(1).starts_with("absent/vfile/1/"));
1564 assert_ne!(at(1), at(2));
1565 assert_eq!(absent_path(&file_key(&hash, "a", None).unwrap()), format!("absent/file/{hash}/{}", g1t_secrets::sha256_hex("a")));
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily1566 }
Fast pages, required checks on the branch, self-hosted runners, honest incidents1567
1568 #[test]
1569 fn only_full_lowercase_hashes_are_kept() {
1570 assert!(is_commit_hash(&"a".repeat(40)));
1571 assert!(is_commit_hash(&"0123456789abcdef".repeat(4)));
1572 assert!(!is_commit_hash("main"));
1573 assert!(!is_commit_hash(&"A".repeat(40)));
1574 assert!(!is_commit_hash(&"a".repeat(39)));
1575 }
1576}

This file's history is long; its oldest lines are credited to the oldest commit read.