flagon-io/g1t

public

Where people and agents ship software together. The open-source git platform for the whole job: issues, agents, checks and deploys to the edge.

git agents cloudflare rust forge mcp

Commit

Blame is kept by commit and path, reads ahead and stops in time, and crawlers are asked to leave it alone; a push works out its repository's About, and a run's reads are kept for the next. Workers Logs at full sampling (2026-10-14, ~21:30-22:15 UTC) had blame as 56 of 101 repos answers over a second (median 2.4 s, up to 20.3 s), and the site's own logs showed who asked: 1,154 blame pages in four hours, about 1,015 of them ClaudeBot, 93 Googlebot, the rest Amazonbot, GPTBot and Reflectionbot, and a handful from people, each a file at a commit (`/flagon-io/g1t/blob/<sha>/<path>?blame=1`, `/pulls/<id>/blob/...`). Nothing was kept, so every one walked up to 400 commits again, reading the trees to the file one commit at a time, each read a round trip. Now repos keeps a blame in the colo's cache by repository, commit and path (blame.rs `cache_url`, for good; one cut short for ten minutes), finding the commit first when asked by branch; the walk reads the trees of the next 32 commits level by level at once, then at once the texts it will compare, reads each tree and text once, stops as soon as every line has its commit, and stops after 6 s (BLAME_WALK_MS) or 2,500 reads, giving what is left to the oldest commit it reached them in, marked partial as a history past 400 commits already was. robots.txt disallows `/?blame=` and `/&blame=` (app/lib/robots.ts), the Blame links are `nofollow`, and a blame page says `noindex`; the file itself stays open to crawlers. The About's 22.7 s and 22.0 s "about" invocations were its background run hitting its 22 s budget in the answer's waitUntil (the answer itself was quick): its history was read 200 commits a page, and each page after the first starts wherever the last ended, which moves with every push, so g1t's 728 first-parent commits were four reads with three of them read whole from the store again each time; pages are now 1,000, the length lib.rs already reads (MAX_ANCESTRY), so most repositories are one spliced read. And refresh_later never waited for the cache writes its run started (store.rs keeps trees, sizes and logs through Deferred, which only push handling settled), so a run could leave the next to read the same again; it now settles them. A push that moves the default branch works the About out after recording the push, within what is left of 24 s, so a view after a push finds it current; a run that is slow or partial logs `about_run` with `took_ms`. last_commits no longer waits on its two writes (the memo, which also goes to KV, and the finished answer) before answering. Blob had no waste worth taking out: its slow answers are the store's first read of a file, which is then kept. Repos and work log a quarter of requests (head_sampling_rate 0.25), down from every one for the stall hunt (no ~60 s stall recurred); slow RPC answers still log `answered_ms`. The deploy guide's Logs section, why-blame and managing repositories say so. Checked: cargo test -p g1t-repos (356 passing, 11 new), cargo test -p g1t-api openapi (9), npm test in apps/web (944 passing, 2 new), npx tsc -b --force, npm run test:deploy (92), node scripts/deploy.mjs manifest --check, and the docs build.

G1t-Session:: ses_01m4y12gdde2r9q832m7xk87kd

syntaqxcommitted yesterdayParent5fd6a02Browse files

241	241		Ok((Outcome::Ok(answer), refresh))
242	242		}
243	243
244		−	/// Works out `repo`'s About again, if no one else is. In the
245		−	/// background: a failure is logged and the lease let go early.
246		−	pub(crate) async fn refresh(&self, repo: &Repo) {
	244	+	/// Works out `repo`'s About again, if no one else is, reading for
	245	+	/// `budget_ms` at most. In the background: a failure is logged and the
	246	+	/// lease let go early. A run that was slow or partial is logged with how
	247	+	/// long it took (`about_run`), as a slow RPC answer is.
	248	+	pub(crate) async fn refresh(&self, repo: &Repo, budget_ms: u64) {
247	249		let db = self.db();
	250	+	let started = now_ms();
248	251		match stats::claim(db, &repo.id, now_ms()).await {
249	252		Ok(true) => {}
250	253		Ok(false) => return,
⋯
258	261		let Some(head) = git.log(&repo.default_branch, 1).await?.into_iter().next() else {
259	262		return Ok(None);
260	263		};
261		−	stats::work_out(&git, &head.hash, &head.tree_hash, self.identity.as_ref()).await.map(Some)
	264	+	stats::work_out(&git, &head.hash, &head.tree_hash, self.identity.as_ref(), budget_ms).await.map(Some)
262	265		};
263	266		match worked.await {
264	267		Ok(Some(worked)) => {
	268	+	let took = now_ms().saturating_sub(started);
	269	+	if worked.partial \|\| took >= SLOW_RUN_MS {
	270	+	worker::console_log!(
	271	+	"{}",
	272	+	serde_json::json!({ "about_run": repo.id, "took_ms": took, "partial": worked.partial, "commits": worked.commits })
	273	+	);
	274	+	}
265	275		if let Err(error) = stats::keep(db, &repo.id, &worked).await {
266	276		worker::console_error!("the About of {} was worked out but not kept: {error}", repo.id);
267	277		stats::release(db, &repo.id).await;
⋯
657	667		}
658	668		}
659	669
	670	+	/// Runs this long or longer are logged (`refresh`).
	671	+	const SLOW_RUN_MS: u64 = 5_000;
	672	+
660	673		/// Works out `repo`'s About after the answer has gone, when an answer
661		−	/// found it behind.
	674	+	/// found it behind. What the run read is kept for the next one (trees,
	675	+	/// blob sizes and history by hash, store.rs) by writes it starts as it
	676	+	/// goes; the `waitUntil` lasts until they are done, or the next run would
	677	+	/// read the same again.
662	678		pub(crate) fn refresh_later(env: &Env, ctx: &worker::Context, refresh: Refresh) {
663	679		let Some(repo) = refresh else { return };
664	680		let env = env.clone();
665	681		ctx.wait_until(async move {
666	682		match crate::service(&env) {
667		−	Ok(repos) => repos.refresh(&repo).await,
	683	+	Ok(repos) => {
	684	+	repos.refresh(&repo, stats::BUDGET_MS).await;
	685	+	repos.deferred.settle().await;
	686	+	}
668	687		Err(error) => worker::console_error!("the About of {} was not worked out: {error}", repo.id),
669	688		}
670	689		});

5	5		//! none of them was written by the commit. Following every parent, not just
6	6		//! the first, matters here: catch-up merges bring the default branch into a
7	7		//! pull request, and their lines belong to whoever wrote them on that branch.
	8	+	//!
	9	+	//! Reading is what costs: each commit's file is found by reading the trees
	10	+	//! on the way to it, and each read is a round trip to the store or its cache.
	11	+	//! So the walk reads ahead: the trees of the next [`READ_AHEAD`] commits,
	12	+	//! level by level and each level at once, and then, at once, the texts of the
	13	+	//! versions it will compare. The walk stops when every line has its commit,
	14	+	//! after [`MAX_COMMITS`] commits, before more than the caller's reads of the
	15	+	//! store, or once the caller's time is up; stopped early, a line goes to the
	16	+	//! oldest commit the walk reached it in, and the answer is marked partial.
	17	+	//!
	18	+	//! A blame of a commit and path never changes, so `lib.rs` keeps the answer
	19	+	//! by repository, commit and path ([`cache_url`]).
8	20
9		−	use std::collections::HashMap;
	21	+	use std::collections::{HashMap, HashSet};
	22	+	use std::rc::Rc;
10	23
11	24		use g1t_contracts::repos::{Blame, BlameRange, Commit, EntryKind, TreeEntry};
12	25		use similar::{ChangeTag, TextDiff};
⋯
16	29
17	30		/// How far back the history is searched. Lines older than this are given to
18	31		/// the oldest commit reached, and the answer is marked partial.
19		−	const MAX_COMMITS: u32 = 400;
	32	+	pub const MAX_COMMITS: u32 = 400;
20	33		/// Files larger than this are not blamed.
21	34		const MAX_FILE_BYTES: usize = 512 * 1024;
	35	+	/// Commits whose trees and texts are read together, ahead of the walk.
	36	+	const READ_AHEAD: usize = 32;
	37	+	/// Reads of the store one blame makes at most. Each costs up to three
	38	+	/// subrequests (a Cache API look, the store, a Cache API put), so this keeps
	39	+	/// a call well inside the 10,000 a Worker invocation may make.
	40	+	pub const MAX_READS: u32 = 2_500;
	41	+
	42	+	/// Where a blame is kept: by repository, the commit it was read at, and a
	43	+	/// hash of the path. `v1`: a change to how lines are credited moves it.
	44	+	pub fn cache_url(repo_id: &str, head: &str, path: &str) -> String {
	45	+	format!("https://blame.g1t.internal/v1/{repo_id}/{head}/{}", g1t_secrets::sha256_hex(path))
	46	+	}
	47	+
	48	+	/// How long a blame is kept. One that ran to its end (the first commit, or
	49	+	/// [`MAX_COMMITS`]) is the same every time, so for good; one cut short by
	50	+	/// time or reads could do better on a quieter store, so briefly.
	51	+	pub fn cache_control(blamed: &Blamed) -> &'static str {
	52	+	if blamed.cut_short { "max-age=600" } else { "max-age=31536000, immutable" }
	53	+	}
	54	+
	55	+	/// A blame, and whether time or reads cut it short.
	56	+	#[derive(Debug)]
	57	+	pub struct Blamed {
	58	+	pub blame: Blame,
	59	+	pub cut_short: bool,
	60	+	}
	61	+
	62	+	/// A version of the file: its lines, each ended by a newline, as the diff
	63	+	/// compares them, and how many there are.
	64	+	struct Text {
	65	+	joined: String,
	66	+	lines: usize,
	67	+	}
22	68
	69	+	impl Text {
	70	+	fn of(text: &str) -> Text {
	71	+	let mut joined = String::with_capacity(text.len() + 1);
	72	+	let mut lines = 0;
	73	+	// Every line ends in a newline, so a last line matches the same line elsewhere.
	74	+	for line in text.lines() {
	75	+	joined.push_str(line);
	76	+	joined.push('\n');
	77	+	lines += 1;
	78	+	}
	79	+	Text { joined, lines }
	80	+	}
	81	+	}
	82	+
23	83		/// Reads files out of commits, remembering trees and texts already read:
24	84		/// most commits share most of their trees.
25	85		struct Reader<'a, R: GitRepo> {
26	86		repo: &'a R,
27		−	trees: HashMap<String, Vec<TreeEntry>>,
28		−	texts: HashMap<String, Option<String>>,
	87	+	trees: HashMap<String, Option<Vec<TreeEntry>>>,
	88	+	texts: HashMap<String, Option<Rc<Text>>>,
	89	+	/// Reads of the store so far, trees, texts and history.
	90	+	reads: u32,
29	91		}
30	92
31	93		impl<'a, R: GitRepo> Reader<'a, R> {
	94	+	/// Reads the trees not read yet, all at once.
	95	+	async fn read_trees(&mut self, hashes: Vec<String>) -> Result<()> {
	96	+	let wanted: Vec<String> = hashes
	97	+	.into_iter()
	98	+	.filter(\|hash\| !self.trees.contains_key(hash))
	99	+	.collect::<HashSet<_>>()
	100	+	.into_iter()
	101	+	.collect();
	102	+	self.reads += wanted.len() as u32;
	103	+	let found = futures_util::future::join_all(wanted.iter().map(\|hash\| self.repo.read_tree(hash))).await;
	104	+	for (hash, tree) in wanted.into_iter().zip(found) {
	105	+	self.trees.insert(hash, tree?);
	106	+	}
	107	+	Ok(())
	108	+	}
	109	+
	110	+	/// Reads, level by level and each level at once, the trees on the way
	111	+	/// to `path` in each of `roots`.
	112	+	async fn read_paths(&mut self, roots: Vec<String>, path: &str) -> Result<()> {
	113	+	let parts: Vec<&str> = path.split('/').filter(\|part\| !part.is_empty()).collect();
	114	+	let mut level = roots;
	115	+	for (index, part) in parts.iter().enumerate() {
	116	+	self.read_trees(level.clone()).await?;
	117	+	if index + 1 == parts.len() {
	118	+	break;
	119	+	}
	120	+	level = level
	121	+	.iter()
	122	+	.filter_map(\|hash\| {
	123	+	let entries = self.trees.get(hash)?.as_ref()?;
	124	+	entries.iter().find(\|entry\| entry.name == *part && entry.kind == EntryKind::Tree).map(\|entry\| entry.hash.clone())
	125	+	})
	126	+	.collect();
	127	+	}
	128	+	Ok(())
	129	+	}
	130	+
	131	+	/// Reads the texts not read yet, all at once.
	132	+	async fn read_texts(&mut self, blobs: Vec<String>) -> Result<()> {
	133	+	let wanted: Vec<String> =
	134	+	blobs.into_iter().filter(\|blob\| !self.texts.contains_key(blob)).collect::<HashSet<_>>().into_iter().collect();
	135	+	self.reads += wanted.len() as u32;
	136	+	let found = futures_util::future::join_all(wanted.iter().map(\|blob\| self.repo.read_blob(blob))).await;
	137	+	for (blob, bytes) in wanted.into_iter().zip(found) {
	138	+	let text = bytes?
	139	+	.filter(\|bytes\| bytes.len() <= MAX_FILE_BYTES && !bytes.contains(&0))
	140	+	.and_then(\|bytes\| String::from_utf8(bytes).ok())
	141	+	.map(\|text\| Rc::new(Text::of(&text)));
	142	+	self.texts.insert(blob, text);
	143	+	}
	144	+	Ok(())
	145	+	}
	146	+
32	147		/// The blob hash of `path` in the tree `root`, if it is a file there.
33	148		async fn blob_at(&mut self, root: &str, path: &str) -> Result<Option<String>> {
34	149		let mut tree = root.to_owned();
35	150		let parts: Vec<&str> = path.split('/').filter(\|part\| !part.is_empty()).collect();
36	151		for (index, part) in parts.iter().enumerate() {
37	152		if !self.trees.contains_key(&tree) {
38		−	let entries = self.repo.read_tree(&tree).await?.unwrap_or_default();
39		−	self.trees.insert(tree.clone(), entries);
	153	+	self.read_trees(vec![tree.clone()]).await?;
40	154		}
41		−	let Some(entry) = self.trees[&tree].iter().find(\|entry\| entry.name == *part) else {
	155	+	let Some(entry) = self.trees[&tree].as_ref().and_then(\|entries\| entries.iter().find(\|entry\| entry.name == *part)) else {
42	156		return Ok(None);
43	157		};
44	158		let last = index == parts.len() - 1;
⋯
51	165		Ok(None)
52	166		}
53	167
54		−	/// The lines of a blob, or `None` if it is binary or too large.
55		−	async fn lines(&mut self, blob: &str) -> Result<Option<Vec<String>>> {
	168	+	/// A version's text, or `None` if it is binary or too large.
	169	+	async fn text(&mut self, blob: &str) -> Result<Option<Rc<Text>>> {
56	170		if !self.texts.contains_key(blob) {
57		−	let text = self
58		−	.repo
59		−	.read_blob(blob)
60		−	.await?
61		−	.filter(\|bytes\| bytes.len() <= MAX_FILE_BYTES && !bytes.contains(&0))
62		−	.and_then(\|bytes\| String::from_utf8(bytes).ok());
63		−	self.texts.insert(blob.to_owned(), text);
	171	+	self.read_texts(vec![blob.to_owned()]).await?;
64	172		}
65		−	Ok(self.texts[blob]
66		−	.as_ref()
67		−	.map(\|text\| text.lines().map(str::to_owned).collect()))
	173	+	Ok(self.texts[blob].clone())
68	174		}
	175	+
	176	+	/// Reads ahead for `commits`: the trees to `path` in each, then the
	177	+	/// texts of each version that differs from the next commit's (its
	178	+	/// parent on the walk), which are the ones the walk compares.
	179	+	async fn read_ahead(&mut self, commits: &[Commit], path: &str) -> Result<()> {
	180	+	self.read_paths(commits.iter().map(\|commit\| commit.tree_hash.clone()).collect(), path).await?;
	181	+	let mut blobs = Vec::with_capacity(commits.len());
	182	+	for commit in commits {
	183	+	blobs.push(self.blob_at(&commit.tree_hash, path).await?);
	184	+	}
	185	+	let mut wanted = Vec::new();
	186	+	for pair in blobs.windows(2) {
	187	+	if let [Some(child), Some(parent)] = pair
	188	+	&& child != parent
	189	+	{
	190	+	wanted.push(child.clone());
	191	+	wanted.push(parent.clone());
	192	+	}
	193	+	}
	194	+	self.read_texts(wanted).await
	195	+	}
69	196		}
70	197
71	198		/// For each line of `new`, the index of the same, unchanged line in `old`.
72		−	fn carried(old: &[String], new: &[String]) -> Vec<Option<usize>> {
73		−	// Every line ends in a newline, so a last line matches the same line elsewhere.
74		−	let text = \|lines: &[String]\| lines.iter().map(\|line\| format!("{line}\n")).collect::<String>();
75		−	let (before, after) = (text(old), text(new));
76		−	let diff = TextDiff::from_lines(&before, &after);
77		−	let mut map = vec![None; new.len()];
	199	+	fn carried(old: &Text, new: &Text) -> Vec<Option<usize>> {
	200	+	let diff = TextDiff::from_lines(&old.joined, &new.joined);
	201	+	let mut map = vec![None; new.lines];
78	202		for change in diff.iter_all_changes() {
79	203		if change.tag() == ChangeTag::Equal
80	204		&& let (Some(o), Some(n)) = (change.old_index(), change.new_index())
81		−	&& let Some(slot) = map.get_mut(n) {
82		−	*slot = Some(o);
83		−	}
	205	+	&& let Some(slot) = map.get_mut(n)
	206	+	{
	207	+	*slot = Some(o);
	208	+	}
84	209		}
85	210		map
86	211		}
87	212
88	213		/// Who last changed each line of `path` as of `head`. `None` if the file is
89		−	/// missing or not text.
	214	+	/// missing or not text. The walk stops before more than `max_reads` reads of
	215	+	/// the store, or once `out_of_time` says so (asked between read-aheads,
	216	+	/// after the first).
90	217		pub async fn blame<R: GitRepo>(
91	218		repo: &R,
92	219		head: &str,
93	220		path: &str,
94		−	) -> Result<Option<Blame>> {
	221	+	out_of_time: &dyn Fn() -> bool,
	222	+	max_reads: u32,
	223	+	) -> Result<Option<Blamed>> {
95	224		let history = repo.log(head, MAX_COMMITS).await?;
96	225		let Some(first) = history.first() else {
97	226		return Ok(None);
⋯
100	229		repo,
101	230		trees: HashMap::new(),
102	231		texts: HashMap::new(),
	232	+	reads: 1,
103	233		};
	234	+	// The first read-ahead also finds the file at the head.
	235	+	let first_window = &history[..history.len().min(READ_AHEAD + 1)];
	236	+	reader.read_ahead(first_window, path).await?;
	237	+	let mut read_ahead_to = first_window.len();
104	238		let Some(blob) = reader.blob_at(&first.tree_hash, path).await? else {
105	239		return Ok(None);
106	240		};
107		−	let Some(lines) = reader.lines(&blob).await? else {
	241	+	let Some(text) = reader.text(&blob).await? else {
108	242		return Ok(None);
109	243		};
110		−	let total = lines.len();
111		−	let by_hash: HashMap<&str, &Commit> =
112		−	history.iter().map(\|commit\| (commit.hash.as_str(), commit)).collect();
	244	+	let total = text.lines;
	245	+	let by_hash: HashMap<&str, &Commit> = history.iter().map(\|commit\| (commit.hash.as_str(), commit)).collect();
	246	+	let segments = path.split('/').filter(\|part\| !part.is_empty()).count();
	247	+	// The most one read-ahead reads: every tree on the way, two texts each.
	248	+	let window_reads = (READ_AHEAD * (segments + 2)) as u32;
113	249
114	250		// For each commit still to visit: (line in its version, line in the head's).
115	251		let mut pending: HashMap<String, Vec<(usize, usize)>> = HashMap::new();
116	252		pending.insert(first.hash.clone(), (0..total).map(\|line\| (line, line)).collect());
117	253		let mut owner: Vec<Option<String>> = vec![None; total];
118	254		let mut partial = false;
	255	+	let mut cut_short = false;
119	256
120	257		// The log is newest first, so a commit is reached after its children.
121		−	for commit in &history {
	258	+	for (index, commit) in history.iter().enumerate() {
	259	+	if pending.is_empty() {
	260	+	break;
	261	+	}
	262	+	if index + 1 >= read_ahead_to && read_ahead_to < history.len() {
	263	+	if out_of_time() \|\| reader.reads + window_reads > max_reads {
	264	+	cut_short = true;
	265	+	break;
	266	+	}
	267	+	let window = &history[index..history.len().min(index + READ_AHEAD + 1)];
	268	+	reader.read_ahead(window, path).await?;
	269	+	read_ahead_to = index + window.len();
	270	+	}
122	271		let Some(mut lines_here) = pending.remove(&commit.hash) else {
123	272		continue;
124	273		};
⋯
127	276		let Some(blob) = reader.blob_at(&commit.tree_hash, path).await? else {
128	277		continue;
129	278		};
130		−	let Some(text) = reader.lines(&blob).await? else {
131		−	continue;
132		−	};
	279	+	// Read only when a parent's version differs.
	280	+	let mut text: Option<Rc<Text>> = None;
133	281		let mut unexplained = lines_here;
134	282		for parent_hash in &commit.parents {
135	283		if unexplained.is_empty() {
⋯
146	294		let handed: Vec<(usize, usize)> = if parent_blob == blob {
147	295		std::mem::take(&mut unexplained)
148	296		} else {
149		−	let Some(parent_text) = reader.lines(&parent_blob).await? else {
	297	+	if text.is_none() {
	298	+	text = reader.text(&blob).await?;
	299	+	}
	300	+	let (Some(here), Some(parent_text)) = (&text, reader.text(&parent_blob).await?) else {
	301	+	// A version that is not text: its lines are taken as
	302	+	// this commit's.
150	303		continue;
151	304		};
152		−	let map = carried(&parent_text, &text);
	305	+	let map = carried(&parent_text, here);
153	306		let (moved, kept): (Vec<_>, Vec<_>) = unexplained
154	307		.into_iter()
155	308		.partition(\|(here, _)\| map.get(*here).copied().flatten().is_some());
⋯
167	320		owner[final_line] = Some(commit.hash.clone());
168	321		}
169	322		}
170		−	// Lines handed to commits older than the history read.
	323	+	// Lines the walk did not finish with: each goes to the oldest commit it
	324	+	// was followed to.
171	325		if !pending.is_empty() {
172	326		partial = true;
173		−	let oldest = history.last().map(\|commit\| commit.hash.clone());
174		−	for line in owner.iter_mut().filter(\|line\| line.is_none()) {
175		−	*line = oldest.clone();
	327	+	for (hash, lines) in pending {
	328	+	for (_, final_line) in lines {
	329	+	if owner[final_line].is_none() {
	330	+	owner[final_line] = Some(hash.clone());
	331	+	}
	332	+	}
176	333		}
177	334		}
178	335
⋯
188	345		let mut commits: Vec<Commit> = Vec::new();
189	346		for range in &ranges {
190	347		if !commits.iter().any(\|commit\| commit.hash == range.commit)
191		−	&& let Some(commit) = by_hash.get(range.commit.as_str()) {
192		−	commits.push((*commit).clone());
193		−	}
	348	+	&& let Some(commit) = by_hash.get(range.commit.as_str())
	349	+	{
	350	+	commits.push((*commit).clone());
	351	+	}
194	352		}
195		−	Ok(Some(Blame {
196		−	head: first.hash.clone(),
197		−	ranges,
198		−	commits,
199		−	partial,
	353	+	Ok(Some(Blamed {
	354	+	blame: Blame {
	355	+	head: first.hash.clone(),
	356	+	ranges,
	357	+	commits,
	358	+	partial,
	359	+	},
	360	+	cut_short,
200	361		}))
201	362		}
202	363
203	364		#[cfg(test)]
204	365		mod tests {
	366	+	use std::cell::{Cell, RefCell};
	367	+	use std::future::Future;
	368	+	use std::pin::pin;
	369	+	use std::task::{Context, Poll, Waker};
	370	+
	371	+	use g1t_contracts::repos::{Branch, GitAccess, Signature};
	372	+
205	373		use super::*;
	374	+	use crate::store::Scope;
206	375
207		−	fn lines(text: &str) -> Vec<String> {
208		−	text.lines().map(str::to_owned).collect()
	376	+	fn run<F: Future>(future: F) -> F::Output {
	377	+	match pin!(future).as_mut().poll(&mut Context::from_waker(Waker::noop())) {
	378	+	Poll::Ready(output) => output,
	379	+	Poll::Pending => panic!("the fake store never waits"),
	380	+	}
209	381		}
210	382
	383	+	fn text(text: &str) -> Text {
	384	+	Text::of(text)
	385	+	}
	386	+
211	387		#[test]
212	388		fn unchanged_lines_are_carried_to_the_parent() {
213		−	let map = carried(&lines("a\nb\nc"), &lines("a\nx\nb\nc"));
	389	+	let map = carried(&text("a\nb\nc"), &text("a\nx\nb\nc"));
214	390		assert_eq!(map, vec![Some(0), None, Some(1), Some(2)]);
215	391		}
216	392
217	393		#[test]
218	394		fn a_changed_line_is_not_carried() {
219		−	let map = carried(&lines("a\nb"), &lines("a\nB"));
	395	+	let map = carried(&text("a\nb"), &text("a\nB"));
220	396		assert_eq!(map, vec![Some(0), None]);
221	397		}
	398	+
	399	+	#[test]
	400	+	fn a_last_line_without_a_newline_matches_one_with() {
	401	+	let map = carried(&text("a\nb\n"), &text("a\nb"));
	402	+	assert_eq!(map, vec![Some(0), Some(1)]);
	403	+	}
	404	+
	405	+	/// A repository whose history is `src/f.txt` growing a line per commit.
	406	+	#[derive(Default)]
	407	+	struct Fake {
	408	+	trees: HashMap<String, Vec<TreeEntry>>,
	409	+	blobs: HashMap<String, String>,
	410	+	history: Vec<Commit>,
	411	+	trees_read: RefCell<Vec<String>>,
	412	+	blobs_read: RefCell<Vec<String>>,
	413	+	}
	414	+
	415	+	impl GitRepo for Fake {
	416	+	async fn access(&self, _scope: Scope) -> Result<GitAccess> {
	417	+	unimplemented!()
	418	+	}
	419	+	async fn branches(&self) -> Result<Vec<Branch>> {
	420	+	Ok(Vec::new())
	421	+	}
	422	+	async fn log(&self, git_ref: &str, limit: u32) -> Result<Vec<Commit>> {
	423	+	let start = if git_ref == "main" { Some(0) } else { self.history.iter().position(\|commit\| commit.hash == git_ref) };
	424	+	Ok(start.map(\|start\| self.history.iter().skip(start).take(limit as usize).cloned().collect()).unwrap_or_default())
	425	+	}
	426	+	async fn parents(&self, _commit_hash: &str) -> Result<Option<Vec<String>>> {
	427	+	Ok(None)
	428	+	}
	429	+	async fn read_tree(&self, tree_hash: &str) -> Result<Option<Vec<TreeEntry>>> {
	430	+	self.trees_read.borrow_mut().push(tree_hash.to_owned());
	431	+	Ok(self.trees.get(tree_hash).cloned())
	432	+	}
	433	+	async fn read_blob(&self, blob_hash: &str) -> Result<Option<Vec<u8>>> {
	434	+	self.blobs_read.borrow_mut().push(blob_hash.to_owned());
	435	+	Ok(self.blobs.get(blob_hash).map(\|text\| text.as_bytes().to_vec()))
	436	+	}
	437	+	async fn read_file(&self, _git_ref: &str, _path: &str) -> Result<Option<Vec<u8>>> {
	438	+	Ok(None)
	439	+	}
	440	+	async fn fork(&self, _target_key: &str, _default_branch_only: bool) -> Result<()> {
	441	+	Ok(())
	442	+	}
	443	+	}
	444	+
	445	+	fn commit(hash: &str, tree: &str, parent: Option<&str>) -> Commit {
	446	+	Commit {
	447	+	hash: hash.into(),
	448	+	tree_hash: tree.into(),
	449	+	message: format!("commit {hash}"),
	450	+	author: Signature { name: "a".into(), email: "a@example.com".into() },
	451	+	parents: parent.map(\|p\| vec![p.to_owned()]).unwrap_or_default(),
	452	+	authored_at: String::new(),
	453	+	}
	454	+	}
	455	+
	456	+	/// `commits` commits, newest first: commit n appends line n to
	457	+	/// `src/f.txt`. Every tenth commit only touches `README.md` beside it,
	458	+	/// so `src` stays the same tree there. The repository root changes in
	459	+	/// every commit.
	460	+	fn growing(commits: usize) -> Fake {
	461	+	let mut fake = Fake::default();
	462	+	let mut lines: Vec<String> = Vec::new();
	463	+	for n in 0..commits {
	464	+	let changes_file = n == 0 \|\| n % 10 != 0;
	465	+	if changes_file {
	466	+	lines.push(format!("line {n}"));
	467	+	}
	468	+	let blob = format!("b{}", lines.len());
	469	+	fake.blobs.insert(blob.clone(), lines.join("\n"));
	470	+	let src = format!("s{}", lines.len());
	471	+	fake.trees.insert(src.clone(), vec![TreeEntry { name: "f.txt".into(), hash: blob, kind: EntryKind::Blob }]);
	472	+	let root = format!("r{n}");
	473	+	fake.trees.insert(
	474	+	root.clone(),
	475	+	vec![
	476	+	TreeEntry { name: "README.md".into(), hash: format!("readme{n}"), kind: EntryKind::Blob },
	477	+	TreeEntry { name: "src".into(), hash: src, kind: EntryKind::Tree },
	478	+	],
	479	+	);
	480	+	let parent = (n > 0).then(\|\| format!("c{}", n - 1));
	481	+	fake.history.insert(0, commit(&format!("c{n}"), &root, parent.as_deref()));
	482	+	}
	483	+	fake
	484	+	}
	485	+
	486	+	fn owners(blame: &Blame) -> Vec<(u32, u32, String)> {
	487	+	blame.ranges.iter().map(\|range\| (range.start, range.end, range.commit.clone())).collect()
	488	+	}
	489	+
	490	+	#[test]
	491	+	fn each_line_goes_to_the_commit_that_wrote_it() {
	492	+	let fake = growing(5);
	493	+	let blamed = run(blame(&fake, "main", "src/f.txt", &\|\| false, MAX_READS)).unwrap().unwrap();
	494	+	assert!(!blamed.cut_short && !blamed.blame.partial);
	495	+	assert_eq!(blamed.blame.head, "c4");
	496	+	assert_eq!(
	497	+	owners(&blamed.blame),
	498	+	(0..5).map(\|n\| (n + 1, n + 1, format!("c{n}"))).collect::<Vec<_>>()
	499	+	);
	500	+	assert_eq!(blamed.blame.commits.len(), 5);
	501	+	}
	502	+
	503	+	#[test]
	504	+	fn a_missing_or_binary_file_has_no_blame() {
	505	+	let mut fake = growing(3);
	506	+	assert!(run(blame(&fake, "main", "src/nope.txt", &\|\| false, MAX_READS)).unwrap().is_none());
	507	+	assert!(run(blame(&fake, "unknown", "src/f.txt", &\|\| false, MAX_READS)).unwrap().is_none());
	508	+	fake.blobs.insert("b3".into(), "a\0b".into());
	509	+	assert!(run(blame(&fake, "main", "src/f.txt", &\|\| false, MAX_READS)).unwrap().is_none());
	510	+	}
	511	+
	512	+	#[test]
	513	+	fn each_tree_and_text_is_read_once() {
	514	+	let fake = growing(60);
	515	+	let blamed = run(blame(&fake, "main", "src/f.txt", &\|\| false, MAX_READS)).unwrap().unwrap();
	516	+	assert!(!blamed.cut_short);
	517	+	let trees = fake.trees_read.borrow();
	518	+	assert_eq!(trees.len(), trees.iter().collect::<HashSet<_>>().len(), "no tree is read twice");
	519	+	let blobs = fake.blobs_read.borrow();
	520	+	assert_eq!(blobs.len(), blobs.iter().collect::<HashSet<_>>().len(), "no text is read twice");
	521	+	// Each root, and each distinct `src`, once.
	522	+	assert_eq!(trees.len(), 60 + 55);
	523	+	}
	524	+
	525	+	#[test]
	526	+	fn the_walk_stops_once_every_line_has_its_commit() {
	527	+	// The newest 30 commits rewrite the whole file, so nothing older is needed.
	528	+	let mut fake = growing(200);
	529	+	fake.blobs.insert("b181".into(), "only line".into());
	530	+	let blamed = run(blame(&fake, "main", "src/f.txt", &\|\| false, MAX_READS)).unwrap().unwrap();
	531	+	assert_eq!(owners(&blamed.blame), vec![(1, 1, "c199".to_owned())]);
	532	+	assert!(fake.trees_read.borrow().len() < 2 * (READ_AHEAD + 1) * 2, "read only the first read-ahead or two");
	533	+	}
	534	+
	535	+	#[test]
	536	+	fn out_of_time_stops_the_walk_and_credits_the_oldest_commit_reached() {
	537	+	let fake = growing(100);
	538	+	let asked = Cell::new(0);
	539	+	let out_of_time = \|\| {
	540	+	asked.set(asked.get() + 1);
	541	+	true
	542	+	};
	543	+	let blamed = run(blame(&fake, "main", "src/f.txt", &out_of_time, MAX_READS)).unwrap().unwrap();
	544	+	assert!(blamed.cut_short && blamed.blame.partial);
	545	+	assert_eq!(asked.get(), 1, "asked once, after the first read-ahead");
	546	+	// The newest lines still have their own commits; the rest stop at
	547	+	// the last commit of the first read-ahead.
	548	+	let ranges = owners(&blamed.blame);
	549	+	assert_eq!(ranges.last().unwrap().2, "c99");
	550	+	let reached = format!("c{}", 99 - READ_AHEAD);
	551	+	assert_eq!(ranges[0].2, reached);
	552	+	assert_eq!(cache_control(&blamed), "max-age=600");
	553	+	}
	554	+
	555	+	#[test]
	556	+	fn reads_are_bounded() {
	557	+	let fake = growing(300);
	558	+	let max = 150;
	559	+	let blamed = run(blame(&fake, "main", "src/f.txt", &\|\| false, max)).unwrap().unwrap();
	560	+	assert!(blamed.cut_short && blamed.blame.partial);
	561	+	let reads = fake.trees_read.borrow().len() + fake.blobs_read.borrow().len() + 1;
	562	+	assert!(reads <= max as usize, "{reads} reads, at most {max}");
	563	+	}
	564	+
	565	+	#[test]
	566	+	fn a_history_longer_than_is_read_is_partial_but_kept_for_good() {
	567	+	let fake = growing(MAX_COMMITS as usize + 20);
	568	+	let blamed = run(blame(&fake, "main", "src/f.txt", &\|\| false, 100_000)).unwrap().unwrap();
	569	+	assert!(blamed.blame.partial && !blamed.cut_short);
	570	+	assert_eq!(cache_control(&blamed), "max-age=31536000, immutable");
	571	+	}
	572	+
	573	+	#[test]
	574	+	fn a_blame_is_kept_by_repository_commit_and_path() {
	575	+	let url = cache_url("repo_1", "abc", "src/f.txt");
	576	+	assert!(url.starts_with("https://blame.g1t.internal/v1/repo_1/abc/"));
	577	+	assert_ne!(url, cache_url("repo_1", "abc", "src/g.txt"));
	578	+	assert_ne!(url, cache_url("repo_1", "abd", "src/f.txt"));
	579	+	assert_ne!(url, cache_url("repo_2", "abc", "src/f.txt"));
	580	+	// A path never reaches the URL as it is.
	581	+	assert!(!cache_url("r", "h", "a/../b?x#y").contains(".."));
	582	+	}
222	583		}

100	100		const LAST_COMMITS_WALK_MS: u64 = 20_000;
101	101		/// How long a path's history may walk when the caller gives no budget.
102	102		const PATH_LOG_WALK_MS: u64 = 10_000;
	103	+	/// How long a blame walks before it answers with what it has: a page waits
	104	+	/// on it. A blame cut short is kept only briefly (blame.rs, `cache_control`).
	105	+	const BLAME_WALK_MS: u64 = 6_000;
103	106
104	107		/// One path segment, percent-encoded for a cache key.
105	108		fn urlencoding_segment(segment: &str) -> String {
⋯
978	981		};
979	982		let git = self.read_git(&repo).await?;
980	983		let git_ref = a.git_ref.unwrap_or_else(\|\| repo.default_branch.clone());
981		−	Ok(match blame::blame(&git, &git_ref, &a.file_path).await? {
982		−	Some(blame) => Outcome::Ok(blame),
983		−	None => not_found(),
984		−	})
	984	+	// The commit the ref names, so the answer can be kept by it: a
	985	+	// blame of a commit and path never changes. Most blames are of a
	986	+	// commit already (the site links files by commit, and crawlers
	987	+	// follow those links), which needs no read.
	988	+	let head = if store::is_commit_hash(&git_ref) {
	989	+	git_ref
	990	+	} else {
	991	+	match git.log(&git_ref, 1).await?.into_iter().next() {
	992	+	Some(head) => head.hash,
	993	+	None => return Ok(not_found()),
	994	+	}
	995	+	};
	996	+	let url = blame::cache_url(&repo.id, &head, &a.file_path);
	997	+	if let Ok(Some(mut kept)) = worker::Cache::default().get(url.as_str(), false).await
	998	+	&& let Ok(found) = kept.json::<Blame>().await
	999	+	{
	1000	+	return Ok(Outcome::Ok(found));
	1001	+	}
	1002	+	let started = now_ms();
	1003	+	let out_of_time = move \|\| now_ms().saturating_sub(started) > BLAME_WALK_MS;
	1004	+	let Some(blamed) = blame::blame(&git, &head, &a.file_path, &out_of_time, blame::MAX_READS).await? else {
	1005	+	return Ok(not_found());
	1006	+	};
	1007	+	// Kept once the answer has gone.
	1008	+	if let Ok(mut response) = worker::Response::from_json(&blamed.blame) {
	1009	+	let _ = response.headers_mut().set("cache-control", blame::cache_control(&blamed));
	1010	+	self.deferred.spawn(async move {
	1011	+	let _ = worker::Cache::default().put(url.as_str(), response).await;
	1012	+	});
	1013	+	}
	1014	+	Ok(Outcome::Ok(blamed.blame))
985	1015		}
986	1016
987	1017		async fn log(&self, a: LogArgs) -> Result<Outcome<Vec<Commit>>> {
⋯
1035	1065		head.hash,
1036	1066		a.tree_path.split('/').map(urlencoding_segment).collect::<Vec<_>>().join("/")
1037	1067		);
1038		−	let cache = worker::Cache::default();
1039		−	if let Ok(Some(mut kept)) = cache.get(key.as_str(), false).await {
1040		−	if let Ok(found) = kept.json::<g1t_contracts::repos::LastCommits>().await {
1041		−	return Ok(Outcome::Ok(found));
1042		−	}
	1068	+	if let Ok(Some(mut kept)) = worker::Cache::default().get(key.as_str(), false).await
	1069	+	&& let Ok(found) = kept.json::<g1t_contracts::repos::LastCommits>().await
	1070	+	{
	1071	+	return Ok(Outcome::Ok(found));
1043	1072		}
1044	1073		let memo = last_commits::Memo::new(&repo.id, &git_ref, &a.tree_path);
1045		−	let shared = self.shared.as_deref();
1046		−	let progress = memo.get(shared).await;
	1074	+	let shared = self.shared.clone();
	1075	+	let progress = memo.get(shared.as_deref()).await;
1047	1076		// Asked with a budget, the walk stops past it; without one, past
1048	1077		// LAST_COMMITS_WALK_MS, well before the caller's waitUntil ends.
1049	1078		let started = worker::Date::now().as_millis();
1050	1079		let budget = a.budget_ms.unwrap_or(LAST_COMMITS_WALK_MS);
1051	1080		let out_of_time = move \|\| worker::Date::now().as_millis().saturating_sub(started) > budget;
1052	1081		let walk = last_commits::last_commits(&git, &head.hash, &a.tree_path, progress, &out_of_time, last_commits::MAX_READS).await?;
	1082	+	let settled = walk.progress.settled();
	1083	+	// What the walk found is kept once the answer has gone: the caller
	1084	+	// waits on neither write.
1053	1085		if walk.reads > 0 {
1054		−	memo.keep(shared, &walk.progress).await;
	1086	+	let progress = walk.progress.clone();
	1087	+	self.deferred.spawn(async move { memo.keep(shared.as_deref(), &progress).await });
1055	1088		}
1056		−	let settled = walk.progress.settled();
1057	1089		let found = g1t_contracts::repos::LastCommits { complete: walk.progress.complete(), entries: walk.progress.found };
1058		−	if !settled {
1059		−	return Ok(Outcome::Ok(found));
1060		−	}
1061		−	if let Ok(mut response) = worker::Response::from_json(&found) {
	1090	+	if settled && let Ok(mut response) = worker::Response::from_json(&found) {
1062	1091		let _ = response.headers_mut().set("cache-control", "max-age=604800");
1063		−	let _ = cache.put(key.as_str(), response).await;
	1092	+	self.deferred.spawn(async move {
	1093	+	let _ = worker::Cache::default().put(key.as_str(), response).await;
	1094	+	});
1064	1095		}
1065	1096		Ok(Outcome::Ok(found))
1066	1097		}
⋯
2577	2608		announced
2578	2609		}
2579	2610
	2611	+	/// How long after a push's answer its About may still be reading: the
	2612	+	/// waitUntil it runs in ends 30 s after the answer, and what it worked out
	2613	+	/// must still be written.
	2614	+	const PUSH_ABOUT_BY_MS: u64 = 24_000;
	2615	+	/// Less time left than this after recording a push, and its About is left
	2616	+	/// for the next view to start.
	2617	+	const MIN_PUSH_ABOUT_MS: u64 = 5_000;
	2618	+
	2619	+	/// How long a push's About may read, `elapsed` after the push's answer:
	2620	+	/// what is left before [`PUSH_ABOUT_BY_MS`], or `None` when that is too
	2621	+	/// little to be worth starting.
	2622	+	fn push_about_budget(elapsed: u64) -> Option<u64> {
	2623	+	let left = PUSH_ABOUT_BY_MS.saturating_sub(elapsed);
	2624	+	(left >= MIN_PUSH_ABOUT_MS).then_some(left)
	2625	+	}
	2626	+
	2627	+	/// The repository whose About a push should work out: one that has an
	2628	+	/// About, when the push moved its default branch.
	2629	+	fn about_after_push(push: &PushDone) -> Option<Repo> {
	2630	+	let moved_default = push.pushed.iter().any(\|pushed\| pushed.branch() == Some(push.repo.default_branch.as_str()));
	2631	+	(stats::has_about(&push.repo) && moved_default).then(\|\| push.repo.clone())
	2632	+	}
	2633	+
2580	2634		/// A push the store accepted, to be recorded once git has its answer.
2581	2635		struct PushDone {
2582	2636		repo: Repo,
⋯
2618	2672		return;
2619	2673		}
2620	2674		let env = env.clone();
	2675	+	// A push that moved the default branch has its About worked out
	2676	+	// now, so the next view finds it current rather than starting the
	2677	+	// run itself (about.rs, `refresh`).
	2678	+	let about = self.push.as_ref().and_then(about_after_push);
	2679	+	let started = now_ms();
2621	2680		ctx.wait_until(async move {
2622	2681		let repos = match service(&env) {
2623	2682		Ok(repos) => repos,
⋯
2633	2692		worker::console_error!("push not recorded: {error}");
2634	2693		}
2635	2694		repos.deferred.settle().await;
	2695	+	if let Some(repo) = about
	2696	+	&& let Some(budget) = push_about_budget(now_ms().saturating_sub(started))
	2697	+	{
	2698	+	repos.refresh(&repo, budget).await;
	2699	+	repos.deferred.settle().await;
	2700	+	}
2636	2701		});
2637	2702		}
2638	2703		}
⋯
3306	3371		}
3307	3372
3308	3373		#[cfg(test)]
	3374	+	mod push_about_tests {
	3375	+	use super::*;
	3376	+
	3377	+	fn push(refs: &[&str], fork_of: Option<&str>) -> PushDone {
	3378	+	let repo: Repo = serde_json::from_value(serde_json::json!({
	3379	+	"id": "rep_1", "namespace": "acme", "name": "web", "description": null, "isPrivate": false,
	3380	+	"ownerId": "usr_1", "defaultBranch": "main", "forkOf": fork_of, "createdAt": ""
	3381	+	}))
	3382	+	.unwrap();
	3383	+	PushDone {
	3384	+	repo,
	3385	+	pushed: refs.iter().map(\|git_ref\| git_http::Pushed { git_ref: (*git_ref).into(), before: None, after: "abc".into() }).collect(),
	3386	+	deleted: Vec::new(),
	3387	+	pack_bytes: 0,
	3388	+	actor: None,
	3389	+	unscanned: false,
	3390	+	caused_by_job: None,
	3391	+	credit: None,
	3392	+	}
	3393	+	}
	3394	+
	3395	+	#[test]
	3396	+	fn a_push_to_the_default_branch_works_out_the_about() {
	3397	+	assert!(about_after_push(&push(&["refs/heads/main"], None)).is_some());
	3398	+	assert!(about_after_push(&push(&["refs/heads/topic", "refs/heads/main"], None)).is_some());
	3399	+	assert!(about_after_push(&push(&["refs/heads/topic"], None)).is_none());
	3400	+	assert!(about_after_push(&push(&["refs/tags/main"], None)).is_none());
	3401	+	// A pull request's working copy has no About.
	3402	+	assert!(about_after_push(&push(&["refs/heads/main"], Some("rep_0"))).is_none());
	3403	+	}
	3404	+
	3405	+	#[test]
	3406	+	fn a_push_about_reads_only_within_what_is_left() {
	3407	+	assert_eq!(push_about_budget(0), Some(PUSH_ABOUT_BY_MS));
	3408	+	assert_eq!(push_about_budget(4_000), Some(PUSH_ABOUT_BY_MS - 4_000));
	3409	+	assert_eq!(push_about_budget(PUSH_ABOUT_BY_MS - MIN_PUSH_ABOUT_MS), Some(MIN_PUSH_ABOUT_MS));
	3410	+	assert_eq!(push_about_budget(PUSH_ABOUT_BY_MS - MIN_PUSH_ABOUT_MS + 1), None);
	3411	+	assert_eq!(push_about_budget(60_000), None);
	3412	+	}
	3413	+	}
	3414	+
	3415	+	#[cfg(test)]
3309	3416		mod push_to_create_tests {
3310	3417		use super::*;
3311	3418

639	639		</Link>
640	640		<Link
641	641		to={`${fileBase}/${file.path}?blame=1`}
	642	+	rel="nofollow"
642	643		className="rounded px-1.5 py-0.5 text-muted hover:bg-raised hover:text-fg"
643	644		>
644	645		Blame

1	+	import assert from "node:assert/strict";
2	+	import { test } from "node:test";
3	+
4	+	import { robotsTxt } from "./robots.ts";
5	+
6	+	test("everything may be crawled but a file's blame", () => {
7	+	const text = robotsTxt("https://g1t.example");
8	+	assert.match(text, /^User-agent: \*\nAllow: \/\n/);
9	+	assert.match(text, /^Disallow: \/\*\?blame=$/m);
10	+	assert.match(text, /^Disallow: \/\*&blame=$/m);
11	+	assert.match(text, /^Sitemap: https:\/\/g1t\.example\/sitemap\.xml$/m);
12	+	assert.doesNotMatch(text, /docs\.g1t\.sh/, "a self-hosted g1t has no docs site");
13	+	assert.ok(text.endsWith("\n"));
14	+	});
15	+
16	+	test("g1t.sh names its docs' map too", () => {
17	+	assert.match(robotsTxt("https://g1t.sh"), /^Sitemap: https:\/\/docs\.g1t\.sh\/sitemap-index\.xml$/m);
18	+	});

1	+	/**
2	+	* What /robots.txt says: everything may be crawled but a file's blame
3	+	* (`?blame=1`), and where the site's maps are.
4	+	*
5	+	* Blame is the costliest file view to work out, one per file per commit a
6	+	* crawler finds, and adds nothing an index needs that the file itself does
7	+	* not have: on 2026-10-14 crawlers asked for 1,150 blame pages in four
8	+	* hours, against a handful from people. The links to it are `nofollow` too,
9	+	* and its page is `noindex` (routes/repo/blob.tsx).
10	+	*/
11	+
12	+	/** g1t.sh's docs keep their own map; a self-hosted g1t has no docs site. */
13	+	const DOCS_SITEMAP = "https://docs.g1t.sh/sitemap-index.xml";
14	+
15	+	/** Paths crawlers are kept from, wherever the query puts `blame`. */
16	+	export const DISALLOWED = ["/?blame=", "/&blame="];
17	+
18	+	export function robotsTxt(site: string): string {
19	+	const lines = ["User-agent: *", "Allow: /", ...DISALLOWED.map((path) => `Disallow: ${path}`), "", `Sitemap: ${site}/sitemap.xml`];
20	+	if (site === "https://g1t.sh") lines.push(`Sitemap: ${DOCS_SITEMAP}`);
21	+	return `${lines.join("\n")}\n`;
22	+	}

793	793		pub ranges: Vec<BlameRange>,
794	794		/// The commits the ranges name, each once.
795	795		pub commits: Vec<Commit>,
796		−	/// True when the history was too long to read in full, so the oldest
797		−	/// lines are given to the oldest commit read.
	796	+	/// True when the history was too long to read in full (too many
	797	+	/// commits, or too long to read), so the oldest lines are given to the
	798	+	/// oldest commit read that has them.
798	799		pub partial: bool,
799	800		}
800	801

477	477
478	478		Each Worker keeps a share of its invocations' logs, set by
479	479		`head_sampling_rate` in its `wrangler.jsonc`: every one for billing,
480		−	identity, runner, actions, deployments, status and sudo, and a tenth for
481		−	the rest. Repos and work keep every one for now, while rare ~60 s stalls
482		−	are traced, and go back to a tenth once they are found. Metrics are not sampled. To chase a rare error on a sampled
483		−	Worker, raise its rate to 1 while you look.
	480	+	identity, runner, actions, deployments, status and sudo, a quarter for
	481	+	repos and work, which every page asks, and a tenth for the rest. Metrics
	482	+	are not sampled. To chase a rare error on a sampled Worker, raise its rate
	483	+	to 1 while you look.
	484	+
	485	+	Slow work in repos logs itself as fields, so the invocations kept can be
	486	+	counted by what was slow: an RPC answer that took a second or more logs
	487	+	`rpc` (the method) and `answered_ms`, and a slow or partial About run logs
	488	+	`about_run` (the repository's id), `took_ms` and `partial`. In Workers
	489	+	Logs, filter `g1t-repos` on `answered_ms` greater than 0, and multiply
	490	+	what you count by four.

33	33
34	34		The license, security policy, languages and contributors are read from the
35	35		default branch in the background each time it moves, and kept by commit, so
36		−	the page never waits for them. A repository pushed to for the first time
37		−	shows Reading the default branch… for a few seconds.
	36	+	the page never waits for them. A push to the default branch starts the
	37	+	reading as soon as git has its answer; a branch that moved otherwise, such
	38	+	as by merging a pull request, is read again when its page is next opened.
	39	+	A repository pushed to for the first time shows **Reading the default
	40	+	branch…** for a few seconds.
38	41
39	42		### Languages
40	43

94	94		request when it caught up with `main` is credited to whoever wrote it on
95	95		`main`, not to the merge.
96	96
	97	+	Blame reads up to 400 commits back, for up to six seconds. A line older
	98	+	than that is credited to the oldest commit read that has it, and the view
	99	+	says the history is long. A file's blame at a commit is kept once worked
	100	+	out, so opening it again, or from a commit's link, shows at once. Search
	101	+	engines and other crawlers are asked to leave blame pages alone
	102	+	(`Disallow: /*?blame=` in `/robots.txt`); the file itself stays open to
	103	+	them.
	104	+
97	105		### Why over the API
98	106
99	107		The `repository` tool's `why` action, or

460	460		const toggle = (label: string, on: boolean, search: string) => (
461	461		<Link
462	462		to={{ search }}
	463	+	// Blame is costly to work out and adds nothing to a search index.
	464	+	rel={search ? "nofollow" : undefined}
463	465		preventScrollReset
464	466		className={`rounded px-2.5 py-1.5 transition-colors sm:px-2 sm:py-0.5 ${on ? "bg-raised text-fg" : "hover:text-fg"}`}
465	467		>

11	11		import { getViewer, unwrap } from "../../lib/session.server";
12	12
13	13		export function meta({ params, ...args }: Route.MetaArgs) {
14		−	return page(args, { title: `${params["*"]} · ${params.owner}/${params.repo} · g1t` });
	14	+	const tags = page(args, { title: `${params["*"]} · ${params.owner}/${params.repo} · g1t` });
	15	+	// A file's blame is kept out of search indexes; the file itself is not (lib/robots.ts).
	16	+	const blame = new URLSearchParams(args.location.search).has("blame");
	17	+	return blame ? [...tags, { name: "robots", content: "noindex, nofollow" }] : tags;
15	18		}
16	19
17	20		export async function loader({ params, context, request }: Route.LoaderArgs) {

1	1		/**
2		−	* /robots.txt: everything may be crawled, and where the site's maps are.
3		−	* Served from the request's own origin, so a self-hosted g1t names itself.
	2	+	* /robots.txt (lib/robots.ts). Served from the request's own origin, so a
	3	+	* self-hosted g1t names itself.
4	4		*/
5	5		import type { Route } from "./+types/robots-txt";
6		−
7		−	/** g1t.sh's docs keep their own map; a self-hosted g1t has no docs site. */
8		−	const DOCS_SITEMAP = "https://docs.g1t.sh/sitemap-index.xml";
	6	+	import { robotsTxt } from "../lib/robots";
9	7
10	8		export function loader({ request }: Route.LoaderArgs) {
11		−	const site = new URL(request.url).origin;
12		−	const lines = ["User-agent: *", "Allow: /", "", `Sitemap: ${site}/sitemap.xml`];
13		−	if (site === "https://g1t.sh") lines.push(`Sitemap: ${DOCS_SITEMAP}`);
14		−	return new Response(`${lines.join("\n")}\n`, {
	9	+	return new Response(robotsTxt(new URL(request.url).origin), {
15	10		headers: {
16	11		"content-type": "text/plain; charset=utf-8",
17	12		"cache-control": "public, max-age=86400",

477	477		ranges: BlameRange[];
478	478		/** The commits the ranges name, each once. */
479	479		commits: Commit[];
480		−	/** True when the history was too long to read in full. */
	480	+	/** True when the history was too long to read in full (too many commits, or too long to read). */
481	481		partial: boolean;
482	482		};
483	483

130	130		self.inner.read_blob(blob_hash).await
131	131		}
132	132
	133	+	/// The store's own, which keeps sizes by hash without reading bytes.
	134	+	async fn blob_size(&self, blob_hash: &str) -> Result<Option<u64>> {
	135	+	self.inner.blob_size(blob_hash).await
	136	+	}
	137	+
133	138		async fn read_file(&self, git_ref: &str, path: &str) -> Result<Option<Vec<u8>>> {
134	139		self.inner.read_file(self.resolve(git_ref), path).await
135	140		}

29	29
30	30		/// A run older than this is taken to have died.
31	31		pub const LEASE_MS: u64 = 120_000;
32		−	/// How long a run reads before keeping what it has.
33		−	const BUDGET_MS: u64 = 22_000;
	32	+	/// How long a run reads before keeping what it has, at most.
	33	+	pub const BUDGET_MS: u64 = 22_000;
34	34		/// The most commits of history read for contributors.
35	35		pub const MAX_HISTORY: usize = 3_000;
36		−	/// Commits read at a time.
37		−	const LOG_PAGE: u32 = 200;
	36	+	/// Commits read at a time. A history is kept by the commit it starts from
	37	+	/// (store.rs), and a page from a new head is the few commits since the last
	38	+	/// one, spliced onto the page kept from it; but each later page starts
	39	+	/// wherever the one before it ended, which moves with every push, and so is
	40	+	/// read from the store again. Pages as long as the store reads in one go
	41	+	/// (as `MAX_ANCESTRY` in lib.rs does) leave most repositories one page: 200
	42	+	/// made g1t's own history four, three of them read again after each push.
	43	+	const LOG_PAGE: u32 = 1_000;
38	44		/// The most blobs sized for languages.
39	45		const MAX_SIZED: usize = 6_000;
40	46		/// Blobs sized at once: each is a round trip to the store.
⋯
258	264		}
259	265
260	266		/// Works out the About of `repo` at `head` (a commit). `identity` matches
261		−	/// authors' addresses to accounts.
262		−	pub async fn work_out<R: GitRepo>(git: &R, head: &str, tree: &str, identity: Option<&Fetcher>) -> Result<Worked> {
	267	+	/// authors' addresses to accounts. Reads stop past `budget_ms` (at most
	268	+	/// [`BUDGET_MS`]), leaving the answer partial.
	269	+	pub async fn work_out<R: GitRepo>(git: &R, head: &str, tree: &str, identity: Option<&Fetcher>, budget_ms: u64) -> Result<Worked> {
263	270		let started = now_ms();
264		−	let out_of_time = move \|\| now_ms().saturating_sub(started) > BUDGET_MS;
	271	+	let budget_ms = budget_ms.min(BUDGET_MS);
	272	+	let out_of_time = move \|\| now_ms().saturating_sub(started) > budget_ms;
265	273		let never: Vec<String> = NEVER_READ.iter().map(\|dir\| (*dir).to_owned()).collect();
266	274		let files = async {
267	275		let (files, truncated) = crate::listing::changed(git, None, tree, &never, MAX_LISTED_FILES).await?;

136	136		{ "name": "PACK_FILL_LIMIT", "namespace_id": "4301", "simple": { "limit": 30, "period": 60 } },
137	137		{ "name": "ANONYMOUS_FETCH_LIMIT", "namespace_id": "4302", "simple": { "limit": 120, "period": 60 } }
138	138		],
139		−	// Logs of every request for now, to trace the rare ~60 s stalls a tenth
140		−	// of requests kept missing. Put it back to 0.1 once they are found:
141		−	// every git request and page is one.
142		−	"observability": { "enabled": true, "head_sampling_rate": 1 }
	139	+	// Logs of a quarter of requests: every git request and page is one, so
	140	+	// keeping every one costs more than it tells, and a quarter still
	141	+	// catches a rare error within hours. Slow answers still show: an RPC
	142	+	// answer that took a second or more logs `{"rpc", "answered_ms"}`
	143	+	// (src/lib.rs, SLOW_RPC_MS) and a slow or partial About run
	144	+	// `{"about_run", "took_ms"}` (src/about.rs), so in the quarter kept they
	145	+	// can be counted by method without reading request lines.
	146	+	"observability": { "enabled": true, "head_sampling_rate": 0.25 }
143	147		}

26	26		"queues": {
27	27		"consumers": [{ "queue": "g1t-events-work", "max_batch_size": 100, "max_batch_timeout": 1, "max_retries": 3, "dead_letter_queue": "g1t-events-dlq" }]
28	28		},
29		−	// Logs of every request for now, to trace the rare ~60 s stalls a tenth
30		−	// of requests kept missing. Put it back to 0.1 once they are found:
31		−	// every issue and pull request page asks it.
32		−	"observability": { "enabled": true, "head_sampling_rate": 1 }
	29	+	// Logs of a quarter of requests: every issue and pull request page asks
	30	+	// it, so keeping every one costs more than it tells, and a quarter
	31	+	// still catches a rare error within hours. Repos, which it asks most,
	32	+	// logs its slow answers as fields (`answered_ms`, services/repos/src/lib.rs).
	33	+	"observability": { "enabled": true, "head_sampling_rate": 0.25 }
33	34		}

flagon-io/g1t

Comments