Skip to content
Sign in

flagon-io/g1t

public
0

g1t/services/security/src/history.rs

230 lines11,517 bytesCodeBlame

Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.

Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API1//! Scanning a repository's history for secrets once, in the background, a
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily2//! page of commits at a time, metered to its workspace; and the new commits
3//! of a push too large to scan before it was stored, after it landed.
Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API4
5use g1t_contracts::billing::{CheckLimitArgs, Limit, LimitState, NotePendingArgs};
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily6use g1t_contracts::identity::NotifyOwnersArgs;
Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API7use g1t_contracts::security::{HistoryPage, ScanHistoryArgs, SecretStatus};
8use g1t_contracts::Outcome;
9use worker::Result;
10
11use crate::Security;
12use crate::store::RepoRow;
13
14/// Commits read per call to the repos service.
15const PAGE: u32 = 25;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily16/// Pages of a large push scanned as soon as it is heard of; the sweep
17/// continues the rest.
18pub const PUSH_PAGES_AT_ONCE: u32 = 4;
19/// A push's scan stops after this many pages (25 000 commits): beyond it,
20/// a rescan of the whole history is the way to look.
21const MAX_PUSH_PAGES: i64 = 1_000;
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put22// What scanning costs g1t, from Cloudflare's published prices on the
23// Workers Paid plan (October 2026), the same as billing's `scan_cpu` and
24// `scan_rows` meters:
25//
26// - Worker CPU time: $0.02 per million CPU milliseconds, so 0.02 millionths
27// of a dollar per millisecond.
28// - D1 rows written: $1.00 per million, so 1 millionth of a dollar a row.
29// Rows read ($0.001 per million) come to nothing measurable.
30// - Requests: the scan's calls between g1t's services go over service
31// bindings, which Cloudflare does not charge as requests.
32// - Artifacts: its operations are priced for create, push, pull and clone;
33// reading a git object through the binding is none of those.
34// - OSV, which dependency checks query, is free.
35//
36// So a scan costs the CPU it takes and the rows it writes. The CPU per
37// object read is an estimate: decoding the object and running every
38// secret pattern over it, generously rounded up. Billing charges the
39// total at cost plus its margin once the month is over.
40
41/// Worker CPU, in millionths of a dollar per millisecond.
42pub const MICROS_PER_CPU_MS: f64 = 0.02;
43/// One D1 row written, in millionths of a dollar.
44pub const MICROS_PER_ROW_WRITTEN: f64 = 1.0;
45/// CPU one git object read takes in a history scan, in milliseconds.
46pub const CPU_MS_PER_READ: f64 = 5.0;
47
48/// What a page of history scanning cost g1t, in millionths of a dollar,
49/// rounded up: the CPU of its reads, and the rows it writes (where the
50/// scan stands, the month's usage, and each secret found).
51pub fn history_page_cost(reads: u32, secrets: usize) -> i64 {
52 let cpu = f64::from(reads) * CPU_MS_PER_READ * MICROS_PER_CPU_MS;
53 let rows = (2 + secrets) as f64 * MICROS_PER_ROW_WRITTEN;
54 (cpu + rows).ceil() as i64
55}
Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API56
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily57/// What the email about secrets in a push that landed unscanned says.
58pub fn landed_secrets_intro(namespace: &str, name: &str, branch: &str, count: usize) -> String {
59 let what = if count == 1 { "a secret that looks real".to_owned() } else { format!("{count} secrets that look real") };
60 format!(
61 "A push to {branch} in {namespace}/{name} was too large to check before it was stored, so g1t scanned it after it landed and found {what}. \
62 Rotate each one with whoever issued it, then mark the alert revoked, or dismiss it if it is not a real secret."
63 )
64}
65
Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API66impl Security {
67 /// Whether the workspace's usage has reached its limit, which stops
68 /// background work. Unknown counts as not.
69 async fn over_limit(&self, workspace: &str) -> bool {
70 let limit: Result<Outcome<Limit>> =
71 g1t_kit::call(&self.billing, "check_limit", &CheckLimitArgs { workspace: workspace.to_owned() }).await;
72 matches!(limit, Ok(Outcome::Ok(limit)) if limit.state == LimitState::Stopped)
73 }
74
75 /// Records what scanning cost, and tells billing the month's total so
76 /// the workspace's limit counts it.
77 pub async fn meter(&self, workspace: &str, reads: u32, commits: u32, osv_calls: u32, cost_micros: i64) -> Result<()> {
78 let total = self.store.meter(workspace, reads, commits, osv_calls, cost_micros).await?;
79 let noted: Result<bool> = g1t_kit::call(
80 &self.billing,
81 "note_pending",
status.g1t.sh with incident management, invites that land you in the workspace, settings as pages, usage without quotas82 &NotePendingArgs { workspace: workspace.to_owned(), source: "security".to_owned(), cost_micros: total, detail: None },
Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API83 )
84 .await;
85 if let Err(error) = noted {
86 worker::console_error!("security: usage for {workspace} not noted: {error}");
87 }
88 Ok(())
89 }
90
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily91 /// Scans up to `pages` pages of the new commits of a push that reached
92 /// the store unscanned, from where its scan stopped. What it finds is
93 /// recorded open (it has landed), never blocking anything; a secret that
94 /// looks real is emailed to the workspace's owners once per push.
Teams and CODEOWNERS, labels and milestones, dependency updates, the security suite, and a clearer top bar95 /// The custom patterns a scan of `repo` looks for too.
96 async fn scan_patterns(&self, repo: &RepoRow) -> Vec<g1t_contracts::security_suite::PatternSpec> {
97 let args = g1t_contracts::security_suite::PatternsForArgs { repo_id: repo.repo_id.clone(), namespace: repo.namespace.clone(), private: None };
98 self.patterns_for(args).await.unwrap_or_else(|error| {
99 worker::console_error!("security: patterns for {}: {error}", repo.repo_id);
100 Vec::new()
101 })
102 }
103
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily104 pub async fn advance_push_scan(&self, id: &str, pages: u32) -> Result<()> {
105 let Some(scan) = self.store.push_scan(id).await? else {
106 return Ok(());
107 };
108 let Some(repo) = self.store.repo(&scan.repo_id).await? else {
109 return self.store.advance_push_scan(id, None, 0, 0).await;
110 };
111 if self.over_limit(&repo.namespace).await {
112 // Picked up again by the sweep once the workspace is under it.
113 return Ok(());
114 }
115 let mut cursor = scan.cursor.clone();
116 let mut real = 0usize;
Teams and CODEOWNERS, labels and milestones, dependency updates, the security suite, and a clearer top bar117 let patterns = self.scan_patterns(&repo).await;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily118 for pages_done in (scan.pages + 1)..=(scan.pages + i64::from(pages)) {
119 let page: HistoryPage = g1t_kit::call(
120 &self.repos,
121 "scan_history",
122 &ScanHistoryArgs {
123 repo_id: repo.repo_id.clone(),
124 after: cursor.clone(),
125 limit: PAGE,
126 from: Some(scan.head.clone()),
127 until: scan.base.clone(),
Teams and CODEOWNERS, labels and milestones, dependency updates, the security suite, and a clearer top bar128 patterns: patterns.clone(),
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily129 },
130 )
131 .await?;
132 let fingerprints: Vec<String> = page.secrets.iter().map(|secret| secret.fingerprint.clone()).collect();
133 let fresh: Vec<String> = {
134 let known = self.store.known(&repo.repo_id, &fingerprints).await?;
135 page.secrets
136 .iter()
137 .filter(|secret| secret.test_value.is_none())
138 .filter(|secret| !known.iter().any(|(fingerprint, _, _)| *fingerprint == secret.fingerprint))
139 .map(|secret| secret.fingerprint.clone())
140 .collect()
141 };
142 real += fresh.len();
143 self.store.landed(&repo.repo_id, &fingerprints).await?;
144 self.store
145 .add_secrets(&repo.repo_id, &page.secrets, SecretStatus::Open, "history", scan.pusher.as_deref())
146 .await?;
Teams and CODEOWNERS, labels and milestones, dependency updates, the security suite, and a clearer top bar147 self.secrets_found(&repo, &page.secrets, "history", &fresh, None).await?;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily148 let cost = history_page_cost(page.reads, page.secrets.len());
149 self.meter(&repo.namespace, page.reads, page.commits, 0, cost).await?;
150 let next = page.next.filter(|_| pages_done < MAX_PUSH_PAGES);
151 self.store
152 .advance_push_scan(id, next.as_deref(), page.commits, page.secrets.len() as u32)
153 .await?;
154 match next {
155 Some(next) => cursor = Some(next),
156 None => break,
157 }
158 }
159 if real > 0 {
160 self.tell_owners_of_landed_secrets(&repo, &scan.git_ref, real).await;
161 }
162 Ok(())
163 }
164
165 /// Emails a workspace's owners, who are Admins of every repository in
166 /// it, that a push which landed unscanned holds secrets that look real.
167 async fn tell_owners_of_landed_secrets(&self, repo: &RepoRow, git_ref: &str, count: usize) {
168 let branch = git_ref.strip_prefix("refs/heads/").unwrap_or(git_ref);
169 let args = NotifyOwnersArgs {
170 workspace: repo.namespace.clone(),
171 subject: format!("Secrets found in a push to {}/{}", repo.namespace, repo.name),
172 intro: landed_secrets_intro(&repo.namespace, &repo.name, branch, count),
173 action: "Review the alerts".to_owned(),
Teams and CODEOWNERS, labels and milestones, dependency updates, the security suite, and a clearer top bar174 link: format!("https://g1t.sh/{}/{}/security/secret-scanning", repo.namespace, repo.name),
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily175 footer: "You get this because you own this workspace on g1t. Very large pushes are scanned for secrets after they land: https://docs.g1t.sh/guides/security/".to_owned(),
176 };
177 if let Err(error) = g1t_kit::call::<_, u32>(&self.identity, "notify_owners", &args).await {
178 worker::console_error!("security: owners of {} not told of landed secrets: {error}", repo.namespace);
179 }
180 }
181
Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API182 /// Scans up to `pages` pages of a repository's history from where the
183 /// last scan stopped.
184 pub async fn advance_history(&self, repo: &RepoRow, pages: u32) -> Result<()> {
185 if repo.history == "done" {
186 return Ok(());
187 }
188 if self.over_limit(&repo.namespace).await {
189 return self.store.set_history(&repo.repo_id, "stopped", repo.history_cursor.as_deref(), 0).await;
190 }
191 let mut cursor = repo.history_cursor.clone();
Teams and CODEOWNERS, labels and milestones, dependency updates, the security suite, and a clearer top bar192 let patterns = self.scan_patterns(repo).await;
Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API193 for _ in 0..pages {
194 let page: HistoryPage = g1t_kit::call(
195 &self.repos,
196 "scan_history",
Teams and CODEOWNERS, labels and milestones, dependency updates, the security suite, and a clearer top bar197 &ScanHistoryArgs {
198 repo_id: repo.repo_id.clone(),
199 after: cursor.clone(),
200 limit: PAGE,
201 from: None,
202 until: None,
203 patterns: patterns.clone(),
204 },
Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API205 )
206 .await?;
207 let fingerprints: Vec<String> = page.secrets.iter().map(|secret| secret.fingerprint.clone()).collect();
Teams and CODEOWNERS, labels and milestones, dependency updates, the security suite, and a clearer top bar208 let known: Vec<String> = self.store.known(&repo.repo_id, &fingerprints).await?.into_iter().map(|(fingerprint, _, _)| fingerprint).collect();
209 let fresh: Vec<String> = page
210 .secrets
211 .iter()
212 .filter(|secret| secret.test_value.is_none() && !known.contains(&secret.fingerprint))
213 .map(|secret| secret.fingerprint.clone())
214 .collect();
Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API215 self.store.landed(&repo.repo_id, &fingerprints).await?;
216 self.store.add_secrets(&repo.repo_id, &page.secrets, SecretStatus::Open, "history", None).await?;
Teams and CODEOWNERS, labels and milestones, dependency updates, the security suite, and a clearer top bar217 self.secrets_found(repo, &page.secrets, "history", &fresh, None).await?;
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put218 let cost = history_page_cost(page.reads, page.secrets.len());
219 self.meter(&repo.namespace, page.reads, page.commits, 0, cost).await?;
Agents get guardrails, run credentials, an audit log, a context hub, repository instructions and mentions; security upkeep; snake_case API220 match page.next {
221 Some(next) => {
222 self.store.set_history(&repo.repo_id, "running", Some(&next), page.commits).await?;
223 cursor = Some(next);
224 }
225 None => return self.store.set_history(&repo.repo_id, "done", None, page.commits).await,
226 }
227 }
228 Ok(())
229 }
230}