Skip to content
Sign up

flagon-io/g1t

public
0
366 linesCodeBlameRaw

Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.

Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.1//! Brokered secrets: a secret a job uses only to call an API never enters
2//! its sandbox. Its row names the hosts it is for (settings.rs); when a job
3//! starts, `job_spec` gives the job a stand-in in its place, one per job and
4//! secret, and keeps here what the stand-in stands for, sealed and bound to
5//! it, until the job's time is up. The runner Worker sees every HTTP and
6//! HTTPS request a job on g1t's machines makes (services/runner broker.ts),
7//! asks `brokered_secrets` with the job's token for its stand-ins, and puts
8//! the real value in place of a stand-in on requests to the secret's hosts
9//! as they leave. Anywhere else the stand-in goes as it is, which is
10//! useless to whoever receives it.
11//!
Merge branch 'main' into rm-agents12//! A self-hosted runner's traffic never passes the Worker, and the real
13//! value is never sent to a machine g1t does not run. A harness that can
14//! broker (on Linux or macOS, `JobCallArgs::broker`) is given each
15//! stand-in's hosts in its spec (`brokered`), never a value: its job's
16//! HTTPS to those hosts goes through a proxy of the harness's, which
17//! passes each request carrying a stand-in to g1t
18//! (`POST /actions/jobs/{job}/broker`), where the runner Worker fills it
19//! in (`relayed_secrets`), sends it and scrubs the answer. Any other
20//! harness's job keeps the stand-in, and its log says why it will not
21//! work.
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.22
23use g1t_contracts::actions::{BrokeredSecret, JobCallArgs, STAND_IN_PREFIX};
24use g1t_contracts::time::rfc3339;
Merge branch 'main' into rm-agents25use g1t_contracts::{FailureCode, Outcome};
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.26use g1t_kit::now_ms;
27use g1t_secrets::random_hex;
28use serde::Deserialize;
29use serde_json::{Map, Value};
30use worker::Result;
31use worker::wasm_bindgen::JsValue;
32
33use crate::plan::LOG_INSERT;
Merge branch 'main' into rm-agents34use crate::{Actions, Count, check, fail};
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.35
36/// The most hosts one brokered secret names.
37pub(crate) const MAX_HOSTS: usize = 20;
38/// The shortest value a brokered secret may have: a shorter one would be
39/// found, and swapped, inside unrelated secrets.
40pub(crate) const MIN_VALUE_CHARS: usize = 8;
41
42/// Keeps a job's stand-in: what it stands for, sealed and bound to it.
43pub(crate) const STAND_IN_INSERT: &str =
44 "INSERT OR REPLACE INTO brokered_secrets (stand_in, job_id, name, value, hosts, expires_at) VALUES (?1, ?2, ?3, ?4, ?5, ?6)";
45/// A job's stand-ins that still work at `?2`.
46pub(crate) const STAND_INS_OF_JOB: &str =
47 "SELECT stand_in, name, value, hosts FROM brokered_secrets WHERE job_id = ?1 AND expires_at > ?2";
48/// Stand-ins whose jobs' time is up.
49pub(crate) const STAND_INS_EXPIRED: &str = "DELETE FROM brokered_secrets WHERE expires_at <= ?1";
50
51/// A brokered secret as one reader resolved it: its key, real value and
52/// the hosts it is for.
53#[derive(Clone, Debug, PartialEq)]
54pub(crate) struct Broker {
55 pub name: String,
56 pub value: String,
57 pub hosts: Vec<String>,
58}
59
60/// A stand-in made for one job.
61#[derive(Clone, Debug, PartialEq)]
62pub(crate) struct StandIn {
63 pub stand_in: String,
64 pub broker: Broker,
65}
66
67/// A new stand-in: the prefix and 48 random hex characters.
68pub(crate) fn stand_in() -> String {
69 format!("{STAND_IN_PREFIX}{}", random_hex(24))
70}
71
72/// Puts a stand-in in place of each brokered value wherever it appears in
73/// `secrets`, as a whole value or inside one a called workflow built from
74/// it (`${{ secrets.USER }}:${{ secrets.TOKEN }}`), and returns the
75/// stand-ins made: one per brokered value found, whatever keys it is under.
76/// Longer values go first, so one that holds another is replaced whole.
77pub(crate) fn substitute(secrets: &mut Map<String, Value>, brokers: &[Broker], mut make: impl FnMut() -> String) -> Vec<StandIn> {
78 let mut ordered: Vec<&Broker> = brokers.iter().filter(|b| b.value.chars().count() >= MIN_VALUE_CHARS).collect();
79 ordered.sort_by_key(|b| std::cmp::Reverse(b.value.len()));
80 let mut made: Vec<StandIn> = Vec::new();
81 for broker in ordered {
82 if made.iter().any(|m| m.broker.value == broker.value) {
83 continue;
84 }
85 let mut stand_in: Option<String> = None;
86 for value in secrets.values_mut() {
87 let Value::String(text) = value else { continue };
88 if text.contains(&broker.value) {
89 let with = stand_in.get_or_insert_with(&mut make);
90 *text = text.replace(&broker.value, with);
91 }
92 }
93 if let Some(stand_in) = stand_in {
94 made.push(StandIn { stand_in, broker: broker.clone() });
95 }
96 }
97 made
98}
99
Merge branch 'main' into rm-agents100/// What a job on a self-hosted runner whose harness cannot broker (a
101/// Windows one, or one from before brokering) is told for each brokered
102/// secret it was given.
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.103pub(crate) fn self_hosted_note(name: &str) -> String {
104 format!(
Merge branch 'main' into rm-agents105 "##[warning]{name} is a brokered secret: this job has a stand-in for it, which this runner cannot have filled in (that takes an up-to-date runner whose jobs run on Linux or macOS). Requests that use it will not be signed in."
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.106 )
107}
108
Merge branch 'main' into rm-agents109/// What a job on a self-hosted runner that brokers is told for each
110/// brokered secret it was given.
111pub(crate) fn brokered_note(name: &str, hosts: &[String]) -> String {
112 format!(
113 "{name} is a brokered secret: this job's requests to {} go through g1t, which fills it in, so its value never reaches this computer.",
114 hosts.join(", ")
115 )
116}
117
118/// What a brokering harness is given for its job's stand-ins: each one,
119/// its secret's key (for the log) and its hosts. Never a value.
120pub(crate) fn spec_entries(stand_ins: &[StandIn]) -> Value {
121 Value::Array(
122 stand_ins
123 .iter()
124 .map(|made| serde_json::json!({ "standIn": made.stand_in, "name": made.broker.name, "hosts": made.broker.hosts }))
125 .collect(),
126 )
127}
128
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.129/// A line of a job's log, stamped as the runner stamps its own.
130pub(crate) fn stamped(text: &str) -> String {
131 let at = rfc3339(now_ms());
132 text.lines().map(|line| format!("{at} {line}\n")).collect()
133}
134
135#[derive(Deserialize)]
136struct StandInRow {
137 stand_in: String,
138 name: String,
139 value: String,
140 hosts: String,
141}
142
143impl Actions {
144 /// Keeps a job's stand-ins until its time is up, and marks the job so
145 /// its log is masked for their values. Stand-ins whose time is up go.
146 pub(crate) async fn keep_stand_ins(&self, job_id: &str, timeout_minutes: u32, stand_ins: &[StandIn]) -> Result<()> {
147 let Some(sealer) = &self.sealer else { return Ok(()) };
148 let now = now_ms();
149 let expires = now + (u64::from(timeout_minutes) * 60 + 600) * 1000;
150 let mut statements = vec![self.db.prepare(STAND_INS_EXPIRED).bind(&[(now as f64).into()])?];
151 for made in stand_ins {
152 statements.push(self.db.prepare(STAND_IN_INSERT).bind(&[
153 made.stand_in.as_str().into(),
154 job_id.into(),
155 made.broker.name.as_str().into(),
156 sealer.seal(&made.broker.value, &made.stand_in).into(),
157 serde_json::to_string(&made.broker.hosts)?.into(),
158 (expires as f64).into(),
159 ])?);
160 }
161 statements.push(self.db.prepare("UPDATE jobs SET brokered = 1 WHERE id = ?").bind(&[job_id.into()])?);
162 self.db.batch(statements).await?;
163 Ok(())
164 }
165
166 /// A job's stand-ins that still work, opened.
167 pub(crate) async fn stand_ins_of(&self, job_id: &str) -> Result<Vec<BrokeredSecret>> {
168 let Some(sealer) = &self.sealer else { return Ok(Vec::new()) };
169 let rows = self
170 .db
171 .prepare(STAND_INS_OF_JOB)
172 .bind(&[job_id.into(), (now_ms() as f64).into()])?
173 .all()
174 .await?
175 .results::<StandInRow>()?;
176 Ok(rows
177 .into_iter()
178 .filter_map(|row| {
179 Some(BrokeredSecret {
180 value: sealer.open(&row.value, &row.stand_in)?,
181 hosts: serde_json::from_str(&row.hosts).unwrap_or_default(),
182 name: row.name,
183 stand_in: row.stand_in,
184 })
185 })
186 .collect())
187 }
188
189 /// Whether a job in the repository could be given a brokered secret:
190 /// its project or its workspace has one. Asked as a job starts, so the
191 /// runner sends the job's HTTP through the Worker even when its network
192 /// is open; a job that turns out to get none just goes the long way.
193 pub(crate) async fn may_broker(&self, repo_id: &str, namespace: &str) -> Result<bool> {
194 let Some(project) = self.project_of(repo_id).await? else { return Ok(false) };
195 let found = self
196 .db
197 .prepare("SELECT COUNT(*) AS n FROM settings WHERE kind = 'secret' AND brokered_hosts IS NOT NULL AND owner IN (?, ?)")
198 .bind(&[project.id.as_str().into(), namespace.to_lowercase().into()])?
199 .first::<Count>(None)
200 .await?;
201 Ok(found.is_some_and(|count| count.n > 0))
202 }
203
204 /// Adds g1t's own lines to a job's log, under `step` (0 for its setup).
205 pub(crate) async fn note_log(&self, job_id: &str, step: u32, text: &str) -> Result<()> {
206 self.db
207 .prepare(LOG_INSERT)
208 .bind(&[job_id.into(), step.into(), stamped(text).into(), JsValue::NULL])?
209 .run()
210 .await?;
211 Ok(())
212 }
213
214 /// `brokered_secrets`: a running job's stand-ins and what they stand
215 /// for, for the runner Worker to fill in on the job's requests. Asked
216 /// with the job's own token, which only the runner and the job hold;
Merge main into rm-agents: mirroring removal, open signups, free-tier limits, the g1t CLI and runner fixes, with the agent removal kept. The CLI's table of operations is regenerated from the API without the agent, customize and AI credit operations; no CLI command used them.217 /// the API never passes this method on, so the job cannot ask it. The
218 /// Worker holds the token the job started with, so that one still
219 /// answers here after the job's harness swapped it for another
220 /// (`job_spec` with `rotate`), and nowhere else.
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.221 pub async fn brokered_secrets(&self, a: JobCallArgs) -> Result<Outcome<Vec<BrokeredSecret>>> {
Merge main into rm-agents: mirroring removal, open signups, free-tier limits, the g1t CLI and runner fixes, with the agent removal kept. The CLI's table of operations is regenerated from the API without the agent, customize and AI credit operations; no CLI command used them.222 let job = check!(self.job_for_started_token(&a).await?);
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.223 Ok(Outcome::Ok(self.stand_ins_of(&job.id).await?))
224 }
Merge branch 'main' into rm-agents225
226 /// `relayed_secrets`: a running self-hosted job's stand-ins and what
227 /// they stand for, for the runner Worker to fill in on a request the
228 /// job's harness passed to g1t (`POST /actions/jobs/{job}/broker`).
229 /// Only with the job's current token, which its harness holds: the one
230 /// it started with may have been left on the runner's computer. A job
231 /// on g1t's machines has its requests filled in as they leave its
232 /// sandbox, and is refused here.
233 pub async fn relayed_secrets(&self, a: JobCallArgs) -> Result<Outcome<Vec<BrokeredSecret>>> {
234 let job = check!(self.job_for_token(&a).await?);
235 if job.runner_id.is_none() {
236 return Ok(fail(FailureCode::Forbidden, "Only a self-hosted runner's job relays its requests."));
237 }
238 Ok(Outcome::Ok(self.stand_ins_of(&job.id).await?))
239 }
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.240}
241
242#[cfg(test)]
243mod tests {
244 use rusqlite::{Connection, params};
245 use serde_json::{Map, Value, json};
246
Merge branch 'main' into rm-agents247 use super::{Broker, STAND_IN_INSERT, STAND_INS_EXPIRED, STAND_INS_OF_JOB, StandIn, brokered_note, self_hosted_note, spec_entries, stand_in, substitute};
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.248
249 fn broker(name: &str, value: &str, hosts: &[&str]) -> Broker {
250 Broker { name: name.into(), value: value.into(), hosts: hosts.iter().map(|h| (*h).to_owned()).collect() }
251 }
252
253 fn counter() -> impl FnMut() -> String {
254 let mut n = 0;
255 move || {
256 n += 1;
257 format!("g1tsec_{n}")
258 }
259 }
260
261 #[test]
262 fn stand_ins_are_long_random_tokens() {
263 let one = stand_in();
264 assert!(one.starts_with("g1tsec_"));
265 assert_eq!(one.len(), "g1tsec_".len() + 48);
266 assert!(one["g1tsec_".len()..].chars().all(|c| c.is_ascii_hexdigit()));
267 assert_ne!(one, stand_in());
268 }
269
270 #[test]
271 fn a_brokered_value_never_reaches_the_job() {
272 let mut secrets: Map<String, Value> = json!({
273 "NPM_TOKEN": "npm_realvalue123",
274 "ALSO_NPM": "npm_realvalue123",
275 "BASIC": "me:npm_realvalue123",
276 "PLAIN": "not-brokered-value",
277 })
278 .as_object()
279 .cloned()
280 .unwrap();
281 let made = substitute(&mut secrets, &[broker("NPM_TOKEN", "npm_realvalue123", &["registry.npmjs.org"])], counter());
282 assert_eq!(made.len(), 1);
283 assert_eq!(made[0].stand_in, "g1tsec_1");
284 assert_eq!(made[0].broker.hosts, ["registry.npmjs.org"]);
285 assert_eq!(secrets["NPM_TOKEN"], "g1tsec_1");
286 assert_eq!(secrets["ALSO_NPM"], "g1tsec_1");
287 assert_eq!(secrets["BASIC"], "me:g1tsec_1");
288 assert_eq!(secrets["PLAIN"], "not-brokered-value");
289 assert!(!serde_json::to_string(&secrets).unwrap().contains("npm_realvalue123"));
290 }
291
292 #[test]
293 fn each_secret_gets_its_own_stand_in_and_longer_values_go_first() {
294 let mut secrets: Map<String, Value> =
295 json!({ "SHORT": "abcdefgh", "LONG": "abcdefgh-and-more" }).as_object().cloned().unwrap();
296 let made = substitute(
297 &mut secrets,
298 &[broker("SHORT", "abcdefgh", &["a.example.com"]), broker("LONG", "abcdefgh-and-more", &["b.example.com"])],
299 counter(),
300 );
301 assert_eq!(made.len(), 2);
302 assert_eq!(made[0].broker.name, "LONG");
303 assert_eq!(secrets["LONG"], "g1tsec_1");
304 assert_eq!(secrets["SHORT"], "g1tsec_2");
305 }
306
307 #[test]
308 fn values_too_short_or_not_given_make_no_stand_in() {
309 let mut secrets: Map<String, Value> = json!({ "PIN": "1234" }).as_object().cloned().unwrap();
310 assert!(substitute(&mut secrets, &[broker("PIN", "1234", &["x.example.com"])], counter()).is_empty());
311 assert!(substitute(&mut secrets, &[broker("GONE", "absent-value", &["x.example.com"])], counter()).is_empty());
312 assert_eq!(secrets["PIN"], "1234");
313 }
314
315 #[test]
316 fn the_self_hosted_note_is_a_warning_naming_the_secret() {
317 let note = self_hosted_note("NPM_TOKEN");
318 assert!(note.starts_with("##[warning]NPM_TOKEN"));
Merge branch 'main' into rm-agents319 assert!(note.contains("cannot have filled in"));
320 }
321
322 #[test]
323 fn a_brokering_harness_gets_hosts_and_stand_ins_but_never_a_value() {
324 let made = vec![StandIn {
325 stand_in: "g1tsec_1".into(),
326 broker: broker("NPM_TOKEN", "npm_realvalue123", &["registry.npmjs.org", "*.npm.example.com"]),
327 }];
328 let entries = spec_entries(&made);
329 assert_eq!(entries, json!([{ "standIn": "g1tsec_1", "name": "NPM_TOKEN", "hosts": ["registry.npmjs.org", "*.npm.example.com"] }]));
330 assert!(!entries.to_string().contains("npm_realvalue123"));
331 let note = brokered_note("NPM_TOKEN", &made[0].broker.hosts);
332 assert!(!note.starts_with("##["), "not a warning: {note}");
333 assert!(note.contains("registry.npmjs.org, *.npm.example.com") && note.contains("never reaches this computer"));
Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset.334 }
335
336 fn database() -> Connection {
337 let db = Connection::open_in_memory().unwrap();
338 for name in ["0001_init.sql", "0019_brokered_secrets.sql"] {
339 let path = format!("{}/migrations/{name}", env!("CARGO_MANIFEST_DIR"));
340 db.execute_batch(&std::fs::read_to_string(path).unwrap()).unwrap();
341 }
342 db
343 }
344
345 #[test]
346 fn a_job_finds_only_its_own_stand_ins_while_they_work() {
347 let db = database();
348 let keep = |stand_in: &str, job: &str, expires: i64| {
349 db.execute(STAND_IN_INSERT, params![stand_in, job, "NPM_TOKEN", "sealed", "[\"registry.npmjs.org\"]", expires]).unwrap()
350 };
351 keep("g1tsec_a", "job_1", 2_000);
352 keep("g1tsec_b", "job_2", 2_000);
353 keep("g1tsec_c", "job_1", 500);
354 let of = |job: &str, now: i64| -> Vec<String> {
355 let mut rows = db.prepare(STAND_INS_OF_JOB).unwrap();
356 rows.query_map(params![job, now], |row| row.get::<_, String>(0)).unwrap().map(Result::unwrap).collect()
357 };
358 assert_eq!(of("job_1", 1_000), ["g1tsec_a"]);
359 assert_eq!(of("job_2", 1_000), ["g1tsec_b"]);
360 assert!(of("job_1", 3_000).is_empty());
361 assert_eq!(db.execute(STAND_INS_EXPIRED, params![1_000]).unwrap(), 1);
362 // Settings rows and jobs carry the new columns.
363 db.execute("UPDATE settings SET brokered_hosts = '[]'", []).unwrap();
364 db.execute("UPDATE jobs SET brokered = 1", []).unwrap();
365 }
366}

This file's history is long; its oldest lines are credited to the oldest commit read.