Skip to content
Sign up

flagon-io/g1t

public
0

Commit

A public repository's job is no longer handed to a self-hosted runner in host mode, only to fail there. Runners say how they are isolated on every poll as well as when they register (an older g1t ignores it), so a runner restarted with or without Docker is known as it is now; the queue gives a host-mode runner no public repository's jobs or agent work, reading each repository's visibility once a poll, and a job that no online runner which may take it could, sandboxed and with its labels, group and repository, waits saying "Waiting for a sandboxed runner: this repository is public, and host-mode runners don't run public repositories' jobs." until one comes online. A runner that never said how it is isolated is given the job as before and refuses it itself. The Self-hosted runners guide says so, with the message in its troubleshooting table.

Claude-Session:
https://claude.ai/code/session_01PnwTRSNaC7vEHTAuBzjhrn
Claudeagent and Claudeagentcommitted Parent20b74aeBrowse files
6 files+208−150/6 viewed
+1−0
7070 args["version"] = json!(body["version"].as_str().unwrap_or_default());
7171 args["running"] = json!(body["running"].as_array().cloned().unwrap_or_default());
7272 args["wait_ms"] = json!(body["wait_ms"].as_u64().unwrap_or(0));
73+ args["isolation"] = json!(body["isolation"].as_str());
7374 "runner_poll"
7475 }
7576 "finished" => {
+15−5
144144 jobs directly on this machine (--no-docker), unsandboxed.*
145145 - Every job's log starts with a warning: *This job ran on the host,
146146 unsandboxed …*
147−- It refuses a public repository's work, which anyone can propose code
148− to: *acme/web is public, and this self-hosted runner runs jobs directly
149− on its machine (--no-docker), unsandboxed, so it does not take a public
150− repository's work.* The job fails with that message.
147+- It is not given a public repository's work, which anyone can propose
148+ code to. The runner says how it is isolated when it registers and on
149+ every poll, and g1t hands a public repository's jobs and agent work
150+ only to runners that are not in host mode. A job no other online runner
151+ could take waits, and its page says *Waiting for a sandboxed runner:
152+ this repository is public, and host-mode runners don't run public
153+ repositories' jobs.* It starts once a sandboxed runner with its labels
154+ comes online.
155+- A runner from before this, which does not say how it is isolated, may
156+ still be given such a job and refuses it itself: *acme/web is public,
157+ and this self-hosted runner runs jobs directly on its machine
158+ (--no-docker), unsandboxed, so it does not take a public repository's
159+ work.* The job fails with that message. Update the runner.
151160
152161 Move a Linux runner to [the container](#run-it-as-a-container). A macOS
153162 or Windows runner stays in host mode until sandboxes for them are built.
550559 | `run` says *g1t no longer knows this runner* | It was removed, or its credential was. Register it again. |
551560 | *this runner is not sandboxed from its machine, so it will not take work* | The runner's container can reach the machine: it lists why. Remove the machine's Docker socket and `--privileged`, and run it with `--runtime=sysbox-runc`. See [run it as a container](#run-it-as-a-container). |
552561 | *the runner's own Docker Engine could not start* | The container is not running under Sysbox. Add `--runtime=sysbox-runc`, and check that Sysbox is installed (`docker info` lists `sysbox-runc` under runtimes). |
553−| *… is public, and this self-hosted runner runs jobs directly on its machine* | A runner in host mode does not take public repositories' work. Run the runner's image under Sysbox. |
562+| The job says *Waiting for a sandboxed runner: this repository is public …* | Every online runner with the job's labels is in host mode, which never runs a public repository's jobs. Run a runner's image under Sysbox, or register one without `--no-docker`. |
563+| *… is public, and this self-hosted runner runs jobs directly on its machine* | An older runner in host mode refused a public repository's job. Update it, so g1t gives the job to a sandboxed runner instead, and run the runner's image under Sysbox. |
554564 | A step's `sudo` fails | Job containers have `no-new-privileges`. Steps run as the image's user, root in most CI images, so drop the `sudo`. |
555565 | *could not run docker* | Install Docker and start it, and give the runner's user access to it. |
556566 | Agent work fails at once | Give the runner `--agent-image`. |
+32−2
361361 /// How the runner keeps work from its machine: `nested-container` (a
362362 /// container per job, inside the runner's own sandboxed container),
363363 /// `container` (a container per job, on a Docker Engine it shares with
364− /// its machine) or `host` (directly on the machine). Kept, and shown
365− /// nowhere yet; older runners leave it out.
364+ /// its machine) or `host` (directly on the machine). A `host` runner is
365+ /// given no public repository's work ([`isolation_allows`]); older
366+ /// runners leave it out.
366367 #[serde(default)]
367368 pub isolation: Option<String>,
368369 }
397398 pub running: Vec<String>,
398399 #[serde(default)]
399400 pub wait_ms: u64,
401+ /// How the runner keeps work from its machine now, as at registration
402+ /// (`RegisterArgs::isolation`); kept up to date from each poll. Older
403+ /// runners leave it out.
404+ #[serde(default)]
405+ pub isolation: Option<String>,
400406 }
401407
402408 /// Work for a runner.
614620 }
615621 }
616622
623+/// What a public repository's job says while the only runners that could
624+/// take it run jobs directly on their machines (`host` isolation), which
625+/// are never given a public repository's work.
626+pub const WAITING_FOR_SANDBOX: &str =
627+ "Waiting for a sandboxed runner: this repository is public, and host-mode runners don't run public repositories' jobs.";
628+
629+/// Whether a runner of `isolation` may be given a repository's work, by
630+/// its visibility (`public` or `private`; unknown lets it through, and the
631+/// runner decides). A host-mode runner takes no public repository's work.
632+pub fn isolation_allows(isolation: Option<&str>, visibility: Option<&str>) -> bool {
633+ !(isolation == Some("host") && visibility == Some("public"))
634+}
635+
617636 /// What a job waiting for a runner says.
618637 pub fn waiting_reason(wanted: &Wanted) -> String {
619638 format!("Waiting for a self-hosted runner with labels {}.", wanted.describe())
628647 }
629648
630649 #[test]
650+ fn a_host_mode_runner_takes_no_public_repositorys_work() {
651+ assert!(!isolation_allows(Some("host"), Some("public")));
652+ assert!(isolation_allows(Some("host"), Some("private")));
653+ assert!(isolation_allows(Some("host"), None));
654+ assert!(isolation_allows(Some("container"), Some("public")));
655+ assert!(isolation_allows(None, Some("public")), "an older runner decides itself");
656+ let poll: PollArgs = serde_json::from_value(serde_json::json!({ "runner": "rnr_1", "credential": "c" })).unwrap();
657+ assert_eq!(poll.isolation, None, "an older runner's poll");
658+ }
659+
660+ #[test]
631661 fn a_runner_always_has_self_hosted_its_os_and_its_arch() {
632662 let labels = runner_labels(&strings(&["GPU", "gpu", " cuda-12 "]), "linux", "x64").unwrap();
633663 assert_eq!(labels, strings(&["self-hosted", "linux", "x64", "gpu", "cuda-12"]));
+22−2
113113 Ok(serde_json::from_value(answer)?)
114114 }
115115
116− pub fn poll(&self, running: &[String], wait_ms: u64) -> std::result::Result<Poll, Failure> {
117− let body = json!({ "version": VERSION, "running": running, "wait_ms": wait_ms });
116+ /// Asks for work. `isolation` is how this runner keeps work from its
117+ /// machine now, which g1t keeps up to date from each poll (it gives a
118+ /// host-mode runner no public repository's work); an older g1t ignores it.
119+ pub fn poll(&self, running: &[String], wait_ms: u64, isolation: &str) -> std::result::Result<Poll, Failure> {
120+ let body = poll_body(running, wait_ms, isolation);
118121 let answer = post(&format!("{}/runners/{}/poll", self.base, self.runner), Some(&self.credential), &body)?;
119122 serde_json::from_value(answer).map_err(|e| Failure::Unreachable(e.to_string()))
120123 }
149152 std::io::Read::read_to_end(&mut response.into_reader(), &mut bytes)?;
150153 Ok(bytes)
151154 }
155+
156+fn poll_body(running: &[String], wait_ms: u64, isolation: &str) -> Value {
157+ json!({ "version": VERSION, "running": running, "wait_ms": wait_ms, "isolation": isolation })
158+}
159+
160+#[cfg(test)]
161+mod tests {
162+ use super::*;
163+
164+ #[test]
165+ fn a_poll_says_how_the_runner_is_isolated() {
166+ let body = poll_body(&["job_1".to_owned()], 0, "host");
167+ assert_eq!(body["isolation"], "host");
168+ assert_eq!(body["running"], json!(["job_1"]));
169+ assert_eq!(body["version"], VERSION);
170+ }
171+}
+1−1
499499 }
500500 let running: Vec<String> = current.iter().map(|r| r.id.clone()).collect();
501501 let wait = if current.is_some() { 0 } else { IDLE_WAIT_MS };
502− let poll = match api.poll(&running, wait) {
502+ let poll = match api.poll(&running, wait, isolation.as_str()) {
503503 Ok(poll) => {
504504 backoff = Duration::from_secs(2);
505505 poll
+137−5
3232 use crate::plan::JobRow;
3333 use crate::{Actions, Count, SILENT_MS, check, fail, optional};
3434
35+/// What one poll has learned while it looks for work, so its looks again
36+/// do not ask again: each repository's visibility, and the jobs it already
37+/// said wait for a sandboxed runner.
38+#[derive(Default)]
39+struct Looked {
40+ visibilities: std::collections::HashMap<String, Option<String>>,
41+ noted: std::collections::HashSet<String>,
42+}
43+
3544 /// How often a poll looks again for work while it waits.
3645 const POLL_EVERY_MS: u64 = 2_500;
3746 /// A runner that has been offline this long is removed, as on GitHub.
6978 pub last_seen_at: Option<String>,
7079 pub created_at: String,
7180 pub created_by: Option<String>,
81+ /// `nested-container`, `container` or `host` (migration 0015); none
82+ /// for a runner that never said.
83+ #[serde(default)]
84+ pub isolation: Option<String>,
7285 }
7386
7487 impl RunnerRow {
953966 let version: String = if a.version.is_empty() { row.version.clone() } else { a.version.chars().take(40).collect() };
954967 let work_id = active.as_ref().map(|(id, _)| id.as_str());
955968 let work_kind = active.as_ref().map(|(_, kind)| *kind);
956− let changed = version != row.version || work_id != row.work_id.as_deref() || work_kind != row.work_kind.as_deref();
969+ // How it is isolated, as it says now: a runner re-registered or
970+ // restarted with or without Docker says so on its next poll.
971+ let isolation = a
972+ .isolation
973+ .as_deref()
974+ .filter(|level| model::ISOLATIONS.contains(level))
975+ .map(str::to_owned)
976+ .or_else(|| row.isolation.clone());
977+ let changed = version != row.version
978+ || work_id != row.work_id.as_deref()
979+ || work_kind != row.work_kind.as_deref()
980+ || isolation != row.isolation;
957981 if changed || seen_is_stale(row.last_seen_at.as_deref(), at) {
958982 self.db
959− .prepare("UPDATE runners SET last_seen_at = ?, version = ?, work_id = ?, work_kind = ? WHERE id = ?")
960− .bind(&[seen.as_str().into(), version.into(), optional(work_id), optional(work_kind), row.id.as_str().into()])?
983+ .prepare("UPDATE runners SET last_seen_at = ?, version = ?, work_id = ?, work_kind = ?, isolation = ? WHERE id = ?")
984+ .bind(&[
985+ seen.as_str().into(),
986+ version.into(),
987+ optional(work_id),
988+ optional(work_kind),
989+ optional(isolation.as_deref()),
990+ row.id.as_str().into(),
991+ ])?
961992 .run()
962993 .await?;
963994 }
995+ let row = RunnerRow { isolation, ..row };
964996 let mut poll = Poll { assignment: None, cancel, credential, removed: false };
965997 // Busy, or an ephemeral runner that has had its one job.
966998 if active.is_some() || !a.running.is_empty() {
9741006 let groups = self.group_rows(&row.workspace).await?;
9751007 let group = row.group_id.as_ref().and_then(|id| groups.iter().find(|g| &g.id == id)).cloned();
9761008 let deadline = at + a.wait_ms.min(model::MAX_POLL_WAIT_MS);
1009+ let mut looked = Looked::default();
9771010 loop {
978− if let Some(assignment) = self.claim(&row, group.as_ref()).await? {
1011+ if let Some(assignment) = self.claim(&row, group.as_ref(), &mut looked).await? {
9791012 poll.assignment = Some(assignment);
9801013 return Ok(Outcome::Ok(poll));
9811014 }
9871020 }
9881021
9891022 /// Takes the oldest work this runner can do, if there is any.
990− async fn claim(&self, runner: &RunnerRow, group: Option<&GroupRow>) -> Result<Option<Assignment>> {
1023+ async fn claim(&self, runner: &RunnerRow, group: Option<&GroupRow>, looked: &mut Looked) -> Result<Option<Assignment>> {
9911024 let labels = runner.labels();
9921025 let group_name = group.map(|g| g.name.clone());
9931026 // Workflow jobs.
10211054 if runner.repo_id.is_none() && !group.is_none_or(|g| g.allows(&run_repo)) {
10221055 continue;
10231056 }
1057+ // A host-mode runner takes no public repository's job: it waits
1058+ // for a sandboxed runner, and says so when none could take it.
1059+ if runner.isolation.as_deref() == Some("host") {
1060+ let visibility = self.visibility_once(looked, &job.repo_id).await;
1061+ if !model::isolation_allows(runner.isolation.as_deref(), visibility.as_deref()) {
1062+ if looked.noted.insert(job.id.clone()) {
1063+ self.wait_for_sandbox(&job, &run_repo, &wanted).await?;
1064+ }
1065+ continue;
1066+ }
1067+ }
10241068 if let Some(max) = job.max_parallel {
10251069 let siblings = self
10261070 .db
11031147 if !wanted.matches(&labels, None) {
11041148 continue;
11051149 }
1150+ // Nor a public repository's agent work.
1151+ if runner.isolation.as_deref() == Some("host")
1152+ && let Some(repo_id) = &task.repo_id
1153+ {
1154+ let visibility = self.visibility_once(looked, repo_id).await;
1155+ if !model::isolation_allows(runner.isolation.as_deref(), visibility.as_deref()) {
1156+ continue;
1157+ }
1158+ }
11061159 if let Some(repo_id) = &runner.repo_id
11071160 && task.repo_id.as_deref() != Some(repo_id.as_str())
11081161 {
11681221 Ok(None)
11691222 }
11701223
1224+ /// [`Self::visibility`], asked once per repository in a poll.
1225+ async fn visibility_once(&self, looked: &mut Looked, repo_id: &str) -> Option<String> {
1226+ if let Some(known) = looked.visibilities.get(repo_id) {
1227+ return known.clone();
1228+ }
1229+ let found = self.visibility(repo_id).await;
1230+ looked.visibilities.insert(repo_id.to_owned(), found.clone());
1231+ found
1232+ }
1233+
1234+ /// A public repository's job that a host-mode runner passed over: when
1235+ /// no online runner that may take it could (its labels, group,
1236+ /// repository, and isolation), its page says it waits for a sandboxed
1237+ /// one. Written only when that changes what it says.
1238+ async fn wait_for_sandbox(&self, job: &JobRow, run_repo: &str, wanted: &Wanted) -> Result<()> {
1239+ let at = now_ms();
1240+ let groups = self.group_rows(&job.namespace.to_lowercase()).await?;
1241+ let runners = self.runner_rows(&job.namespace.to_lowercase()).await?;
1242+ let sandboxed = runners.iter().any(|other| takes_public_job(other, &groups, &job.repo_id, run_repo, wanted, at));
1243+ if sandboxed {
1244+ return Ok(());
1245+ }
1246+ self.db
1247+ .prepare("UPDATE jobs SET reason = ? WHERE id = ? AND status = 'queued' AND reason IS NOT ?")
1248+ .bind(&[model::WAITING_FOR_SANDBOX.into(), job.id.as_str().into(), model::WAITING_FOR_SANDBOX.into()])?
1249+ .run()
1250+ .await?;
1251+ Ok(())
1252+ }
1253+
11711254 /// A repository's visibility for a runner, `public` or `private`; none
11721255 /// when it cannot be read, and the runner decides without it.
11731256 async fn visibility(&self, repo_id: &str) -> Option<String> {
16051688 id: String,
16061689 }
16071690
1691+/// Whether `runner` is online and could take a public repository's job:
1692+/// not host-mode, and its repository, group and labels let it.
1693+fn takes_public_job(runner: &RunnerRow, groups: &[GroupRow], repo_id: &str, run_repo: &str, wanted: &Wanted, at: u64) -> bool {
1694+ let group = runner.group_id.as_ref().and_then(|id| groups.iter().find(|g| &g.id == id));
1695+ runner.online(at)
1696+ && model::isolation_allows(runner.isolation.as_deref(), Some("public"))
1697+ && runner.repo_id.as_deref().is_none_or(|repo| repo == repo_id)
1698+ && (runner.repo_id.is_some() || group.is_none_or(|g| g.allows(run_repo)))
1699+ && wanted.matches(&runner.labels(), group.map(|g| g.name.as_str()))
1700+}
1701+
16081702 fn group_view(group: &GroupRow, runners: &[RunnerRow]) -> RunnerGroup {
16091703 RunnerGroup {
16101704 id: group.id.clone(),
16611755 }
16621756
16631757 #[test]
1758+ fn only_a_sandboxed_runner_that_could_take_it_serves_a_public_job() {
1759+ let at = 1_000_000_000;
1760+ let runner = |isolation: Option<&str>| RunnerRow {
1761+ id: "rnr_1".into(),
1762+ workspace: "acme".into(),
1763+ repo_id: None,
1764+ repo: None,
1765+ group_id: None,
1766+ name: "a".into(),
1767+ labels: r#"["self-hosted","linux","x64"]"#.into(),
1768+ os: "linux".into(),
1769+ arch: "x64".into(),
1770+ version: String::new(),
1771+ ephemeral: 0,
1772+ credential_hash: String::new(),
1773+ previous_hash: None,
1774+ rotated_at: String::new(),
1775+ work_id: None,
1776+ work_kind: None,
1777+ spent: 0,
1778+ last_seen_at: Some(rfc3339(at)),
1779+ created_at: String::new(),
1780+ created_by: None,
1781+ isolation: isolation.map(str::to_owned),
1782+ };
1783+ let wanted = Wanted::of(&["self-hosted".to_owned(), "linux".to_owned()], None);
1784+ let takes = |row: &RunnerRow| takes_public_job(row, &[], "rep_1", "acme/web", &wanted, at);
1785+ assert!(!takes(&runner(Some("host"))));
1786+ assert!(takes(&runner(Some("container"))));
1787+ assert!(takes(&runner(Some("nested-container"))));
1788+ assert!(takes(&runner(None)), "an older runner, which refuses for itself");
1789+ assert!(!takes(&RunnerRow { repo_id: Some("rep_2".into()), ..runner(None) }), "another repository's");
1790+ assert!(!takes(&RunnerRow { labels: r#"["self-hosted","macos"]"#.into(), ..runner(None) }));
1791+ assert!(!takes(&RunnerRow { last_seen_at: None, ..runner(None) }), "offline");
1792+ }
1793+
1794+ #[test]
16641795 fn online_means_seen_in_the_last_ninety_seconds() {
16651796 let row = RunnerRow {
16661797 id: "rnr_1".into(),
16831814 last_seen_at: Some(rfc3339(1_000_000_000)),
16841815 created_at: String::new(),
16851816 created_by: None,
1817+ isolation: None,
16861818 };
16871819 assert!(row.online(1_000_000_000 + 60_000));
16881820 assert!(!row.online(1_000_000_000 + 120_000));