Skip to content
Sign up

flagon-io/g1t

public
00

Where people and agents ship software together. The open-source git platform for the whole job: issues, agents, checks and deploys to the edge.

g1t.sh
gitagentscloudflarerustforgemcp
270 linesCodeBlameRaw

Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.

Fast pages, required checks on the branch, self-hosted runners, honest incidents1//! Releases: where `g1t-runner` is published, how it updates itself, and
2//! the Linux build it runs inside containers.
3//!
4//! Every release is at `<site>/downloads/runner/<version>/`: one binary per
5//! platform (`g1t-runner-linux-x64`, `-linux-arm64`, `-macos-arm64`,
6//! `-macos-x64`, `-windows-x64.exe`), `SHA256SUMS`, and `manifest.json`.
7//! `<site>/downloads/runner/latest.json` is the newest release's manifest,
8//! and `latest.json.sig` its Ed25519 signature, made with g1t's release key
9//! (`scripts/runner-release.mjs`). A runner only trusts a manifest whose
10//! signature checks out against the key built into it, and only runs a
11//! binary whose SHA-256 is the manifest's.
12
13use std::path::{Path, PathBuf};
14
15use anyhow::{Context, Result, anyhow, bail};
16use base64::Engine;
17use base64::engine::general_purpose::STANDARD;
18use serde::Deserialize;
19use sha2::{Digest, Sha256};
20
21use super::api::download;
22use super::config::Config;
23use super::{VERSION, log};
24
25/// g1t's release key, as base64 of its 32 bytes: set when release builds
26/// are made. Without it a runner never updates itself and only takes the
27/// Linux harness from the release when its SHA-256 matches the unsigned
28/// manifest it fetched over HTTPS.
29const RELEASE_KEY: Option<&str> = option_env!("G1T_RUNNER_RELEASE_KEY");
30
31#[derive(Clone, Debug, Deserialize)]
32pub struct File {
33 pub name: String,
34 pub sha256: String,
35}
36
37#[derive(Clone, Debug, Deserialize)]
38pub struct Manifest {
39 pub version: String,
40 /// The image agent work runs in.
41 #[serde(default)]
42 pub agent_image: Option<String>,
43 /// By platform: `linux-x64`, `macos-arm64`, `windows-x64`…
44 pub files: std::collections::BTreeMap<String, File>,
Self-hosted runners on a person's own computer are sandboxes that nothing a job runs can get out of. `g1t-runner sandbox up|down|status|update [--json]` gives a Windows, macOS or Linux computer up to two capabilities, each a runner of its own: Linux jobs in a Linux VM the runner owns, where the runner's image runs under Sysbox and each job is a fresh container inside it (nested-container, three walls from the computer); and jobs native to Windows, each in a Windows Sandbox of its own, discarded after, with one read-only folder holding the harness (a new `vm` isolation level, also `g1t-runner register --native`). On Windows the VM is a WSL 2 distribution imported from the release's signed root filesystem: its wsl.conf turns off drives, Windows programs and Windows' PATH, and the supervisor inside unmounts what WSL shares with every distribution anyway (/mnt/wsl, where Docker Desktop keeps its sockets to Windows, WSLg, the GPU driver folders), refuses to run while anything of the computer's is reachable, and drops containers' traffic to the computer's address but for DNS. It needs no administrator once WSL is installed, and works on Windows 11 Home. On macOS the VM runs under vfkit and on Linux without Sysbox under QEMU with KVM, with no shared folders and nothing forwarded in (built and tested as commands, not yet run on those machines); on Linux with Sysbox no VM is needed. `status --json` reports the state, versions, isolation, the current job and a sentence when something is wrong, and each capability's availability with its cause and fix, for the desktop app. `update` drains the runner, swaps in the newest signed image and starts again without registering again. The sandbox's images are built from deploy/sandbox/Dockerfile by scripts/sandbox-image.mjs and listed with their SHA-256 in the release's signed manifest, beside the runner image's digest. The release workflow now builds its images on g1t's own machines (g1t-4core) rather than waiting forever for a self-hosted runner, registering QEMU for arm64 where the machine allows it and building amd64 alone with a warning where it does not, and signs and publishes once the images are built. The Runners page shows each runner's isolation (Sandboxed, VM per job, Shared Docker, Unsandboxed), which the API now returns as `isolation`. The Self-hosted runners guide has a new "Sandbox on your computer" section, with what it cannot stop.45 /// The sandbox's images (`g1t-runner sandbox`, launcher/), signed with
46 /// the rest. None in releases from before it.
47 #[serde(default)]
48 pub sandbox: Option<SandboxRelease>,
Fast pages, required checks on the branch, self-hosted runners, honest incidents49}
50
Self-hosted runners on a person's own computer are sandboxes that nothing a job runs can get out of. `g1t-runner sandbox up|down|status|update [--json]` gives a Windows, macOS or Linux computer up to two capabilities, each a runner of its own: Linux jobs in a Linux VM the runner owns, where the runner's image runs under Sysbox and each job is a fresh container inside it (nested-container, three walls from the computer); and jobs native to Windows, each in a Windows Sandbox of its own, discarded after, with one read-only folder holding the harness (a new `vm` isolation level, also `g1t-runner register --native`). On Windows the VM is a WSL 2 distribution imported from the release's signed root filesystem: its wsl.conf turns off drives, Windows programs and Windows' PATH, and the supervisor inside unmounts what WSL shares with every distribution anyway (/mnt/wsl, where Docker Desktop keeps its sockets to Windows, WSLg, the GPU driver folders), refuses to run while anything of the computer's is reachable, and drops containers' traffic to the computer's address but for DNS. It needs no administrator once WSL is installed, and works on Windows 11 Home. On macOS the VM runs under vfkit and on Linux without Sysbox under QEMU with KVM, with no shared folders and nothing forwarded in (built and tested as commands, not yet run on those machines); on Linux with Sysbox no VM is needed. `status --json` reports the state, versions, isolation, the current job and a sentence when something is wrong, and each capability's availability with its cause and fix, for the desktop app. `update` drains the runner, swaps in the newest signed image and starts again without registering again. The sandbox's images are built from deploy/sandbox/Dockerfile by scripts/sandbox-image.mjs and listed with their SHA-256 in the release's signed manifest, beside the runner image's digest. The release workflow now builds its images on g1t's own machines (g1t-4core) rather than waiting forever for a self-hosted runner, registering QEMU for arm64 where the machine allows it and building amd64 alone with a warning where it does not, and signs and publishes once the images are built. The Runners page shows each runner's isolation (Sandboxed, VM per job, Shared Docker, Unsandboxed), which the API now returns as `isolation`. The Self-hosted runners guide has a new "Sandbox on your computer" section, with what it cannot stop.51/// What `g1t-runner sandbox` runs, in a release.
52#[derive(Clone, Debug, Default, Deserialize)]
53pub struct SandboxRelease {
54 /// The runner's image inside the sandbox, by digest
55 /// (`g1t.sh/flagon-io/g1t-runner@sha256:…`), so pulling it checks it.
56 #[serde(default)]
57 pub runner_image: Option<String>,
58 /// By kind and architecture: `wsl-x64` (a root filesystem, `.tar.gz`,
59 /// for a WSL distribution), `vm-arm64-kernel`, `vm-arm64-initrd`,
60 /// `vm-arm64-disk` (a VM's boot files and its system disk, `.gz`).
61 #[serde(default)]
62 pub files: std::collections::BTreeMap<String, SandboxFile>,
63}
64
65#[derive(Clone, Debug, Deserialize)]
66pub struct SandboxFile {
67 pub name: String,
68 pub sha256: String,
69 #[serde(default)]
70 pub size: u64,
71}
72
Fast pages, required checks on the branch, self-hosted runners, honest incidents73/// This machine's platform, as releases name it.
74pub fn platform() -> String {
75 let os = match std::env::consts::OS {
76 "macos" => "macos",
77 "windows" => "windows",
78 _ => "linux",
79 };
80 let arch = if std::env::consts::ARCH == "aarch64" { "arm64" } else { "x64" };
81 format!("{os}-{arch}")
82}
83
84fn downloads(config: &Config) -> String {
Self-hosted runners on a person's own computer are sandboxes that nothing a job runs can get out of. `g1t-runner sandbox up|down|status|update [--json]` gives a Windows, macOS or Linux computer up to two capabilities, each a runner of its own: Linux jobs in a Linux VM the runner owns, where the runner's image runs under Sysbox and each job is a fresh container inside it (nested-container, three walls from the computer); and jobs native to Windows, each in a Windows Sandbox of its own, discarded after, with one read-only folder holding the harness (a new `vm` isolation level, also `g1t-runner register --native`). On Windows the VM is a WSL 2 distribution imported from the release's signed root filesystem: its wsl.conf turns off drives, Windows programs and Windows' PATH, and the supervisor inside unmounts what WSL shares with every distribution anyway (/mnt/wsl, where Docker Desktop keeps its sockets to Windows, WSLg, the GPU driver folders), refuses to run while anything of the computer's is reachable, and drops containers' traffic to the computer's address but for DNS. It needs no administrator once WSL is installed, and works on Windows 11 Home. On macOS the VM runs under vfkit and on Linux without Sysbox under QEMU with KVM, with no shared folders and nothing forwarded in (built and tested as commands, not yet run on those machines); on Linux with Sysbox no VM is needed. `status --json` reports the state, versions, isolation, the current job and a sentence when something is wrong, and each capability's availability with its cause and fix, for the desktop app. `update` drains the runner, swaps in the newest signed image and starts again without registering again. The sandbox's images are built from deploy/sandbox/Dockerfile by scripts/sandbox-image.mjs and listed with their SHA-256 in the release's signed manifest, beside the runner image's digest. The release workflow now builds its images on g1t's own machines (g1t-4core) rather than waiting forever for a self-hosted runner, registering QEMU for arm64 where the machine allows it and building amd64 alone with a warning where it does not, and signs and publishes once the images are built. The Runners page shows each runner's isolation (Sandboxed, VM per job, Shared Docker, Unsandboxed), which the API now returns as `isolation`. The Self-hosted runners guide has a new "Sandbox on your computer" section, with what it cannot stop.85 downloads_at(&config.url)
86}
87
88/// Where a site keeps the runner's releases.
89pub fn downloads_at(site: &str) -> String {
90 format!("{}/downloads/runner", site.trim_end_matches('/'))
Fast pages, required checks on the branch, self-hosted runners, honest incidents91}
92
93/// `a.b.c` newer than `x.y.z`.
94pub fn newer(candidate: &str, current: &str) -> bool {
95 let parts = |v: &str| -> Vec<u64> { v.trim_start_matches('v').split(['.', '-']).take(3).map(|p| p.parse().unwrap_or(0)).collect() };
96 parts(candidate) > parts(current)
97}
98
99pub fn sha256(bytes: &[u8]) -> String {
100 hex::encode(Sha256::digest(bytes))
101}
102
103/// Whether `signature` (base64) is g1t's release key's over `message`.
104pub fn verify(message: &[u8], signature: &str, key: &str) -> Result<()> {
105 let key: [u8; 32] = STANDARD
106 .decode(key.trim())
107 .ok()
108 .and_then(|bytes| bytes.try_into().ok())
109 .ok_or_else(|| anyhow!("the release key built into this runner does not read"))?;
110 let signature: [u8; 64] = STANDARD
111 .decode(signature.trim())
112 .ok()
113 .and_then(|bytes| bytes.try_into().ok())
114 .ok_or_else(|| anyhow!("the release's signature does not read"))?;
115 let key = ed25519_dalek::VerifyingKey::from_bytes(&key).map_err(|_| anyhow!("the release key is not a key"))?;
116 key.verify_strict(message, &ed25519_dalek::Signature::from_bytes(&signature))
117 .map_err(|_| anyhow!("the release's signature is not g1t's"))
118}
119
120/// The newest release, signed. `None` when this build has no release key.
121fn latest_signed(config: &Config) -> Result<Option<Manifest>> {
Self-hosted runners on a person's own computer are sandboxes that nothing a job runs can get out of. `g1t-runner sandbox up|down|status|update [--json]` gives a Windows, macOS or Linux computer up to two capabilities, each a runner of its own: Linux jobs in a Linux VM the runner owns, where the runner's image runs under Sysbox and each job is a fresh container inside it (nested-container, three walls from the computer); and jobs native to Windows, each in a Windows Sandbox of its own, discarded after, with one read-only folder holding the harness (a new `vm` isolation level, also `g1t-runner register --native`). On Windows the VM is a WSL 2 distribution imported from the release's signed root filesystem: its wsl.conf turns off drives, Windows programs and Windows' PATH, and the supervisor inside unmounts what WSL shares with every distribution anyway (/mnt/wsl, where Docker Desktop keeps its sockets to Windows, WSLg, the GPU driver folders), refuses to run while anything of the computer's is reachable, and drops containers' traffic to the computer's address but for DNS. It needs no administrator once WSL is installed, and works on Windows 11 Home. On macOS the VM runs under vfkit and on Linux without Sysbox under QEMU with KVM, with no shared folders and nothing forwarded in (built and tested as commands, not yet run on those machines); on Linux with Sysbox no VM is needed. `status --json` reports the state, versions, isolation, the current job and a sentence when something is wrong, and each capability's availability with its cause and fix, for the desktop app. `update` drains the runner, swaps in the newest signed image and starts again without registering again. The sandbox's images are built from deploy/sandbox/Dockerfile by scripts/sandbox-image.mjs and listed with their SHA-256 in the release's signed manifest, beside the runner image's digest. The release workflow now builds its images on g1t's own machines (g1t-4core) rather than waiting forever for a self-hosted runner, registering QEMU for arm64 where the machine allows it and building amd64 alone with a warning where it does not, and signs and publishes once the images are built. The Runners page shows each runner's isolation (Sandboxed, VM per job, Shared Docker, Unsandboxed), which the API now returns as `isolation`. The Self-hosted runners guide has a new "Sandbox on your computer" section, with what it cannot stop.122 latest_signed_at(&config.url)
123}
124
125/// The newest release of `site`, signed. `None` when this build has no
126/// release key.
127pub fn latest_signed_at(site: &str) -> Result<Option<Manifest>> {
Fast pages, required checks on the branch, self-hosted runners, honest incidents128 let Some(key) = RELEASE_KEY else { return Ok(None) };
Self-hosted runners on a person's own computer are sandboxes that nothing a job runs can get out of. `g1t-runner sandbox up|down|status|update [--json]` gives a Windows, macOS or Linux computer up to two capabilities, each a runner of its own: Linux jobs in a Linux VM the runner owns, where the runner's image runs under Sysbox and each job is a fresh container inside it (nested-container, three walls from the computer); and jobs native to Windows, each in a Windows Sandbox of its own, discarded after, with one read-only folder holding the harness (a new `vm` isolation level, also `g1t-runner register --native`). On Windows the VM is a WSL 2 distribution imported from the release's signed root filesystem: its wsl.conf turns off drives, Windows programs and Windows' PATH, and the supervisor inside unmounts what WSL shares with every distribution anyway (/mnt/wsl, where Docker Desktop keeps its sockets to Windows, WSLg, the GPU driver folders), refuses to run while anything of the computer's is reachable, and drops containers' traffic to the computer's address but for DNS. It needs no administrator once WSL is installed, and works on Windows 11 Home. On macOS the VM runs under vfkit and on Linux without Sysbox under QEMU with KVM, with no shared folders and nothing forwarded in (built and tested as commands, not yet run on those machines); on Linux with Sysbox no VM is needed. `status --json` reports the state, versions, isolation, the current job and a sentence when something is wrong, and each capability's availability with its cause and fix, for the desktop app. `update` drains the runner, swaps in the newest signed image and starts again without registering again. The sandbox's images are built from deploy/sandbox/Dockerfile by scripts/sandbox-image.mjs and listed with their SHA-256 in the release's signed manifest, beside the runner image's digest. The release workflow now builds its images on g1t's own machines (g1t-4core) rather than waiting forever for a self-hosted runner, registering QEMU for arm64 where the machine allows it and building amd64 alone with a warning where it does not, and signs and publishes once the images are built. The Runners page shows each runner's isolation (Sandboxed, VM per job, Shared Docker, Unsandboxed), which the API now returns as `isolation`. The Self-hosted runners guide has a new "Sandbox on your computer" section, with what it cannot stop.129 let base = downloads_at(site);
Fast pages, required checks on the branch, self-hosted runners, honest incidents130 let manifest = download(&format!("{base}/latest.json"))?;
131 let signature = String::from_utf8(download(&format!("{base}/latest.json.sig"))?)?;
132 verify(&manifest, &signature, key)?;
133 Ok(Some(serde_json::from_slice(&manifest)?))
134}
135
136/// The manifest of a version: signed when this build can check, otherwise
137/// as fetched over HTTPS.
138fn manifest_of(config: &Config, version: &str) -> Result<Manifest> {
139 if let Some(latest) = latest_signed(config)?.filter(|m| m.version == version) {
140 return Ok(latest);
141 }
142 let bytes = download(&format!("{}/{version}/manifest.json", downloads(config)))?;
143 Ok(serde_json::from_slice(&bytes)?)
144}
145
146/// The image agent work runs in, from the release.
147pub fn agent_image(config: &Config) -> Option<String> {
148 manifest_of(config, VERSION).ok().and_then(|m| m.agent_image)
149}
150
151/// A Linux build of the harness to mount into containers: `--harness`;
152/// this program, on Linux; else the release's for this version, fetched
153/// once and checked.
154pub fn linux_harness(config: &Config, folder: &Path) -> Result<PathBuf> {
155 if let Some(path) = &config.harness {
156 return Ok(path.clone());
157 }
158 if cfg!(target_os = "linux") {
159 return std::env::current_exe().context("could not find this program");
160 }
161 let arch = if std::env::consts::ARCH == "aarch64" { "arm64" } else { "x64" };
162 let path = folder.join("harness").join(VERSION).join("g1t-runner");
163 if path.exists() {
164 return Ok(path);
165 }
166 let manifest = manifest_of(config, VERSION)?;
167 let file = manifest
168 .files
169 .get(&format!("linux-{arch}"))
170 .ok_or_else(|| anyhow!("release {VERSION} has no Linux build for {arch}"))?;
171 log(&format!("Fetching the Linux harness for containers ({})", file.name));
172 let bytes = download(&format!("{}/{VERSION}/{}", downloads(config), file.name))?;
173 if sha256(&bytes) != file.sha256 {
174 bail!("the Linux harness's SHA-256 is not the release's; not using it");
175 }
176 std::fs::create_dir_all(path.parent().unwrap_or(folder))?;
177 std::fs::write(&path, bytes)?;
178 Ok(path)
179}
180
181/// Updates this program to the newest release, if there is one. Returns
182/// the new version when it did.
183pub fn update(config: &Config) -> Result<Option<String>> {
184 let Some(latest) = latest_signed(config)? else {
185 bail!("this build of g1t-runner has no release key, so it cannot check releases; download a release from {}", downloads(config));
186 };
187 if !newer(&latest.version, VERSION) {
188 return Ok(None);
189 }
190 let file = latest
191 .files
192 .get(&platform())
193 .ok_or_else(|| anyhow!("release {} has no build for {}", latest.version, platform()))?;
194 let bytes = download(&format!("{}/{}/{}", downloads(config), latest.version, file.name))?;
195 if sha256(&bytes) != file.sha256 {
196 bail!("the download's SHA-256 is not the release's; not updating");
197 }
198 replace_self(&bytes)?;
199 Ok(Some(latest.version))
200}
201
202/// Puts `bytes` where this program is. The running file is moved aside
203/// first: Windows will not overwrite a running program, but lets it be
204/// renamed.
205fn replace_self(bytes: &[u8]) -> Result<()> {
206 let me = std::env::current_exe()?;
207 let new = me.with_extension("new");
208 let old = me.with_extension("old");
209 std::fs::write(&new, bytes)?;
210 #[cfg(unix)]
211 {
212 use std::os::unix::fs::PermissionsExt;
213 std::fs::set_permissions(&new, std::fs::Permissions::from_mode(0o755))?;
214 }
215 let _ = std::fs::remove_file(&old);
216 std::fs::rename(&me, &old).context("could not move the old version aside")?;
217 if let Err(error) = std::fs::rename(&new, &me) {
218 let _ = std::fs::rename(&old, &me);
219 return Err(error).context("could not put the new version in place");
220 }
221 Ok(())
222}
223
224/// Whether this build can update itself.
225pub fn can_update() -> bool {
226 RELEASE_KEY.is_some()
227}
228
229#[cfg(test)]
230mod tests {
231 use super::*;
232 use ed25519_dalek::Signer;
233
234 #[test]
235 fn versions_compare_as_numbers() {
236 assert!(newer("0.10.0", "0.9.9"));
237 assert!(newer("v1.0.0", "0.9.0"));
238 assert!(!newer("0.2.0", "0.2.0"));
239 assert!(!newer("0.1.9", "0.2.0"));
240 }
241
242 #[test]
243 fn only_the_release_keys_signature_is_trusted() {
244 let signing = ed25519_dalek::SigningKey::from_bytes(&[7u8; 32]);
245 let key = STANDARD.encode(signing.verifying_key().to_bytes());
246 let manifest = br#"{"version":"0.2.0","files":{}}"#;
247 let signature = STANDARD.encode(signing.sign(manifest).to_bytes());
248 assert!(verify(manifest, &signature, &key).is_ok());
249 assert!(verify(br#"{"version":"9.9.9","files":{}}"#, &signature, &key).is_err());
250 let other = STANDARD.encode(ed25519_dalek::SigningKey::from_bytes(&[8u8; 32]).verifying_key().to_bytes());
251 assert!(verify(manifest, &signature, &other).is_err());
252 assert!(verify(manifest, "not base64", &key).is_err());
253 }
254
255 /// Made by scripts/runner-release.mjs: what it signs, the runner checks.
256 #[test]
257 fn the_release_scripts_signatures_check_out() {
258 let key = "4wuwkRbieiO9sWq1UBAcGDjAjin4cwJRG2F8LJVyr6U=";
259 let signature = "aXjDOfxYhm+iHacZwsxMadNxoHX07p62evYNqsXX4UZoDQ7pwWtFnF4jKfiqvAk+AmGVkcE+/uioMR2v+FixCw==";
260 assert!(verify(br#"{"version":"0.2.0","files":{}}"#, signature, key).is_ok());
261 assert!(verify(br#"{"version":"0.2.1","files":{}}"#, signature, key).is_err());
262 }
263
264 #[test]
265 fn platforms_are_named_as_releases_name_them() {
266 let name = platform();
267 assert!(["linux-x64", "linux-arm64", "macos-arm64", "macos-x64", "windows-x64", "windows-arm64"].contains(&name.as_str()), "{name}");
268 assert_eq!(sha256(b"abc"), "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad");
269 }
270}