Merge branch 'worktree-agent-a41ebf64a05c894fe' into claude/cool-newton-7nm8r9
10 files+1260−1330/10 viewed
| 1107 | 1107 | "g1t-scan", | |
| 1108 | 1108 | "hex", | |
| 1109 | 1109 | "libc", | |
| 1110 | + | "regex", | |
| 1110 | 1111 | "serde", | |
| 1111 | 1112 | "serde_json", | |
| 1112 | 1113 | "serde_yaml", |
| 62 | 62 | | `concurrency` with `cancel-in-progress`, for the workflow or for one job | The same: one run, or one job, of a group at a time. | | |
| 63 | 63 | | `permissions:` for the workflow or for one job, `read-all`, `write-all` | The same: they decide what [the job's token](#the-jobs-token) may do. | | |
| 64 | 64 | | `${{ }}` expressions: every operator, function and context | The same, including `hashFiles`, `success()`, `failure()`, `always()` and `cancelled()`. | | |
| 65 | − | | `run:` with `bash`, `sh`, `python` or a custom shell | The same. | | |
| 66 | − | | JavaScript actions (`uses: owner/repo@v7`, `owner/repo/path@v7`) | Fetched from that repository on g1t when g1t has it and your repository may use it, otherwise from GitHub, and run as they are, on Node 24, the runtime current actions declare. See [actions and workflows from other repositories](#actions-and-workflows-from-other-repositories). | | |
| 65 | + | | `run:` with `bash`, `sh`, `python`, `pwsh` or a custom shell | The same, with GitHub's file for each: `.sh` for `bash` and `sh`, `.py` for `python`, `.ps1` for PowerShell (stopping at the first error, and exiting with the last program's code), `.cmd` for `cmd`, and no extension for another program. `python` runs `python`, or `python3` on a machine without it. | | |
| 66 | + | | JavaScript actions (`uses: owner/repo@v7`, `owner/repo/path@v7`) | Fetched from that repository on g1t when g1t has it and your repository may use it, otherwise from GitHub, and run as they are, on Node 24, the runtime current actions declare. An action that declares `node20` runs on Node 20 where the machine has one. See [actions and workflows from other repositories](#actions-and-workflows-from-other-repositories) and [an action's pre and post](#an-actions-pre-and-post). | | |
| 67 | 67 | | Composite actions | The same. | | |
| 68 | 68 | | Reusable workflows (`jobs.<id>.uses: ./.g1t/workflows/build.yml`, or `owner/repo/.g1t/workflows/build.yml@v1` in another repository) | The same: `with:` inputs checked against their `type`, `secrets:` by name or `secrets: inherit`, `on.workflow_call` outputs, `strategy.matrix` on the calling job (one call for each combination), `concurrency` on the calling job, and up to ten levels of workflows with at most 50 different reusable workflows in a run. `.github/workflows/…` finds the workflow under `.g1t/` after the move. See [actions and workflows from other repositories](#actions-and-workflows-from-other-repositories). | | |
| 69 | 69 | | `actions/checkout` | Checks out from g1t, with `ref`, `fetch-depth`, `path`, `repository`, `token` and `submodules`. | | |
| 70 | 70 | | `GITHUB_OUTPUT`, `GITHUB_ENV`, `GITHUB_PATH`, `GITHUB_STATE`, `GITHUB_STEP_SUMMARY` | The same. Step summaries show on the run's page; see [job summaries](#job-summaries). | | |
| 71 | − | | `::error::`, `::warning::`, `::notice::`, `::group::`, `::add-mask::` | The same: errors and warnings become annotations on the run, and [masked](#masking-secrets) values stay hidden. | | |
| 71 | + | | `::error::`, `::warning::`, `::notice::`, `::group::`, `::add-mask::`, `::echo::on` | The same: errors and warnings become annotations on the run, with their `file`, `line`, `col`, `endLine` and `endColumn`, and [masked](#masking-secrets) values stay hidden. | | |
| 72 | + | | Problem matchers (`::add-matcher::`, `::remove-matcher::`) | The same, so `actions/setup-node`, `setup-go` and `setup-python` turn compiler and linter output into annotations. See [problem matchers](#problem-matchers). | | |
| 72 | 73 | | `secrets.*`, `vars.*`, `secrets.GITHUB_TOKEN` | The same. `secrets.G1T_TOKEN` is [the job's own token](#the-jobs-token); `GITHUB_TOKEN` is its alias. | | |
| 73 | 74 | | `environment:` on a job | The job waits for the environment's [protection rules](#environments), then reads each key's row for that environment, as environment secrets work, and the run records a [deployment](/guides/deployments-api/#deployments-from-g1t-actions) to it. `url` gives the deployment its address; `deployment: false` reads the environment's values without making one. The name may be an expression. | | |
| 74 | 75 | | `actions/upload-artifact`, `actions/download-artifact`, `actions/upload-artifact/merge` | The same inputs and outputs as version 4: `retention-days`, `overwrite`, `compression-level`, `include-hidden-files`, `!` exclusions, download by `pattern` with `merge-multiple`, and from another run with `run-id` and `github-token`. Up to 5 GiB each; see [artifacts](#artifacts). | | |
| ⋯ | |||
| 379 | 380 | Jobs run in a fresh sandbox each: Debian with Node 24, Python 3, Go, Rust, | |
| 380 | 381 | Java 21, .NET 8, Ruby 3.3, `build-essential`, `git`, `curl`, `jq`, Docker | |
| 381 | 382 | (with Buildx and Compose) and passwordless `sudo`, in GitHub's layout | |
| 382 | − | (`/home/runner/work`, `RUNNER_TEMP`, `RUNNER_TOOL_CACHE`). | |
| 383 | − | `runner.os` is `Linux`. `ubuntu-latest`, `ubuntu-24.04` and other Linux | |
| 383 | + | (`/home/runner/work`, `RUNNER_WORKSPACE`, `RUNNER_TEMP`, `RUNNER_TOOL_CACHE`). | |
| 384 | + | `runner.os` is `Linux`. Each step's `GITHUB_ACTION` is its `id`, or | |
| 385 | + | GitHub's name for it: `__run`, `__run_2` and on for `run:` steps, | |
| 386 | + | `__actions_checkout` for `uses: actions/checkout`. `ImageOS` and | |
| 387 | + | `ImageVersion` are set when the machine's image sets them. `ubuntu-latest`, `ubuntu-24.04` and other Linux | |
| 384 | 388 | labels all run here. A job whose `runs-on` names `self-hosted` waits for one | |
| 385 | 389 | of your [self-hosted runners](/guides/self-hosted-runners/) instead. Setup actions such as | |
| 386 | 390 | `actions/setup-node` and `actions/setup-python` install other versions as | |
| ⋯ | |||
| 775 | 779 | `actions/download-artifact` and `actions/upload-artifact/merge` work: | |
| 776 | 780 | g1t runs those itself. | |
| 777 | 781 | ||
| 782 | + | ## An action's pre and post | |
| 783 | + | ||
| 784 | + | A JavaScript action's `pre` runs before the job's first step, as on | |
| 785 | + | GitHub: each one is a step of its own, *Pre* and the step's name, in the | |
| 786 | + | order of the steps, when its `pre-if` holds (`always()` when it has | |
| 787 | + | none). Its `post` runs after the last step, *Post* and the step's name, | |
| 788 | + | last first. What `pre` saves with `GITHUB_STATE` or `::save-state::` | |
| 789 | + | reaches the step's main as `STATE_<name>`, and what either saves reaches | |
| 790 | + | `post`. | |
| 791 | + | ||
| 792 | + | An action in your repository (`uses: ./…`) is not checked out when the | |
| 793 | + | job starts, so its `pre` is not run and the step warns, as on GitHub. An | |
| 794 | + | action inside a composite action runs its `pre` right before its main. | |
| 795 | + | ||
| 796 | + | A step that gives an input the action does not declare is warned of: | |
| 797 | + | *Unexpected input(s) 'foo', valid inputs are ['token', 'path']*. The | |
| 798 | + | step still runs. | |
| 799 | + | ||
| 800 | + | ## Problem matchers | |
| 801 | + | ||
| 802 | + | `::add-matcher::path/to/matcher.json` adds the matchers in a file of | |
| 803 | + | GitHub's format, and they read every line of output from then on, for | |
| 804 | + | the rest of the job. A line one matches becomes an error, warning or | |
| 805 | + | notice annotation, with the file, line, column and message it found, in | |
| 806 | + | place of the line in the log. | |
| 807 | + | ||
| 808 | + | ```json | |
| 809 | + | { | |
| 810 | + | "problemMatcher": [ | |
| 811 | + | { | |
| 812 | + | "owner": "my-linter", | |
| 813 | + | "pattern": [ | |
| 814 | + | { | |
| 815 | + | "regexp": "^(.+):(\\d+):(\\d+): (error|warning) (.*)$", | |
| 816 | + | "file": 1, "line": 2, "column": 3, "severity": 4, "message": 5 | |
| 817 | + | } | |
| 818 | + | ] | |
| 819 | + | } | |
| 820 | + | ] | |
| 821 | + | } | |
| 822 | + | ``` | |
| 823 | + | ||
| 824 | + | - A pattern of several lines matches them in order. The last one can | |
| 825 | + | set `loop` to match each issue after a header line, as ESLint's | |
| 826 | + | stylish output has. | |
| 827 | + | - `severity` is the matcher's own default, `error` when it gives none. | |
| 828 | + | A severity other than `error`, `warning` or `notice` is no annotation. | |
| 829 | + | - A file is kept when it exists in a repository in the workspace, as a | |
| 830 | + | path inside it; `fromPath` names the file a relative path is from. | |
| 831 | + | - A matcher added with an owner already there takes its place. | |
| 832 | + | `::remove-matcher owner=my-linter::` removes it. | |
| 833 | + | - Patterns are regular expressions as Rust's `regex` reads them, which | |
| 834 | + | has no lookaround or backreferences. A file whose patterns do not | |
| 835 | + | read is refused, with an error in the log. | |
| 836 | + | ||
| 837 | + | `::echo::on` shows each workflow command in the log as it is read, | |
| 838 | + | until `::echo::off`; with [debug logging](#debug-logging) they are shown | |
| 839 | + | from the start. | |
| 840 | + | ||
| 778 | 841 | ## Caching Rust builds | |
| 779 | 842 | ||
| 780 | 843 | A checkout gives every file a new modification time, so Cargo compiles | |
| 104 | 104 | self.repository.split('/').next().unwrap_or_default() | |
| 105 | 105 | } | |
| 106 | 106 | ||
| 107 | + | /// The owner's id, as the event's `repository.owner.id` gives it. | |
| 108 | + | fn owner_id(&self) -> String { | |
| 109 | + | owner_id(&self.event).unwrap_or_default() | |
| 110 | + | } | |
| 111 | + | ||
| 107 | 112 | /// The `github` context for a job. `token` is `github.token` (and | |
| 108 | 113 | /// `secrets.GITHUB_TOKEN`); `job` is the job's id. | |
| 109 | 114 | pub fn context(&self, job: &str, token: &str, action: Option<&str>) -> Value { | |
| ⋯ | |||
| 131 | 136 | "repository": self.repository, | |
| 132 | 137 | "repository_id": self.repository_id, | |
| 133 | 138 | "repository_owner": self.owner(), | |
| 134 | − | "repository_owner_id": "", | |
| 139 | + | "repository_owner_id": self.owner_id(), | |
| 135 | 140 | "repositoryUrl": format!("{}/{}.git", self.server_url, self.repository), | |
| 136 | 141 | "retention_days": 14, | |
| 137 | 142 | "run_attempt": self.run_attempt.to_string(), | |
| ⋯ | |||
| 174 | 179 | set("GITHUB_REPOSITORY", self.repository.clone()); | |
| 175 | 180 | set("GITHUB_REPOSITORY_ID", self.repository_id.clone()); | |
| 176 | 181 | set("GITHUB_REPOSITORY_OWNER", self.owner().to_owned()); | |
| 182 | + | set("GITHUB_REPOSITORY_OWNER_ID", self.owner_id()); | |
| 177 | 183 | set("GITHUB_RETENTION_DAYS", "14".into()); | |
| 178 | 184 | set("GITHUB_RUN_ATTEMPT", self.run_attempt.to_string()); | |
| 179 | 185 | set("GITHUB_RUN_ID", self.run_id.clone()); | |
| ⋯ | |||
| 191 | 197 | set("RUNNER_TEMP", "/home/runner/_temp".into()); | |
| 192 | 198 | set("RUNNER_TOOL_CACHE", "/home/runner/_tool".into()); | |
| 193 | 199 | set("RUNNER_ENVIRONMENT", "github-hosted".into()); | |
| 200 | + | set("RUNNER_WORKSPACE", RUNNER_WORKSPACE.into()); | |
| 194 | 201 | vars | |
| 195 | 202 | } | |
| 196 | 203 | } | |
| 197 | 204 | ||
| 205 | + | /// The owner's id in a webhook-shaped payload, `repository.owner.id`, a | |
| 206 | + | /// number or a string. | |
| 207 | + | pub fn owner_id(event: &Value) -> Option<String> { | |
| 208 | + | match &event["repository"]["owner"]["id"] { | |
| 209 | + | Value::String(id) => Some(id.clone()), | |
| 210 | + | Value::Number(id) => Some(id.to_string()), | |
| 211 | + | _ => None, | |
| 212 | + | } | |
| 213 | + | } | |
| 214 | + | ||
| 198 | 215 | /// Where a job's repository is checked out, as on GitHub's runners. | |
| 199 | 216 | pub const WORKSPACE: &str = "/home/runner/work/repo"; | |
| 200 | 217 | ||
| 218 | + | /// The folder the workspace is in, as GitHub's `RUNNER_WORKSPACE` is. | |
| 219 | + | pub const RUNNER_WORKSPACE: &str = "/home/runner/work"; | |
| 220 | + | ||
| 201 | 221 | /// The `runner` context. | |
| 202 | 222 | /// Whether a head commit's `message` asks, as GitHub lets it, that the | |
| 203 | 223 | /// `push` and `pull_request` workflows it would start don't run: `[skip ci]`, | |
| ⋯ | |||
| 280 | 300 | assert_eq!(vars["GITHUB_REF_NAME"], "v1.2.0"); | |
| 281 | 301 | assert_eq!(vars["GITHUB_JOB"], "build"); | |
| 282 | 302 | assert_eq!(vars["RUNNER_OS"], "Linux"); | |
| 303 | + | assert_eq!(vars["RUNNER_WORKSPACE"], "/home/runner/work"); | |
| 304 | + | assert_eq!(vars["GITHUB_REPOSITORY_OWNER_ID"], ""); | |
| 305 | + | } | |
| 306 | + | ||
| 307 | + | #[test] | |
| 308 | + | fn the_owners_id_comes_from_the_event() { | |
| 309 | + | let info = RunInfo { | |
| 310 | + | repository: "acme/web".into(), | |
| 311 | + | event: json!({ "repository": { "owner": { "id": 42 } } }), | |
| 312 | + | ..RunInfo::default() | |
| 313 | + | }; | |
| 314 | + | assert_eq!(info.variables("build")["GITHUB_REPOSITORY_OWNER_ID"], "42"); | |
| 315 | + | assert_eq!(info.context("build", "", None)["repository_owner_id"], "42"); | |
| 316 | + | assert_eq!(owner_id(&json!({ "repository": { "owner": { "id": "ws_1" } } })).as_deref(), Some("ws_1")); | |
| 317 | + | assert_eq!(owner_id(&json!({})), None); | |
| 283 | 318 | } | |
| 284 | 319 | } | |
| 285 | 320 | ||
| 18 | 18 | crc32fast = "1" | |
| 19 | 19 | # Checking that a release of the self-hosted runner is g1t's. | |
| 20 | 20 | ed25519-dalek = "3" | |
| 21 | + | # Problem matchers' patterns (actions/matchers.rs). | |
| 22 | + | regex = { version = "1.13", default-features = false, features = ["std", "perf", "unicode-perl", "unicode-case"] } | |
| 21 | 23 | anyhow = "1" | |
| 22 | 24 | base64 = "0.22" | |
| 23 | 25 | serde = { workspace = true } |
| 35 | 35 | pub(crate) id: String, | |
| 36 | 36 | /// `bash`, or `sh` in an image without it. | |
| 37 | 37 | pub(crate) shell: &'static str, | |
| 38 | + | /// `python`, or `python3` in an image without it, for `shell: python`. | |
| 39 | + | pub(crate) python: &'static str, | |
| 38 | 40 | /// Whether the runner's Node runs in it. | |
| 39 | 41 | pub(crate) node: bool, | |
| 40 | 42 | /// The image's own `PATH`. | |
| ⋯ | |||
| 419 | 421 | self.log.line("##[warning]The runner's Node does not run in this image (it needs glibc and libstdc++), so JavaScript actions run beside the container, on g1t's image, with the same files."); | |
| 420 | 422 | } | |
| 421 | 423 | self.job_context.insert("container".into(), json!({ "id": id, "network": network })); | |
| 422 | − | self.container = Some(JobContainer { id: name, shell: if has_bash { "bash" } else { "sh" }, node, path }); | |
| 424 | + | let has_python = self.docker_output(&["exec", &name, "sh", "-c", "command -v python"]).is_some_and(|out| !out.is_empty()); | |
| 425 | + | self.container = Some(JobContainer { | |
| 426 | + | id: name, | |
| 427 | + | shell: if has_bash { "bash" } else { "sh" }, | |
| 428 | + | python: if has_python { "python" } else { "python3" }, | |
| 429 | + | node, | |
| 430 | + | path, | |
| 431 | + | }); | |
| 423 | 432 | } | |
| 424 | 433 | ||
| 425 | 434 | // Services with a health check are waited for. | |
| 1 | + | //! Problem matchers, as GitHub's runner has them. `::add-matcher::` reads a | |
| 2 | + | //! JSON file of patterns (`actions/setup-node`, `setup-go` and | |
| 3 | + | //! `setup-python` each add some), and from then on, for the rest of the | |
| 4 | + | //! job, an output line they match becomes an annotation in place of the | |
| 5 | + | //! line. A pattern can span lines, and the last one of several can `loop` | |
| 6 | + | //! to match many issues after one header. | |
| 7 | + | ||
| 8 | + | use std::path::{Path, PathBuf}; | |
| 9 | + | ||
| 10 | + | use regex::Regex; | |
| 11 | + | use serde_json::{Map, Value}; | |
| 12 | + | ||
| 13 | + | /// One line's pattern, and which of its groups hold what. | |
| 14 | + | #[derive(Debug)] | |
| 15 | + | struct Pattern { | |
| 16 | + | regex: Regex, | |
| 17 | + | file: Option<usize>, | |
| 18 | + | line: Option<usize>, | |
| 19 | + | column: Option<usize>, | |
| 20 | + | severity: Option<usize>, | |
| 21 | + | code: Option<usize>, | |
| 22 | + | message: Option<usize>, | |
| 23 | + | from_path: Option<usize>, | |
| 24 | + | looping: bool, | |
| 25 | + | } | |
| 26 | + | ||
| 27 | + | /// What the lines matched so far say about an issue. | |
| 28 | + | #[derive(Clone, Debug, Default, PartialEq)] | |
| 29 | + | pub(crate) struct Found { | |
| 30 | + | pub(crate) file: Option<String>, | |
| 31 | + | pub(crate) line: Option<String>, | |
| 32 | + | pub(crate) column: Option<String>, | |
| 33 | + | pub(crate) severity: Option<String>, | |
| 34 | + | pub(crate) code: Option<String>, | |
| 35 | + | pub(crate) message: Option<String>, | |
| 36 | + | pub(crate) from_path: Option<String>, | |
| 37 | + | } | |
| 38 | + | ||
| 39 | + | impl Found { | |
| 40 | + | /// `running` with what `pattern` read from `caps` over it. | |
| 41 | + | fn read(running: Option<&Found>, pattern: &Pattern, caps: ®ex::Captures) -> Found { | |
| 42 | + | let mut found = running.cloned().unwrap_or_default(); | |
| 43 | + | let group = |index: Option<usize>, into: &mut Option<String>| { | |
| 44 | + | if let Some(index) = index { | |
| 45 | + | *into = caps.get(index).map(|m| m.as_str().to_owned()); | |
| 46 | + | } | |
| 47 | + | }; | |
| 48 | + | group(pattern.file, &mut found.file); | |
| 49 | + | group(pattern.line, &mut found.line); | |
| 50 | + | group(pattern.column, &mut found.column); | |
| 51 | + | group(pattern.severity, &mut found.severity); | |
| 52 | + | group(pattern.code, &mut found.code); | |
| 53 | + | group(pattern.message, &mut found.message); | |
| 54 | + | group(pattern.from_path, &mut found.from_path); | |
| 55 | + | found | |
| 56 | + | } | |
| 57 | + | } | |
| 58 | + | ||
| 59 | + | /// One matcher: its owner, the severity it gives by default, its patterns, | |
| 60 | + | /// and how far into them the output has gone. | |
| 61 | + | #[derive(Debug)] | |
| 62 | + | pub(crate) struct Matcher { | |
| 63 | + | pub(crate) owner: String, | |
| 64 | + | severity: Option<String>, | |
| 65 | + | patterns: Vec<Pattern>, | |
| 66 | + | /// For each pattern but the last, what it matched on the line before. | |
| 67 | + | state: Vec<Option<Found>>, | |
| 68 | + | } | |
| 69 | + | ||
| 70 | + | impl Matcher { | |
| 71 | + | fn reset(&mut self) { | |
| 72 | + | self.state.iter_mut().for_each(|s| *s = None); | |
| 73 | + | } | |
| 74 | + | ||
| 75 | + | /// The issue `line` completes, if it does, as GitHub's runner reads one: | |
| 76 | + | /// a single pattern matches by itself; several are worked through from | |
| 77 | + | /// the last back, each needing the one before it to have matched the | |
| 78 | + | /// line before. | |
| 79 | + | pub(crate) fn matches(&mut self, line: &str) -> Option<Found> { | |
| 80 | + | let last = self.patterns.len() - 1; | |
| 81 | + | if last == 0 { | |
| 82 | + | let caps = self.patterns[0].regex.captures(line)?; | |
| 83 | + | return Some(self.with_default(Found::read(None, &self.patterns[0], &caps))); | |
| 84 | + | } | |
| 85 | + | for i in (0..=last).rev() { | |
| 86 | + | let running = if i > 0 { self.state[i - 1].clone() } else { None }; | |
| 87 | + | if i > 0 && running.is_none() { | |
| 88 | + | continue; | |
| 89 | + | } | |
| 90 | + | let pattern = &self.patterns[i]; | |
| 91 | + | match pattern.regex.captures(line) { | |
| 92 | + | Some(caps) if i == last => { | |
| 93 | + | let found = Found::read(running.as_ref(), pattern, &caps); | |
| 94 | + | let looping = pattern.looping; | |
| 95 | + | self.reset(); | |
| 96 | + | // A loop keeps what led up to it, for the next line. | |
| 97 | + | if looping { | |
| 98 | + | self.state[i - 1] = running; | |
| 99 | + | } | |
| 100 | + | return Some(self.with_default(found)); | |
| 101 | + | } | |
| 102 | + | Some(caps) => self.state[i] = Some(Found::read(running.as_ref(), pattern, &caps)), | |
| 103 | + | None if i == last => self.state[i - 1] = None, | |
| 104 | + | None => self.state[i] = None, | |
| 105 | + | } | |
| 106 | + | } | |
| 107 | + | None | |
| 108 | + | } | |
| 109 | + | ||
| 110 | + | fn with_default(&self, mut found: Found) -> Found { | |
| 111 | + | if found.severity.as_deref().is_none_or(str::is_empty) { | |
| 112 | + | found.severity = self.severity.clone(); | |
| 113 | + | } | |
| 114 | + | found | |
| 115 | + | } | |
| 116 | + | } | |
| 117 | + | ||
| 118 | + | /// Reads a problem matcher file: `{"problemMatcher": [{"owner", | |
| 119 | + | /// "severity"?, "pattern": [{"regexp", "file", "line", "column", | |
| 120 | + | /// "severity", "code", "message", "fromPath", "loop"}]}]}`. | |
| 121 | + | pub(crate) fn parse(text: &str) -> Result<Vec<Matcher>, String> { | |
| 122 | + | let config: Value = serde_json::from_str(text).map_err(|e| format!("The problem matcher file is not JSON: {e}"))?; | |
| 123 | + | let Some(list) = config.get("problemMatcher").and_then(Value::as_array) else { | |
| 124 | + | return Err("The problem matcher file has no `problemMatcher` list.".into()); | |
| 125 | + | }; | |
| 126 | + | list.iter().map(parse_matcher).collect() | |
| 127 | + | } | |
| 128 | + | ||
| 129 | + | fn parse_matcher(config: &Value) -> Result<Matcher, String> { | |
| 130 | + | let owner = config.get("owner").and_then(Value::as_str).filter(|o| !o.trim().is_empty()).ok_or("A problem matcher needs an `owner`.")?; | |
| 131 | + | let severity = config.get("severity").and_then(Value::as_str).map(str::to_owned); | |
| 132 | + | let patterns: Vec<&Value> = match config.get("pattern") { | |
| 133 | + | Some(Value::Array(items)) => items.iter().collect(), | |
| 134 | + | Some(item @ Value::Object(_)) => vec![item], | |
| 135 | + | _ => Vec::new(), | |
| 136 | + | }; | |
| 137 | + | if patterns.is_empty() { | |
| 138 | + | return Err(format!("The problem matcher `{owner}` has no patterns.")); | |
| 139 | + | } | |
| 140 | + | let count = patterns.len(); | |
| 141 | + | let mut read = Vec::new(); | |
| 142 | + | for (index, pattern) in patterns.into_iter().enumerate() { | |
| 143 | + | let regexp = pattern.get("regexp").and_then(Value::as_str).filter(|r| !r.is_empty()).ok_or(format!("A pattern of `{owner}` needs a `regexp`."))?; | |
| 144 | + | let regex = Regex::new(regexp).map_err(|e| format!("A pattern of `{owner}` does not read: {e}"))?; | |
| 145 | + | let group = |key: &str| -> Result<Option<usize>, String> { | |
| 146 | + | match pattern.get(key) { | |
| 147 | + | None | Some(Value::Null) => Ok(None), | |
| 148 | + | Some(value) => value.as_u64().map(|n| Some(n as usize)).ok_or(format!("`{key}` in a pattern of `{owner}` must be a group's number.")), | |
| 149 | + | } | |
| 150 | + | }; | |
| 151 | + | let looping = pattern.get("loop").and_then(Value::as_bool).unwrap_or(false); | |
| 152 | + | let read_pattern = Pattern { | |
| 153 | + | regex, | |
| 154 | + | file: group("file")?, | |
| 155 | + | line: group("line")?, | |
| 156 | + | column: group("column")?, | |
| 157 | + | severity: group("severity")?, | |
| 158 | + | code: group("code")?, | |
| 159 | + | message: group("message")?, | |
| 160 | + | from_path: group("fromPath")?, | |
| 161 | + | looping, | |
| 162 | + | }; | |
| 163 | + | if looping && (count == 1 || index != count - 1) { | |
| 164 | + | return Err(format!("Only the last of several patterns may `loop` (`{owner}`).")); | |
| 165 | + | } | |
| 166 | + | if looping && read_pattern.message.is_none() { | |
| 167 | + | return Err(format!("The looping pattern of `{owner}` must set `message`.")); | |
| 168 | + | } | |
| 169 | + | read.push(read_pattern); | |
| 170 | + | } | |
| 171 | + | if !read.iter().any(|p| p.message.is_some()) { | |
| 172 | + | return Err(format!("One of the patterns of `{owner}` must set `message`.")); | |
| 173 | + | } | |
| 174 | + | Ok(Matcher { owner: owner.to_owned(), severity, state: vec![None; read.len() - 1], patterns: read }) | |
| 175 | + | } | |
| 176 | + | ||
| 177 | + | /// An issue a matcher found: the annotation's level, its message, and its | |
| 178 | + | /// `file`, `line`, `col` and `code`, as an `::error::` command's properties. | |
| 179 | + | #[derive(Debug, PartialEq)] | |
| 180 | + | pub(crate) struct Issue { | |
| 181 | + | pub(crate) level: &'static str, | |
| 182 | + | pub(crate) message: String, | |
| 183 | + | pub(crate) properties: Map<String, Value>, | |
| 184 | + | } | |
| 185 | + | ||
| 186 | + | /// The job's matchers, in the order they were added. | |
| 187 | + | #[derive(Default)] | |
| 188 | + | pub(crate) struct Matchers { | |
| 189 | + | list: Vec<Matcher>, | |
| 190 | + | /// Where relative files are read from: `github.workspace`. | |
| 191 | + | pub(crate) workspace: PathBuf, | |
| 192 | + | } | |
| 193 | + | ||
| 194 | + | impl Matchers { | |
| 195 | + | /// Adds matchers; one with an owner already here takes its place. | |
| 196 | + | pub(crate) fn add(&mut self, new: Vec<Matcher>) { | |
| 197 | + | for matcher in new { | |
| 198 | + | match self.list.iter_mut().find(|m| m.owner.eq_ignore_ascii_case(&matcher.owner)) { | |
| 199 | + | Some(old) => *old = matcher, | |
| 200 | + | None => self.list.push(matcher), | |
| 201 | + | } | |
| 202 | + | } | |
| 203 | + | } | |
| 204 | + | ||
| 205 | + | pub(crate) fn remove(&mut self, owner: &str) { | |
| 206 | + | self.list.retain(|m| !m.owner.eq_ignore_ascii_case(owner)); | |
| 207 | + | } | |
| 208 | + | ||
| 209 | + | /// The issue `line` makes, if a matcher reads one in it. The first | |
| 210 | + | /// matcher to match has it, and the others start over. | |
| 211 | + | pub(crate) fn issue(&mut self, line: &str) -> Option<Issue> { | |
| 212 | + | if self.list.is_empty() { | |
| 213 | + | return None; | |
| 214 | + | } | |
| 215 | + | let line = strip_colors(line); | |
| 216 | + | for index in 0..self.list.len() { | |
| 217 | + | let Some(found) = self.list[index].matches(&line) else { continue }; | |
| 218 | + | for (other, matcher) in self.list.iter_mut().enumerate() { | |
| 219 | + | if other != index { | |
| 220 | + | matcher.reset(); | |
| 221 | + | } | |
| 222 | + | } | |
| 223 | + | if let Some(issue) = to_issue(&found, &self.workspace) { | |
| 224 | + | return Some(issue); | |
| 225 | + | } | |
| 226 | + | } | |
| 227 | + | None | |
| 228 | + | } | |
| 229 | + | } | |
| 230 | + | ||
| 231 | + | /// `\x1b[…m` colour codes, which a matcher never sees. | |
| 232 | + | fn strip_colors(line: &str) -> String { | |
| 233 | + | let mut out = String::with_capacity(line.len()); | |
| 234 | + | let mut rest = line; | |
| 235 | + | while let Some(start) = rest.find("\x1b[") { | |
| 236 | + | out.push_str(&rest[..start]); | |
| 237 | + | let after = &rest[start + 2..]; | |
| 238 | + | let end = after.find(|c: char| !(c.is_ascii_digit() || c == ';')); | |
| 239 | + | match end { | |
| 240 | + | Some(end) if after[end..].starts_with('m') => rest = &after[end + 1..], | |
| 241 | + | _ => { | |
| 242 | + | out.push_str("\x1b["); | |
| 243 | + | rest = after; | |
| 244 | + | } | |
| 245 | + | } | |
| 246 | + | } | |
| 247 | + | out.push_str(rest); | |
| 248 | + | out | |
| 249 | + | } | |
| 250 | + | ||
| 251 | + | /// What GitHub's runner makes of a match: nothing without a message or | |
| 252 | + | /// with a severity other than error, warning or notice; a line and column | |
| 253 | + | /// only when they are numbers; a file only when it is there, inside a | |
| 254 | + | /// repository in the workspace, and then relative to that repository. | |
| 255 | + | fn to_issue(found: &Found, workspace: &Path) -> Option<Issue> { | |
| 256 | + | let message = found.message.as_deref().filter(|m| !m.trim().is_empty())?; | |
| 257 | + | let level = match found.severity.as_deref().map(str::to_ascii_lowercase).as_deref() { | |
| 258 | + | None | Some("") | Some("error") => "error", | |
| 259 | + | Some("warning") => "warning", | |
| 260 | + | Some("notice") => "notice", | |
| 261 | + | Some(_) => return None, | |
| 262 | + | }; | |
| 263 | + | let mut properties = Map::new(); | |
| 264 | + | for (key, value) in [("line", &found.line), ("col", &found.column)] { | |
| 265 | + | if let Some(number) = value.as_deref().and_then(|v| v.trim().parse::<u32>().ok()) { | |
| 266 | + | properties.insert(key.into(), Value::String(number.to_string())); | |
| 267 | + | } | |
| 268 | + | } | |
| 269 | + | if let Some(code) = found.code.as_deref().map(str::trim).filter(|c| !c.is_empty()) { | |
| 270 | + | properties.insert("code".into(), Value::String(code.to_owned())); | |
| 271 | + | } | |
| 272 | + | if let Some(file) = found.file.as_deref().map(str::trim).filter(|f| !f.is_empty()) | |
| 273 | + | && let Some(file) = repository_file(file, found.from_path.as_deref(), workspace) | |
| 274 | + | { | |
| 275 | + | properties.insert("file".into(), Value::String(file)); | |
| 276 | + | } | |
| 277 | + | Some(Issue { level, message: message.to_owned(), properties }) | |
| 278 | + | } | |
| 279 | + | ||
| 280 | + | /// `file` as a path inside its repository, if it is in one in the workspace. | |
| 281 | + | fn repository_file(file: &str, from_path: Option<&str>, workspace: &Path) -> Option<String> { | |
| 282 | + | let mut path = PathBuf::from(file.replace('\\', "/")); | |
| 283 | + | if !path.is_absolute() | |
| 284 | + | && let Some(from) = from_path.map(Path::new).and_then(Path::parent).filter(|p| !p.as_os_str().is_empty()) | |
| 285 | + | { | |
| 286 | + | path = from.join(path); | |
| 287 | + | } | |
| 288 | + | if !path.is_absolute() { | |
| 289 | + | path = workspace.join(path); | |
| 290 | + | } | |
| 291 | + | if !path.is_file() { | |
| 292 | + | return None; | |
| 293 | + | } | |
| 294 | + | let repository = path.ancestors().skip(1).take_while(|dir| dir.starts_with(workspace)).find(|dir| dir.join(".git").exists())?; | |
| 295 | + | let relative = path.strip_prefix(repository).ok()?; | |
| 296 | + | let parts: Vec<String> = relative.iter().map(|part| part.to_string_lossy().into_owned()).collect(); | |
| 297 | + | Some(parts.join("/")) | |
| 298 | + | } | |
| 299 | + | ||
| 300 | + | #[cfg(test)] | |
| 301 | + | mod tests { | |
| 302 | + | use super::*; | |
| 303 | + | ||
| 304 | + | /// actions/setup-node's `tsc` matcher, as it ships. | |
| 305 | + | const TSC: &str = r#"{ | |
| 306 | + | "problemMatcher": [{ | |
| 307 | + | "owner": "tsc", | |
| 308 | + | "pattern": [{ | |
| 309 | + | "regexp": "^([^\\s].*)[\\(:](\\d+)[,:](\\d+)(?:\\):\\s+|\\s+-\\s+)(error|warning|info)\\s+TS(\\d+)\\s*:\\s*(.*)$", | |
| 310 | + | "file": 1, "line": 2, "column": 3, "severity": 4, "code": 5, "message": 6 | |
| 311 | + | }] | |
| 312 | + | }] | |
| 313 | + | }"#; | |
| 314 | + | ||
| 315 | + | /// actions/setup-node's `eslint-stylish`: a file, then its issues. | |
| 316 | + | const STYLISH: &str = r#"{ | |
| 317 | + | "problemMatcher": [{ | |
| 318 | + | "owner": "eslint-stylish", | |
| 319 | + | "pattern": [ | |
| 320 | + | { "regexp": "^([^\\s].*)$", "file": 1 }, | |
| 321 | + | { "regexp": "^\\s+(\\d+):(\\d+)\\s+(error|warning|info)\\s+(.*)\\s\\s+(.*)$", "line": 1, "column": 2, "severity": 3, "message": 4, "code": 5, "loop": true } | |
| 322 | + | ] | |
| 323 | + | }] | |
| 324 | + | }"#; | |
| 325 | + | ||
| 326 | + | #[test] | |
| 327 | + | fn a_single_pattern_reads_its_groups() { | |
| 328 | + | let mut matchers = parse(TSC).unwrap(); | |
| 329 | + | assert_eq!(matchers[0].owner, "tsc"); | |
| 330 | + | let found = matchers[0].matches("src/app.ts(12,5): error TS2322: Type 'string' is not assignable.").unwrap(); | |
| 331 | + | assert_eq!(found.file.as_deref(), Some("src/app.ts")); | |
| 332 | + | assert_eq!(found.line.as_deref(), Some("12")); | |
| 333 | + | assert_eq!(found.column.as_deref(), Some("5")); | |
| 334 | + | assert_eq!(found.severity.as_deref(), Some("error")); | |
| 335 | + | assert_eq!(found.code.as_deref(), Some("2322")); | |
| 336 | + | assert_eq!(found.message.as_deref(), Some("Type 'string' is not assignable.")); | |
| 337 | + | assert!(matchers[0].matches("Compiled successfully.").is_none()); | |
| 338 | + | } | |
| 339 | + | ||
| 340 | + | #[test] | |
| 341 | + | fn several_patterns_span_lines_and_a_loop_goes_on() { | |
| 342 | + | let mut matcher = parse(STYLISH).unwrap().remove(0); | |
| 343 | + | assert!(matcher.matches("/w/src/a.js").is_none(), "the header alone is no issue"); | |
| 344 | + | let first = matcher.matches(" 1:10 error 'x' is defined but never used no-unused-vars").unwrap(); | |
| 345 | + | assert_eq!(first.file.as_deref(), Some("/w/src/a.js")); | |
| 346 | + | assert_eq!((first.line.as_deref(), first.column.as_deref()), (Some("1"), Some("10"))); | |
| 347 | + | assert_eq!(first.code.as_deref(), Some("no-unused-vars")); | |
| 348 | + | // The loop keeps the file for the next issue. | |
| 349 | + | let second = matcher.matches(" 3:1 warning Unexpected console statement no-console").unwrap(); | |
| 350 | + | assert_eq!(second.file.as_deref(), Some("/w/src/a.js")); | |
| 351 | + | assert_eq!(second.severity.as_deref(), Some("warning")); | |
| 352 | + | // A line that is neither ends the run. | |
| 353 | + | assert!(matcher.matches("").is_none()); | |
| 354 | + | assert!(matcher.matches(" 4:1 error Late semi").is_none(), "no file is held after the run ended"); | |
| 355 | + | } | |
| 356 | + | ||
| 357 | + | #[test] | |
| 358 | + | fn without_a_loop_each_issue_needs_its_header() { | |
| 359 | + | let config = r#"{"problemMatcher": [{"owner": "two", "severity": "warning", "pattern": [ | |
| 360 | + | {"regexp": "^FILE (.*)$", "file": 1}, | |
| 361 | + | {"regexp": "^MSG (.*)$", "message": 1} | |
| 362 | + | ]}]}"#; | |
| 363 | + | let mut matcher = parse(config).unwrap().remove(0); | |
| 364 | + | assert!(matcher.matches("MSG orphan").is_none()); | |
| 365 | + | assert!(matcher.matches("FILE a.txt").is_none()); | |
| 366 | + | let found = matcher.matches("MSG broke").unwrap(); | |
| 367 | + | assert_eq!((found.file.as_deref(), found.message.as_deref()), (Some("a.txt"), Some("broke"))); | |
| 368 | + | assert_eq!(found.severity.as_deref(), Some("warning"), "the matcher's own severity"); | |
| 369 | + | assert!(matcher.matches("MSG again").is_none()); | |
| 370 | + | // A line between header and message breaks the pair. | |
| 371 | + | matcher.matches("FILE b.txt"); | |
| 372 | + | matcher.matches("something else"); | |
| 373 | + | assert!(matcher.matches("MSG late").is_none()); | |
| 374 | + | } | |
| 375 | + | ||
| 376 | + | #[test] | |
| 377 | + | fn bad_matchers_are_refused() { | |
| 378 | + | assert!(parse("not json").is_err()); | |
| 379 | + | assert!(parse(r#"{"problemMatcher": [{"pattern": [{"regexp": "x", "message": 0}]}]}"#).unwrap_err().contains("owner")); | |
| 380 | + | assert!(parse(r#"{"problemMatcher": [{"owner": "a", "pattern": [{"regexp": "x"}]}]}"#).unwrap_err().contains("message")); | |
| 381 | + | assert!(parse(r#"{"problemMatcher": [{"owner": "a", "pattern": [{"regexp": "x", "message": 0, "loop": true}]}]}"#).unwrap_err().contains("loop")); | |
| 382 | + | assert!(parse(r#"{"problemMatcher": [{"owner": "a", "pattern": [{"regexp": "(", "message": 1}]}]}"#).is_err()); | |
| 383 | + | } | |
| 384 | + | ||
| 385 | + | #[test] | |
| 386 | + | fn matchers_are_replaced_by_owner_and_removed() { | |
| 387 | + | let mut matchers = Matchers::default(); | |
| 388 | + | matchers.add(parse(TSC).unwrap()); | |
| 389 | + | matchers.add(parse(TSC).unwrap()); | |
| 390 | + | assert_eq!(matchers.list.len(), 1); | |
| 391 | + | matchers.add(parse(STYLISH).unwrap()); | |
| 392 | + | assert_eq!(matchers.list.len(), 2); | |
| 393 | + | matchers.remove("TSC"); | |
| 394 | + | assert_eq!(matchers.list[0].owner, "eslint-stylish"); | |
| 395 | + | } | |
| 396 | + | ||
| 397 | + | #[test] | |
| 398 | + | fn issues_carry_numbers_and_files_inside_the_repository() { | |
| 399 | + | let workspace = std::env::temp_dir().join(format!("g1t-matchers-{}", std::process::id())); | |
| 400 | + | std::fs::create_dir_all(workspace.join(".git")).unwrap(); | |
| 401 | + | std::fs::create_dir_all(workspace.join("src")).unwrap(); | |
| 402 | + | std::fs::write(workspace.join("src/app.ts"), "").unwrap(); | |
| 403 | + | let mut matchers = Matchers { workspace: workspace.clone(), ..Matchers::default() }; | |
| 404 | + | matchers.add(parse(TSC).unwrap()); | |
| 405 | + | let issue = matchers.issue("\x1b[31msrc/app.ts(12,5): error TS2322: Bad.\x1b[0m").unwrap(); | |
| 406 | + | assert_eq!(issue.level, "error"); | |
| 407 | + | assert_eq!(issue.message, "Bad."); | |
| 408 | + | assert_eq!(issue.properties["file"], "src/app.ts"); | |
| 409 | + | assert_eq!(issue.properties["line"], "12"); | |
| 410 | + | assert_eq!(issue.properties["col"], "5"); | |
| 411 | + | assert_eq!(issue.properties["code"], "2322"); | |
| 412 | + | // A file that is not there is dropped; the issue stays. | |
| 413 | + | let issue = matchers.issue("gone.ts(1,1): warning TS1: Missing.").unwrap(); | |
| 414 | + | assert_eq!(issue.level, "warning"); | |
| 415 | + | assert!(!issue.properties.contains_key("file")); | |
| 416 | + | // An absolute path inside the workspace is made relative. | |
| 417 | + | let absolute = format!("{}(2,3): error TS9: Abs.", workspace.join("src/app.ts").display()); | |
| 418 | + | assert_eq!(matchers.issue(&absolute).unwrap().properties["file"], "src/app.ts"); | |
| 419 | + | // `info` is no annotation GitHub has. | |
| 420 | + | assert!(matchers.issue("src/app.ts(1,1): info TS1: Hint.").is_none()); | |
| 421 | + | let _ = std::fs::remove_dir_all(&workspace); | |
| 422 | + | } | |
| 423 | + | ||
| 424 | + | #[test] | |
| 425 | + | fn colour_codes_are_stripped() { | |
| 426 | + | assert_eq!(strip_colors("\x1b[1;31merror\x1b[0m: x"), "error: x"); | |
| 427 | + | assert_eq!(strip_colors("plain"), "plain"); | |
| 428 | + | assert_eq!(strip_colors("\x1b[Kodd"), "\x1b[Kodd"); | |
| 429 | + | } | |
| 430 | + | } |
| 12 | 12 | mod containers; | |
| 13 | 13 | mod files; | |
| 14 | 14 | mod glob; | |
| 15 | + | mod matchers; | |
| 15 | 16 | mod paths; | |
| 16 | 17 | mod process; | |
| 17 | 18 | mod report; | |
| ⋯ | |||
| 57 | 58 | } | |
| 58 | 59 | ||
| 59 | 60 | pub(crate) enum PostRun { | |
| 60 | − | Node { action_dir: PathBuf, script: String }, | |
| 61 | + | Node { action_dir: PathBuf, script: String, using: String }, | |
| 61 | 62 | CacheSave { key: String, paths: Vec<String>, version: String }, | |
| 62 | 63 | /// A Docker action's `post-entrypoint`. | |
| 63 | 64 | Docker(containers::DockerRun), | |
| ⋯ | |||
| 91 | 92 | /// What the last Node process left, for the step that ran it. | |
| 92 | 93 | pub(crate) last_node_outputs: BTreeMap<String, String>, | |
| 93 | 94 | pub(crate) last_node_state: BTreeMap<String, String>, | |
| 95 | + | /// What each action's `pre` saved, by its step's `GITHUB_ACTION`, for | |
| 96 | + | /// the step's main and post. | |
| 97 | + | pub(crate) pre_state: BTreeMap<String, BTreeMap<String, String>>, | |
| 94 | 98 | /// The names of the sandbox's own variables, which a container does | |
| 95 | 99 | /// not get. | |
| 96 | 100 | host_env: BTreeSet<String>, | |
| ⋯ | |||
| 115 | 119 | .unwrap_or_default() | |
| 116 | 120 | } | |
| 117 | 121 | ||
| 122 | + | /// Where `name_steps` puts a step's `GITHUB_ACTION`. | |
| 123 | + | const ACTION_NAME: &str = "__g1t_action"; | |
| 124 | + | ||
| 125 | + | /// Gives each step its `GITHUB_ACTION`, as GitHub's runner names them: its | |
| 126 | + | /// `id`, else `__run` for a script and `__owner_repo` for an action | |
| 127 | + | /// (`__self` for one in the repository), the second of a name `_2`, the | |
| 128 | + | /// third `_3`. | |
| 129 | + | pub(crate) fn name_steps(steps: &mut [Map<String, Value>]) { | |
| 130 | + | let mut taken: BTreeSet<String> = steps.iter().filter_map(|step| step.get("id").map(expr::to_text)).collect(); | |
| 131 | + | for step in steps.iter_mut() { | |
| 132 | + | let name = match step.get("id").map(expr::to_text) { | |
| 133 | + | Some(id) => id, | |
| 134 | + | None => { | |
| 135 | + | let base = match step.get("uses").map(expr::to_text) { | |
| 136 | + | None => "run".to_owned(), | |
| 137 | + | Some(uses) if uses.starts_with("./") => "self".to_owned(), | |
| 138 | + | Some(uses) => { | |
| 139 | + | let name = uses.split('@').next().unwrap_or_default(); | |
| 140 | + | let name = name.strip_prefix("docker://").unwrap_or(name); | |
| 141 | + | let repository: Vec<&str> = name.splitn(3, '/').take(2).collect(); | |
| 142 | + | repository.join("_").chars().map(|c| if c.is_ascii_alphanumeric() || c == '_' || c == '-' { c } else { '_' }).collect() | |
| 143 | + | } | |
| 144 | + | }; | |
| 145 | + | let base = format!("__{base}"); | |
| 146 | + | let mut name = base.clone(); | |
| 147 | + | let mut n = 2; | |
| 148 | + | while taken.contains(&name) { | |
| 149 | + | name = format!("{base}_{n}"); | |
| 150 | + | n += 1; | |
| 151 | + | } | |
| 152 | + | taken.insert(name.clone()); | |
| 153 | + | name | |
| 154 | + | } | |
| 155 | + | }; | |
| 156 | + | step.insert(ACTION_NAME.into(), Value::String(name)); | |
| 157 | + | } | |
| 158 | + | } | |
| 159 | + | ||
| 118 | 160 | /// A step's title when it has no name, as GitHub shows it. | |
| 119 | 161 | fn default_title(step: &Map<String, Value>) -> String { | |
| 120 | 162 | if let Some(uses) = step.get("uses").and_then(Value::as_str) { | |
| ⋯ | |||
| 285 | 327 | None => ("bash".into(), vec!["-e".into(), "{0}".into()], "sh"), | |
| 286 | 328 | Some("bash") => ("bash".into(), vec!["--noprofile".into(), "--norc".into(), "-eo".into(), "pipefail".into(), "{0}".into()], "sh"), | |
| 287 | 329 | Some("sh") => ("sh".into(), vec!["-e".into(), "{0}".into()], "sh"), | |
| 288 | − | Some("python") => ("python3".into(), vec!["{0}".into()], "py"), | |
| 330 | + | Some("python") => (self.python_program(), vec!["{0}".into()], "py"), | |
| 289 | 331 | Some("pwsh") if cfg!(windows) || program_exists("pwsh") => ("pwsh".into(), powershell_args(), "ps1"), | |
| 290 | 332 | Some("powershell") if cfg!(windows) => ("powershell".into(), powershell_args(), "ps1"), | |
| 291 | 333 | Some("cmd") if cfg!(windows) => ( | |
| ⋯ | |||
| 306 | 348 | if !args.iter().any(|a| a.contains("{0}")) { | |
| 307 | 349 | args.push("{0}".into()); | |
| 308 | 350 | } | |
| 309 | − | (program, args, "sh") | |
| 351 | + | let extension = script_extension(&program); | |
| 352 | + | (program, args, extension) | |
| 310 | 353 | } | |
| 311 | 354 | }; | |
| 312 | − | let script_path = self.temp.join(format!("{id}.{extension}")); | |
| 313 | − | if let Err(error) = std::fs::write(&script_path, script) { | |
| 355 | + | let script_path = self.temp.join(if extension.is_empty() { id.clone() } else { format!("{id}.{extension}") }); | |
| 356 | + | if let Err(error) = std::fs::write(&script_path, script_contents(&program, script)) { | |
| 314 | 357 | self.log.line(&format!("##[error]Could not write the script: {error}")); | |
| 315 | 358 | return (false, BTreeMap::new(), BTreeMap::new()); | |
| 316 | 359 | } | |
| ⋯ | |||
| 420 | 463 | } | |
| 421 | 464 | ||
| 422 | 465 | // The step's own env, read with the contexts before it. | |
| 423 | − | let mut env = env_before.clone(); | |
| 424 | − | if let Some(Value::Object(step_env)) = step.get("env") { | |
| 425 | − | for (name, value) in step_env { | |
| 426 | − | let value = self.with_scope(&contexts, |scope| expr::interpolate_value(value, scope)).unwrap_or(Value::Null); | |
| 427 | − | env.insert(name.clone(), expr::to_text(&value)); | |
| 428 | − | } | |
| 429 | − | } | |
| 466 | + | let env = self.step_env(step, &contexts, env_before.clone()); | |
| 467 | + | let action_name = step.get(ACTION_NAME).map(expr::to_text).or_else(|| id.clone()).unwrap_or_else(|| "__run".into()); | |
| 430 | 468 | let minutes = step | |
| 431 | 469 | .get("timeout-minutes") | |
| 432 | 470 | .and_then(|v| self.with_scope(&contexts, |scope| expr::interpolate_value(v, scope)).ok()) | |
| ⋯ | |||
| 470 | 508 | if let Some(path) = &frame.action_path { | |
| 471 | 509 | env.insert("GITHUB_ACTION_PATH".into(), path.clone()); | |
| 472 | 510 | } | |
| 511 | + | env.insert("GITHUB_ACTION".into(), action_name.clone()); | |
| 473 | 512 | let (ok, outputs, _) = self.run_script(&script, shell.as_deref(), working_directory.as_deref(), &env, timeout); | |
| 474 | 513 | (ok, outputs) | |
| 475 | 514 | } | |
| 476 | 515 | } else if let Some(uses) = step.get("uses").map(expr::to_text) { | |
| 477 | − | let with: BTreeMap<String, String> = match step.get("with") { | |
| 478 | − | Some(Value::Object(with)) => with | |
| 479 | − | .iter() | |
| 480 | − | .map(|(k, v)| { | |
| 481 | − | let value = self.with_scope(&contexts, |scope| expr::interpolate_value(v, scope)).unwrap_or(Value::Null); | |
| 482 | − | (k.clone(), expr::to_text(&value)) | |
| 483 | − | }) | |
| 484 | − | .collect(), | |
| 485 | − | _ => BTreeMap::new(), | |
| 486 | − | }; | |
| 487 | − | self.uses(&uses, &with, &env, frame, &title, id.as_deref(), timeout) | |
| 516 | + | let with = self.step_with(step, &contexts); | |
| 517 | + | self.uses(&uses, &with, &env, frame, &title, &action_name, timeout) | |
| 488 | 518 | } else { | |
| 489 | 519 | self.log.line("##[error]A step needs `run` or `uses`."); | |
| 490 | 520 | (false, BTreeMap::new()) | |
| ⋯ | |||
| 531 | 561 | conclusion == "success" | |
| 532 | 562 | } | |
| 533 | 563 | ||
| 564 | + | /// A step's own `env` over `base`, read with `contexts`. | |
| 565 | + | fn step_env(&self, step: &Map<String, Value>, contexts: &Map<String, Value>, mut base: BTreeMap<String, String>) -> BTreeMap<String, String> { | |
| 566 | + | if let Some(Value::Object(step_env)) = step.get("env") { | |
| 567 | + | for (name, value) in step_env { | |
| 568 | + | let value = self.with_scope(contexts, |scope| expr::interpolate_value(value, scope)).unwrap_or(Value::Null); | |
| 569 | + | base.insert(name.clone(), expr::to_text(&value)); | |
| 570 | + | } | |
| 571 | + | } | |
| 572 | + | base | |
| 573 | + | } | |
| 574 | + | ||
| 575 | + | /// A `uses:` step's `with`, read with `contexts`. | |
| 576 | + | fn step_with(&self, step: &Map<String, Value>, contexts: &Map<String, Value>) -> BTreeMap<String, String> { | |
| 577 | + | match step.get("with") { | |
| 578 | + | Some(Value::Object(with)) => with | |
| 579 | + | .iter() | |
| 580 | + | .map(|(k, v)| { | |
| 581 | + | let value = self.with_scope(contexts, |scope| expr::interpolate_value(v, scope)).unwrap_or(Value::Null); | |
| 582 | + | (k.clone(), expr::to_text(&value)) | |
| 583 | + | }) | |
| 584 | + | .collect(), | |
| 585 | + | _ => BTreeMap::new(), | |
| 586 | + | } | |
| 587 | + | } | |
| 588 | + | ||
| 534 | 589 | fn report_steps(&self) { | |
| 535 | 590 | self.log.steps(&self.step_names); | |
| 536 | 591 | } | |
| ⋯ | |||
| 568 | 623 | vec!["-NoLogo".into(), "-NoProfile".into(), "-NonInteractive".into(), "-Command".into(), ". '{0}'".into()] | |
| 569 | 624 | } | |
| 570 | 625 | ||
| 626 | + | /// The name of a shell's program, without its folder or `.exe`. | |
| 627 | + | fn program_name(program: &str) -> &str { | |
| 628 | + | let name = program.rsplit(['/', '\\']).next().unwrap_or(program); | |
| 629 | + | name.strip_suffix(".exe").or_else(|| name.strip_suffix(".EXE")).unwrap_or(name) | |
| 630 | + | } | |
| 631 | + | ||
| 632 | + | /// The extension GitHub's runner gives a script, by its shell's program: | |
| 633 | + | /// none for a program it does not know. | |
| 634 | + | fn script_extension(program: &str) -> &'static str { | |
| 635 | + | match program_name(program) { | |
| 636 | + | "bash" | "sh" => "sh", | |
| 637 | + | "pwsh" | "powershell" => "ps1", | |
| 638 | + | "cmd" => "cmd", | |
| 639 | + | "python" => "py", | |
| 640 | + | _ => "", | |
| 641 | + | } | |
| 642 | + | } | |
| 643 | + | ||
| 644 | + | /// A script as GitHub's runner writes it: for PowerShell, stopping at the | |
| 645 | + | /// first error and exiting with the last program's exit code. | |
| 646 | + | fn script_contents(program: &str, script: &str) -> String { | |
| 647 | + | let newline = if cfg!(windows) { "\r\n" } else { "\n" }; | |
| 648 | + | match program_name(program) { | |
| 649 | + | "pwsh" | "powershell" => format!( | |
| 650 | + | "$ErrorActionPreference = 'stop'{newline}{script}{newline}if ((Test-Path -LiteralPath variable:\\LASTEXITCODE)) {{ exit $LASTEXITCODE }}" | |
| 651 | + | ), | |
| 652 | + | _ => script.to_owned(), | |
| 653 | + | } | |
| 654 | + | } | |
| 655 | + | ||
| 656 | + | /// Whether `program` is a file in one of `path`'s folders (or, on | |
| 657 | + | /// Windows, `program.exe`), without running it. | |
| 658 | + | fn on_path(program: &str, path: &str) -> bool { | |
| 659 | + | let names = if cfg!(windows) { vec![program.to_owned(), format!("{program}.exe")] } else { vec![program.to_owned()] }; | |
| 660 | + | path.split(paths::PATH_SEPARATOR).filter(|dir| !dir.is_empty()).any(|dir| names.iter().any(|name| Path::new(dir).join(name).is_file())) | |
| 661 | + | } | |
| 662 | + | ||
| 663 | + | impl Job { | |
| 664 | + | /// `shell: python`'s program: `python`, as on GitHub, where there is | |
| 665 | + | /// one, else `python3` (Debian has no `python`). | |
| 666 | + | fn python_program(&self) -> String { | |
| 667 | + | if let Some(container) = &self.container { | |
| 668 | + | return container.python.into(); | |
| 669 | + | } | |
| 670 | + | let path = std::iter::once(self.base_env.get("PATH").cloned().unwrap_or_default()); | |
| 671 | + | let path: Vec<String> = self.path_prepend.iter().cloned().chain(path).collect(); | |
| 672 | + | if on_path("python", &path.join(paths::PATH_SEPARATOR)) { "python".into() } else { "python3".into() } | |
| 673 | + | } | |
| 674 | + | } | |
| 675 | + | ||
| 571 | 676 | /// Whether `program` is on `PATH`. | |
| 572 | 677 | fn program_exists(program: &str) -> bool { | |
| 573 | 678 | Command::new(program) | |
| ⋯ | |||
| 615 | 720 | ||
| 616 | 721 | fn setup(mut spec: Value, api: Api) -> Result<Job> { | |
| 617 | 722 | let masks: Vec<String> = spec["masks"].as_array().map(|m| m.iter().filter_map(|v| v.as_str().map(str::to_owned)).collect()).unwrap_or_default(); | |
| 618 | − | let log = Log::new(api, masks); | |
| 723 | + | let mut log = Log::new(api, masks); | |
| 724 | + | // Variables GitHub sets that a g1t from before them does not send. | |
| 725 | + | default_variables(&mut spec); | |
| 619 | 726 | // On a self-hosted runner's own machine, GitHub's layout lives in a | |
| 620 | 727 | // folder of the runner's (paths.rs). | |
| 621 | 728 | for part in ["variables", "github"] { | |
| ⋯ | |||
| 624 | 731 | paths::relocate(&mut spec["contexts"]["runner"]); | |
| 625 | 732 | let workspace = paths::under_home(WORKSPACE); | |
| 626 | 733 | let temp = paths::under_home(TEMP); | |
| 734 | + | log.matchers.workspace = workspace.clone(); | |
| 627 | 735 | std::fs::create_dir_all(&workspace).context("could not make the workspace")?; | |
| 628 | 736 | std::fs::create_dir_all(&temp).context("could not make the temporary folder")?; | |
| 629 | 737 | std::fs::write(temp.join("event.json"), serde_json::to_string_pretty(&spec["event"])?)?; | |
| ⋯ | |||
| 633 | 741 | let docker_hosted = cfg!(target_os = "linux") | |
| 634 | 742 | && std::env::var("G1T_DOCKER").as_deref() == Ok("on") | |
| 635 | 743 | && spec["variables"]["RUNNER_ENVIRONMENT"].as_str() != Some("self-hosted"); | |
| 636 | − | // This process's environment, less what only it should see. | |
| 744 | + | // This process's environment, less what only it should see. `ImageOS` | |
| 745 | + | // and `ImageVersion` come with it when the machine's image sets them, | |
| 746 | + | // as GitHub's images do; nothing makes them up otherwise. | |
| 637 | 747 | let mut base_env: BTreeMap<String, String> = | |
| 638 | 748 | std::env::vars().filter(|(name, _)| !matches!(name.as_str(), "ACTIONS_TOKEN" | "ACTIONS_JOB" | "MODE") && !name.starts_with("G1T_")).collect(); | |
| 639 | 749 | base_env.insert("HOME".into(), std::env::var("HOME").unwrap_or_else(|_| "/home/node".into())); | |
| ⋯ | |||
| 669 | 779 | debug, | |
| 670 | 780 | last_node_outputs: BTreeMap::new(), | |
| 671 | 781 | last_node_state: BTreeMap::new(), | |
| 782 | + | pre_state: BTreeMap::new(), | |
| 672 | 783 | host_env, | |
| 673 | 784 | docker_hosted, | |
| 674 | 785 | network: None, | |
| ⋯ | |||
| 688 | 799 | Ok(job) | |
| 689 | 800 | } | |
| 690 | 801 | ||
| 802 | + | /// `RUNNER_WORKSPACE` and `GITHUB_REPOSITORY_OWNER_ID` (with | |
| 803 | + | /// `github.repository_owner_id`), when the spec does not have them: the | |
| 804 | + | /// folder the workspace is in, and the owner's id from the event. | |
| 805 | + | fn default_variables(spec: &mut Value) { | |
| 806 | + | if spec["variables"].is_object() && spec["variables"].get("RUNNER_WORKSPACE").is_none() { | |
| 807 | + | spec["variables"]["RUNNER_WORKSPACE"] = json!(g1t_actions::events::RUNNER_WORKSPACE); | |
| 808 | + | } | |
| 809 | + | let owner_id = g1t_actions::events::owner_id(&spec["event"]); | |
| 810 | + | if let Some(owner_id) = owner_id.filter(|id| !id.is_empty()) { | |
| 811 | + | if spec["variables"].is_object() && spec["variables"].get("GITHUB_REPOSITORY_OWNER_ID").and_then(Value::as_str).is_none_or(str::is_empty) { | |
| 812 | + | spec["variables"]["GITHUB_REPOSITORY_OWNER_ID"] = json!(owner_id); | |
| 813 | + | } | |
| 814 | + | if spec["github"].is_object() && spec["github"].get("repository_owner_id").and_then(Value::as_str).is_none_or(str::is_empty) { | |
| 815 | + | spec["github"]["repository_owner_id"] = json!(owner_id); | |
| 816 | + | } | |
| 817 | + | } | |
| 818 | + | } | |
| 819 | + | ||
| 691 | 820 | /// The job's `defaults.run`, its own over the workflow's. | |
| 692 | 821 | fn run_defaults(spec: &Value) -> Map<String, Value> { | |
| 693 | 822 | let mut defaults = spec["workflow"]["defaults"]["run"].as_object().cloned().unwrap_or_default(); | |
| ⋯ | |||
| 698 | 827 | } | |
| 699 | 828 | ||
| 700 | 829 | fn run_job(job: &mut Job) { | |
| 701 | − | let steps: Vec<Map<String, Value>> = job.spec["spec"]["steps"] | |
| 830 | + | let mut steps: Vec<Map<String, Value>> = job.spec["spec"]["steps"] | |
| 702 | 831 | .as_array() | |
| 703 | 832 | .map(|steps| steps.iter().filter_map(|s| s.as_object().cloned()).collect()) | |
| 704 | 833 | .unwrap_or_default(); | |
| 834 | + | name_steps(&mut steps); | |
| 705 | 835 | let defaults = run_defaults(&job.spec); | |
| 706 | 836 | ||
| 707 | 837 | // Step names as they read before anything has run. | |
| ⋯ | |||
| 751 | 881 | } | |
| 752 | 882 | } | |
| 753 | 883 | // Actions first, then containers, as GitHub's "Set up job" and | |
| 754 | − | // "Initialize containers". | |
| 755 | − | let containers_started = job.prepare_actions(&steps) && job.start_containers(); | |
| 884 | + | // "Initialize containers". Each action's `pre` (read from the actions | |
| 885 | + | // just downloaded) is a step of its own, "Pre <step>", before the | |
| 886 | + | // job's first step. | |
| 887 | + | let prepared = job.prepare_actions(&steps); | |
| 888 | + | let pres = if prepared { job.pre_steps(&steps) } else { Vec::new() }; | |
| 889 | + | if !pres.is_empty() { | |
| 890 | + | let mut names: Vec<String> = pres.iter().map(|(index, _)| format!("Pre {}", job.step_names[*index])).collect(); | |
| 891 | + | names.extend(job.step_names.iter().cloned()); | |
| 892 | + | job.step_names = names; | |
| 893 | + | job.report_steps(); | |
| 894 | + | } | |
| 895 | + | let containers_started = prepared && job.start_containers(); | |
| 756 | 896 | job.log.flush(); | |
| 757 | 897 | ||
| 758 | 898 | let mut frame = Frame::default(); | |
| ⋯ | |||
| 762 | 902 | job.log.step_state(index as u32 + 1, name, "completed", Some("skipped")); | |
| 763 | 903 | } | |
| 764 | 904 | } | |
| 905 | + | for (index, (_, pre)) in pres.iter().enumerate().filter(|_| containers_started) { | |
| 906 | + | let number = index as u32 + 1; | |
| 907 | + | let name = job.step_names[index].clone(); | |
| 908 | + | if report::cancelled() { | |
| 909 | + | report::take_cancel(); | |
| 910 | + | } | |
| 911 | + | if !job.pre_runs(pre) { | |
| 912 | + | job.log.step_state(number, &name, "completed", Some("skipped")); | |
| 913 | + | continue; | |
| 914 | + | } | |
| 915 | + | job.log.step(number); | |
| 916 | + | job.log.step_state(number, &name, "in_progress", None); | |
| 917 | + | let ok = job.run_pre(pre); | |
| 918 | + | if !ok { | |
| 919 | + | job.failed = true; | |
| 920 | + | } | |
| 921 | + | let conclusion = if ok { | |
| 922 | + | "success" | |
| 923 | + | } else if report::cancelled() { | |
| 924 | + | "cancelled" | |
| 925 | + | } else { | |
| 926 | + | "failure" | |
| 927 | + | }; | |
| 928 | + | job.log.step_state(number, &name, "completed", Some(conclusion)); | |
| 929 | + | } | |
| 930 | + | let offset = pres.len() as u32; | |
| 765 | 931 | for (index, step) in steps.iter().enumerate().filter(|_| containers_started) { | |
| 766 | − | job.step(&mut frame, step, index as u32 + 1, true, &defaults); | |
| 932 | + | job.step(&mut frame, step, offset + index as u32 + 1, true, &defaults); | |
| 767 | 933 | job.log_docker_notes(); | |
| 768 | 934 | if job.job_remaining().is_zero() { | |
| 769 | 935 | job.log.line("##[error]The job ran past its time limit."); | |
| ⋯ | |||
| 787 | 953 | job.log.step(number); | |
| 788 | 954 | job.log.step_state(number, &post.name, "in_progress", None); | |
| 789 | 955 | let ok = match &post.run { | |
| 790 | − | PostRun::Node { action_dir, script } => job.run_node(action_dir, script, &post.env), | |
| 956 | + | PostRun::Node { action_dir, script, using } => job.run_node(action_dir, script, &post.env, using), | |
| 791 | 957 | PostRun::CacheSave { key, paths, version } => job.cache_save(key, paths, version), | |
| 792 | 958 | PostRun::Docker(run) => job.run_docker(run).0, | |
| 793 | 959 | }; | |
| ⋯ | |||
| 883 | 1049 | } | |
| 884 | 1050 | ||
| 885 | 1051 | #[test] | |
| 1052 | + | fn scripts_get_githubs_extension_for_their_shell() { | |
| 1053 | + | assert_eq!(script_extension("bash"), "sh"); | |
| 1054 | + | assert_eq!(script_extension("/usr/bin/sh"), "sh"); | |
| 1055 | + | assert_eq!(script_extension("pwsh"), "ps1"); | |
| 1056 | + | assert_eq!(script_extension("C:\\Windows\\powershell.exe"), "ps1"); | |
| 1057 | + | assert_eq!(script_extension("cmd"), "cmd"); | |
| 1058 | + | assert_eq!(script_extension("python"), "py"); | |
| 1059 | + | // A program GitHub does not know: no extension. | |
| 1060 | + | assert_eq!(script_extension("perl"), ""); | |
| 1061 | + | assert_eq!(script_extension("python3"), ""); | |
| 1062 | + | } | |
| 1063 | + | ||
| 1064 | + | #[test] | |
| 1065 | + | fn powershell_scripts_stop_at_errors_and_keep_the_exit_code() { | |
| 1066 | + | let newline = if cfg!(windows) { "\r\n" } else { "\n" }; | |
| 1067 | + | assert_eq!( | |
| 1068 | + | script_contents("pwsh", "Write-Output hi"), | |
| 1069 | + | format!("$ErrorActionPreference = 'stop'{newline}Write-Output hi{newline}if ((Test-Path -LiteralPath variable:\\LASTEXITCODE)) {{ exit $LASTEXITCODE }}") | |
| 1070 | + | ); | |
| 1071 | + | assert!(script_contents("powershell.exe", "x").starts_with("$ErrorActionPreference = 'stop'")); | |
| 1072 | + | assert_eq!(script_contents("bash", "echo hi"), "echo hi"); | |
| 1073 | + | } | |
| 1074 | + | ||
| 1075 | + | #[test] | |
| 1076 | + | fn programs_are_found_on_path_without_running_them() { | |
| 1077 | + | let dir = std::env::temp_dir().join(format!("g1t-on-path-{}", std::process::id())); | |
| 1078 | + | std::fs::create_dir_all(&dir).unwrap(); | |
| 1079 | + | std::fs::write(dir.join("python"), "").unwrap(); | |
| 1080 | + | let path = format!("/nowhere{}{}", paths::PATH_SEPARATOR, dir.display()); | |
| 1081 | + | assert!(on_path("python", &path)); | |
| 1082 | + | assert!(!on_path("python9", &path)); | |
| 1083 | + | assert!(!on_path("python", "")); | |
| 1084 | + | let _ = std::fs::remove_dir_all(&dir); | |
| 1085 | + | } | |
| 1086 | + | ||
| 1087 | + | #[test] | |
| 1088 | + | fn steps_are_named_for_github_action_as_github_names_them() { | |
| 1089 | + | let mut steps: Vec<Map<String, Value>> = serde_json::from_value(json!([ | |
| 1090 | + | { "run": "a" }, | |
| 1091 | + | { "run": "b" }, | |
| 1092 | + | { "id": "build", "run": "c" }, | |
| 1093 | + | { "uses": "actions/setup-node@v4" }, | |
| 1094 | + | { "uses": "actions/cache/restore@v4" }, | |
| 1095 | + | { "uses": "actions/setup-node@v4" }, | |
| 1096 | + | { "uses": "./.github/actions/x" }, | |
| 1097 | + | { "run": "d" }, | |
| 1098 | + | { "id": "__run_4", "run": "e" }, | |
| 1099 | + | { "run": "f" }, | |
| 1100 | + | ])) | |
| 1101 | + | .unwrap(); | |
| 1102 | + | name_steps(&mut steps); | |
| 1103 | + | let names: Vec<String> = steps.iter().map(|s| expr::to_text(&s[ACTION_NAME])).collect(); | |
| 1104 | + | assert_eq!( | |
| 1105 | + | names, | |
| 1106 | + | ["__run", "__run_2", "build", "__actions_setup-node", "__actions_cache", "__actions_setup-node_2", "__self", "__run_3", "__run_4", "__run_5"] | |
| 1107 | + | ); | |
| 1108 | + | } | |
| 1109 | + | ||
| 1110 | + | #[test] | |
| 1111 | + | fn variables_a_spec_lacks_are_filled_in() { | |
| 1112 | + | let mut spec = json!({ | |
| 1113 | + | "variables": { "CI": "true" }, | |
| 1114 | + | "github": { "repository_owner_id": "" }, | |
| 1115 | + | "event": { "repository": { "owner": { "id": 7 } } }, | |
| 1116 | + | }); | |
| 1117 | + | default_variables(&mut spec); | |
| 1118 | + | assert_eq!(spec["variables"]["RUNNER_WORKSPACE"], "/home/runner/work"); | |
| 1119 | + | assert_eq!(spec["variables"]["GITHUB_REPOSITORY_OWNER_ID"], "7"); | |
| 1120 | + | assert_eq!(spec["github"]["repository_owner_id"], "7"); | |
| 1121 | + | // What g1t sent stands. | |
| 1122 | + | let mut spec = json!({ "variables": { "RUNNER_WORKSPACE": "/w", "GITHUB_REPOSITORY_OWNER_ID": "9" }, "github": {}, "event": {} }); | |
| 1123 | + | default_variables(&mut spec); | |
| 1124 | + | assert_eq!(spec["variables"]["RUNNER_WORKSPACE"], "/w"); | |
| 1125 | + | assert_eq!(spec["variables"]["GITHUB_REPOSITORY_OWNER_ID"], "9"); | |
| 1126 | + | } | |
| 1127 | + | ||
| 1128 | + | #[test] | |
| 886 | 1129 | fn a_steps_timeout_is_its_own_or_the_nearer_one_around_it() { | |
| 887 | 1130 | let now = Instant::now(); | |
| 888 | 1131 | // No timeout-minutes anywhere: only the job's deadline applies. | |
| 23 | 23 | pub(crate) stopped: Option<String>, | |
| 24 | 24 | /// Whether `::debug::` lines are shown (`ACTIONS_STEP_DEBUG`). | |
| 25 | 25 | pub(crate) debug: bool, | |
| 26 | + | /// `::echo::on` or `::echo::off`: whether command lines are shown as | |
| 27 | + | /// well as acted on. Until a step says, they are with debug logging. | |
| 28 | + | pub(crate) echo: Option<bool>, | |
| 26 | 29 | } | |
| 27 | 30 | ||
| 28 | 31 | /// `%25`, `%0D`, `%0A`, and in properties `%3A` and `%2C`, as the | |
| ⋯ | |||
| 65 | 68 | self.stopped = None; | |
| 66 | 69 | return None; | |
| 67 | 70 | } | |
| 68 | − | return Some(line.to_owned()); | |
| 71 | + | return Some(output(line, log)); | |
| 69 | 72 | } | |
| 70 | 73 | let Some((name, properties, data)) = parse_command(line) else { | |
| 71 | − | return Some(line.to_owned()); | |
| 74 | + | return Some(output(line, log)); | |
| 72 | 75 | }; | |
| 76 | + | // Echoed as it reads, before it is acted on; `::add-mask::` never is. | |
| 77 | + | let known = matches!( | |
| 78 | + | name.as_str(), | |
| 79 | + | "add-mask" | "error" | "warning" | "notice" | "group" | "endgroup" | "debug" | "set-output" | "save-state" | "stop-commands" | "echo" | "add-matcher" | "remove-matcher" | "add-path" | "set-env" | |
| 80 | + | ); | |
| 81 | + | if known && name != "add-mask" && self.echo.unwrap_or(self.debug) { | |
| 82 | + | log.line(line); | |
| 83 | + | } | |
| 73 | 84 | match name.as_str() { | |
| 74 | 85 | "add-mask" => { | |
| 75 | 86 | if !data.trim().is_empty() { | |
| ⋯ | |||
| 79 | 90 | } | |
| 80 | 91 | "error" | "warning" | "notice" => { | |
| 81 | 92 | log.annotation(&name, &data, &properties); | |
| 82 | − | let label = match name.as_str() { | |
| 83 | − | "error" => "Error", | |
| 84 | − | "warning" => "Warning", | |
| 85 | − | _ => "Notice", | |
| 86 | − | }; | |
| 87 | − | Some(format!("##[{name}]{label}: {data}")) | |
| 93 | + | Some(issue_line(&name, &data)) | |
| 88 | 94 | } | |
| 89 | 95 | "group" => Some(format!("##[group]{data}")), | |
| 90 | 96 | "endgroup" => Some("##[endgroup]".to_owned()), | |
| ⋯ | |||
| 105 | 111 | self.stopped = Some(data); | |
| 106 | 112 | None | |
| 107 | 113 | } | |
| 108 | − | "echo" => None, | |
| 114 | + | "echo" => { | |
| 115 | + | match data.trim().to_ascii_lowercase().as_str() { | |
| 116 | + | "on" => self.echo = Some(true), | |
| 117 | + | "off" => self.echo = Some(false), | |
| 118 | + | _ => return Some(format!("##[error]Invalid echo command value. Possible values can be: 'on', 'off'. Current value is: '{}'.", data.trim())), | |
| 119 | + | } | |
| 120 | + | None | |
| 121 | + | } | |
| 122 | + | "add-matcher" => { | |
| 123 | + | let path = matcher_path(data.trim(), &log.matchers.workspace); | |
| 124 | + | let read = std::fs::read_to_string(&path).map_err(|e| format!("{}: {e}", path.display())).and_then(|text| super::matchers::parse(&text)); | |
| 125 | + | match read { | |
| 126 | + | Ok(matchers) => { | |
| 127 | + | log.matchers.add(matchers); | |
| 128 | + | None | |
| 129 | + | } | |
| 130 | + | Err(problem) => Some(format!("##[error]Unable to process command '{line}' successfully.\n##[error]{problem}")), | |
| 131 | + | } | |
| 132 | + | } | |
| 133 | + | "remove-matcher" => { | |
| 134 | + | match properties.get("owner").and_then(Value::as_str).filter(|o| !o.trim().is_empty()) { | |
| 135 | + | Some(owner) => log.matchers.remove(owner.trim()), | |
| 136 | + | // No owner: every matcher in the file named. | |
| 137 | + | None => { | |
| 138 | + | let path = matcher_path(data.trim(), &log.matchers.workspace); | |
| 139 | + | match std::fs::read_to_string(&path).map_err(|e| format!("{}: {e}", path.display())).and_then(|text| super::matchers::parse(&text)) { | |
| 140 | + | Ok(matchers) => matchers.iter().for_each(|m| log.matchers.remove(&m.owner)), | |
| 141 | + | Err(problem) => return Some(format!("##[error]Unable to process command '{line}' successfully.\n##[error]{problem}")), | |
| 142 | + | } | |
| 143 | + | } | |
| 144 | + | } | |
| 145 | + | None | |
| 146 | + | } | |
| 109 | 147 | "add-path" | "set-env" => Some(format!( | |
| 110 | 148 | "##[error]The `{name}` command is disabled, as on GitHub. Write to the file in $GITHUB_{} instead.", | |
| 111 | 149 | if name == "add-path" { "PATH" } else { "ENV" } | |
| 112 | 150 | )), | |
| 113 | − | _ => Some(line.to_owned()), | |
| 151 | + | _ => Some(output(line, log)), | |
| 152 | + | } | |
| 153 | + | } | |
| 154 | + | } | |
| 155 | + | ||
| 156 | + | /// How an annotation shows in the log. | |
| 157 | + | fn issue_line(level: &str, message: &str) -> String { | |
| 158 | + | let label = match level { | |
| 159 | + | "error" => "Error", | |
| 160 | + | "warning" => "Warning", | |
| 161 | + | _ => "Notice", | |
| 162 | + | }; | |
| 163 | + | format!("##[{level}]{label}: {message}") | |
| 164 | + | } | |
| 165 | + | ||
| 166 | + | /// A line of output, as the log shows it: as it is, or, when a problem | |
| 167 | + | /// matcher reads an issue in it, the issue, annotated. | |
| 168 | + | fn output(line: &str, log: &mut Log) -> String { | |
| 169 | + | match log.matchers.issue(line) { | |
| 170 | + | Some(issue) => { | |
| 171 | + | log.annotation(issue.level, &issue.message, &issue.properties); | |
| 172 | + | issue_line(issue.level, &issue.message) | |
| 114 | 173 | } | |
| 174 | + | None => line.to_owned(), | |
| 115 | 175 | } | |
| 116 | 176 | } | |
| 117 | 177 | ||
| 178 | + | /// A matcher file's path: relative to the workspace, and a Docker action's | |
| 179 | + | /// `/github/workspace` read as the workspace it is. | |
| 180 | + | fn matcher_path(given: &str, workspace: &std::path::Path) -> std::path::PathBuf { | |
| 181 | + | if let Some(rest) = given.strip_prefix("/github/workspace") { | |
| 182 | + | return workspace.join(rest.trim_start_matches('/')); | |
| 183 | + | } | |
| 184 | + | let path = std::path::Path::new(given); | |
| 185 | + | if path.is_absolute() { path.to_path_buf() } else { workspace.join(path) } | |
| 186 | + | } | |
| 187 | + | ||
| 118 | 188 | /// How a process ended. | |
| 119 | 189 | pub(crate) enum Ended { | |
| 120 | 190 | Exited(i32), | |
| ⋯ | |||
| 316 | 386 | assert!(text.contains("cleaned up"), "{text}"); | |
| 317 | 387 | } | |
| 318 | 388 | ||
| 389 | + | /// `::echo::on` shows command lines until `::echo::off`, which is | |
| 390 | + | /// itself shown; `::add-mask::` never is. | |
| 391 | + | #[cfg(unix)] | |
| 392 | + | #[test] | |
| 393 | + | fn echo_shows_command_lines_while_on() { | |
| 394 | + | use super::Commands; | |
| 395 | + | use crate::actions::report::{Api, Log}; | |
| 396 | + | ||
| 397 | + | let mut log = Log::new(Api::offline(), Vec::new()); | |
| 398 | + | let mut commands = Commands::default(); | |
| 399 | + | for line in ["::set-output name=a::1", "::echo::on", "::save-state name=b::2", "::add-mask::s3cret", "::echo::off", "::save-state name=c::3"] { | |
| 400 | + | if let Some(shown) = commands.handle(line, &mut log) { | |
| 401 | + | log.line(&shown); | |
| 402 | + | } | |
| 403 | + | } | |
| 404 | + | let text = log.buffered(); | |
| 405 | + | assert_eq!(text, "::save-state name=b::2\n::echo::off\n"); | |
| 406 | + | assert_eq!((commands.outputs.len(), commands.state.len()), (1, 2)); | |
| 407 | + | // Debug logging echoes until a step says otherwise. | |
| 408 | + | let mut log = Log::new(Api::offline(), Vec::new()); | |
| 409 | + | let mut commands = Commands { debug: true, ..Commands::default() }; | |
| 410 | + | commands.handle("::set-output name=a::1", &mut log); | |
| 411 | + | assert!(log.buffered().contains("::set-output name=a::1")); | |
| 412 | + | assert!(commands.handle("::echo::maybe", &mut log).unwrap().contains("Invalid echo command value")); | |
| 413 | + | } | |
| 414 | + | ||
| 415 | + | /// A matcher added by one line reads the lines after it, until removed. | |
| 416 | + | #[cfg(unix)] | |
| 417 | + | #[test] | |
| 418 | + | fn matchers_are_added_and_removed_by_command() { | |
| 419 | + | use super::Commands; | |
| 420 | + | use crate::actions::report::{Api, Log}; | |
| 421 | + | ||
| 422 | + | let dir = std::env::temp_dir().join(format!("g1t-add-matcher-{}", std::process::id())); | |
| 423 | + | std::fs::create_dir_all(&dir).unwrap(); | |
| 424 | + | std::fs::write( | |
| 425 | + | dir.join("go.json"), | |
| 426 | + | r#"{"problemMatcher": [{"owner": "go", "pattern": [{"regexp": "^\\s*(.+\\.go):(?:(\\d+):(\\d+):)? (.*)", "file": 1, "line": 2, "column": 3, "message": 4}]}]}"#, | |
| 427 | + | ) | |
| 428 | + | .unwrap(); | |
| 429 | + | let mut log = Log::new(Api::offline(), Vec::new()); | |
| 430 | + | log.matchers.workspace = dir.clone(); | |
| 431 | + | let mut commands = Commands::default(); | |
| 432 | + | assert_eq!(commands.handle("main.go:3:1: undefined: x", &mut log).unwrap(), "main.go:3:1: undefined: x"); | |
| 433 | + | assert!(commands.handle("::add-matcher::go.json", &mut log).is_none()); | |
| 434 | + | assert_eq!(commands.handle("main.go:3:1: undefined: x", &mut log).unwrap(), "##[error]Error: undefined: x"); | |
| 435 | + | assert!(commands.handle("::remove-matcher owner=go::", &mut log).is_none()); | |
| 436 | + | assert_eq!(commands.handle("main.go:3:1: undefined: x", &mut log).unwrap(), "main.go:3:1: undefined: x"); | |
| 437 | + | assert!(commands.handle("::add-matcher::missing.json", &mut log).unwrap().contains("Unable to process command")); | |
| 438 | + | let _ = std::fs::remove_dir_all(&dir); | |
| 439 | + | } | |
| 440 | + | ||
| 319 | 441 | #[test] | |
| 320 | 442 | fn commands_are_read_with_their_properties() { | |
| 321 | 443 | let (name, properties, data) = parse_command("::error file=app.js,line=10,title=Bad%3A thing::Something%0Abroke").unwrap(); | |
| 167 | 167 | last: Instant, | |
| 168 | 168 | /// When anything was last sent, for the ping. | |
| 169 | 169 | sent: Instant, | |
| 170 | + | /// The problem matchers steps added, which last the whole job. | |
| 171 | + | pub(crate) matchers: super::matchers::Matchers, | |
| 170 | 172 | /// Everything sent so far, so a test reads the whole log however its | |
| 171 | 173 | /// batches fell (`buffered`). | |
| 172 | 174 | #[cfg(test)] | |
| ⋯ | |||
| 184 | 186 | buffer: String::new(), | |
| 185 | 187 | last: Instant::now(), | |
| 186 | 188 | sent: Instant::now(), | |
| 189 | + | matchers: super::matchers::Matchers::default(), | |
| 187 | 190 | #[cfg(test)] | |
| 188 | 191 | flushed: String::new(), | |
| 189 | 192 | } | |
| ⋯ | |||
| 282 | 285 | self.api.report(json!({ "kind": "summary", "step": self.step, "markdown": markdown })); | |
| 283 | 286 | } | |
| 284 | 287 | ||
| 288 | + | /// An annotation, from `::error::` and its kin or a problem matcher: | |
| 289 | + | /// its `file`, `line`, `col`, `endLine` and `endColumn` as GitHub's | |
| 290 | + | /// commands spell them, sent as `file`, `line`, `column`, `end_line` | |
| 291 | + | /// and `end_column`. | |
| 285 | 292 | pub(crate) fn annotation(&mut self, level: &str, message: &str, properties: &serde_json::Map<String, Value>) { | |
| 293 | + | let report = self.annotation_report(level, message, properties); | |
| 294 | + | self.api.report(report); | |
| 295 | + | } | |
| 296 | + | ||
| 297 | + | fn annotation_report(&self, level: &str, message: &str, properties: &serde_json::Map<String, Value>) -> Value { | |
| 286 | 298 | let message = self.mask(message); | |
| 287 | 299 | let title = properties.get("title").and_then(Value::as_str).map(|title| self.mask(title)); | |
| 288 | − | self.api.report(json!({ | |
| 300 | + | let number = |keys: &[&str]| keys.iter().find_map(|key| properties.get(*key).and_then(Value::as_str)).and_then(|n| n.trim().parse::<u32>().ok()); | |
| 301 | + | json!({ | |
| 289 | 302 | "kind": "annotation", | |
| 290 | 303 | "level": level, | |
| 291 | 304 | "message": message, | |
| 292 | 305 | "title": title, | |
| 293 | 306 | "file": properties.get("file"), | |
| 294 | − | "line": properties.get("line").and_then(|l| l.as_str()).and_then(|l| l.parse::<u32>().ok()), | |
| 295 | − | })); | |
| 307 | + | "line": number(&["line"]), | |
| 308 | + | "column": number(&["col"]), | |
| 309 | + | "end_line": number(&["endLine"]), | |
| 310 | + | "end_column": number(&["endColumn"]), | |
| 311 | + | }) | |
| 296 | 312 | } | |
| 297 | 313 | ||
| 298 | 314 | pub(crate) fn done(&mut self, conclusion: &str, outputs: &serde_json::Map<String, Value>, reason: Option<&str>) { | |
| ⋯ | |||
| 367 | 383 | } | |
| 368 | 384 | ||
| 369 | 385 | #[test] | |
| 386 | + | fn annotations_carry_their_columns_and_end() { | |
| 387 | + | let log = log(&["hidden"]); | |
| 388 | + | let properties: serde_json::Map<String, Value> = | |
| 389 | + | serde_json::from_value(json!({ "file": "a.rs", "line": "3", "col": "7", "endLine": "4", "endColumn": "2", "title": "t hidden" })).unwrap(); | |
| 390 | + | let report = log.annotation_report("error", "bad hidden", &properties); | |
| 391 | + | assert_eq!(report["line"], 3); | |
| 392 | + | assert_eq!(report["column"], 7); | |
| 393 | + | assert_eq!(report["end_line"], 4); | |
| 394 | + | assert_eq!(report["end_column"], 2); | |
| 395 | + | assert_eq!(report["title"], "t ***"); | |
| 396 | + | assert_eq!(report["message"], "bad ***"); | |
| 397 | + | let bare = log.annotation_report("notice", "x", &serde_json::Map::new()); | |
| 398 | + | assert!(bare["column"].is_null() && bare["file"].is_null()); | |
| 399 | + | } | |
| 400 | + | ||
| 401 | + | #[test] | |
| 370 | 402 | fn added_masks_cover_every_form() { | |
| 371 | 403 | let mut log = log(&[]); | |
| 372 | 404 | log.add_mask("line one\nline two"); | |
| 381 | 381 | ready | |
| 382 | 382 | } | |
| 383 | 383 | ||
| 384 | − | /// Runs a JavaScript file of an action with Node. | |
| 385 | − | pub(crate) fn run_node(&mut self, action_dir: &Path, script: &str, env: &BTreeMap<String, String>) -> bool { | |
| 384 | + | /// Runs a JavaScript file of an action with Node: the one its | |
| 385 | + | /// `runs.using` names where the machine has it (`node_program`). | |
| 386 | + | pub(crate) fn run_node(&mut self, action_dir: &Path, script: &str, env: &BTreeMap<String, String>, using: &str) -> bool { | |
| 386 | 387 | let id = format!("node{}", super::rand_id()); | |
| 387 | 388 | let Ok(files) = StepFiles::new(&self.temp, &id) else { return false }; | |
| 388 | 389 | let full = self.process_env(env, &files); | |
| ⋯ | |||
| 396 | 397 | let command = match in_container { | |
| 397 | 398 | Some(command) => command, | |
| 398 | 399 | None => { | |
| 399 | − | let mut command = Command::new("node"); | |
| 400 | + | let node = node_program(using, |name| std::env::var(name).ok(), |path| path.is_file()); | |
| 401 | + | let mut command = Command::new(node); | |
| 400 | 402 | command.arg(&script_path).current_dir(&self.workspace).env_clear().envs(full); | |
| 401 | 403 | command | |
| 402 | 404 | } | |
| ⋯ | |||
| 431 | 433 | env: &BTreeMap<String, String>, | |
| 432 | 434 | frame: &Frame, | |
| 433 | 435 | title: &str, | |
| 434 | − | id: Option<&str>, | |
| 436 | + | action_name: &str, | |
| 435 | 437 | _timeout: Duration, | |
| 436 | 438 | ) -> (bool, BTreeMap<String, String>) { | |
| 437 | 439 | let uses = uses.trim(); | |
| ⋯ | |||
| 439 | 441 | // `with.args` and `with.entrypoint` are the container's; every | |
| 440 | 442 | // input is also an `INPUT_` variable, as on GitHub. | |
| 441 | 443 | let mut step_env = env.clone(); | |
| 444 | + | step_env.insert("GITHUB_ACTION".into(), action_name.to_owned()); | |
| 442 | 445 | for (input, value) in with { | |
| 443 | 446 | step_env.insert(format!("INPUT_{}", input.replace(' ', "_").to_ascii_uppercase()), value.clone()); | |
| 444 | 447 | } | |
| ⋯ | |||
| 466 | 469 | "actions/cache/save" => return self.cache_save_now(with), | |
| 467 | 470 | _ => {} | |
| 468 | 471 | } | |
| 469 | − | let source = if let Some(local) = name.strip_prefix("./") { | |
| 470 | − | // A repository moved from GitHub renamed `.github` to `.g1t`, but | |
| 471 | − | // its workflows still say `./.github/actions/…`. | |
| 472 | − | let mut dir = self.workspace.join(local); | |
| 473 | − | if let Some(rest) = local.strip_prefix(".github/") | |
| 474 | − | && !dir.exists() | |
| 475 | − | { | |
| 476 | − | dir = self.workspace.join(".g1t").join(rest); | |
| 477 | − | } | |
| 478 | − | Source::Local(dir) | |
| 479 | − | } else { | |
| 480 | − | let mut parts = name.splitn(3, '/'); | |
| 481 | − | let (Some(owner), Some(repo)) = (parts.next(), parts.next()) else { | |
| 482 | − | self.log.line(&format!("##[error]`{uses}` is not an action: use owner/repo@ref, owner/repo/path@ref, or ./path.")); | |
| 483 | − | return (false, BTreeMap::new()); | |
| 484 | − | }; | |
| 485 | − | if git_ref.is_empty() { | |
| 486 | − | self.log.line(&format!("##[error]`{uses}` needs a version, such as @v4.")); | |
| 487 | − | return (false, BTreeMap::new()); | |
| 488 | − | } | |
| 489 | − | Source::GitHub { | |
| 490 | − | owner: owner.to_owned(), | |
| 491 | − | repo: repo.to_owned(), | |
| 492 | − | path: parts.next().unwrap_or_default().to_owned(), | |
| 493 | − | git_ref: git_ref.to_owned(), | |
| 494 | − | } | |
| 495 | − | }; | |
| 496 | − | let (dir, repository) = match &source { | |
| 497 | − | Source::Local(dir) => (dir.clone(), String::new()), | |
| 498 | − | Source::GitHub { owner, repo, path, git_ref } => match self.fetch_remote_action(owner, repo, git_ref) { | |
| 499 | − | Some(root) => (if path.is_empty() { root } else { root.join(path) }, format!("{owner}/{repo}")), | |
| 500 | − | None => return (false, BTreeMap::new()), | |
| 501 | − | }, | |
| 502 | − | }; | |
| 503 | − | let manifest = ["action.yml", "action.yaml"].iter().map(|f| dir.join(f)).find(|p| p.exists()); | |
| 504 | − | let Some(manifest) = manifest else { | |
| 505 | − | self.log.line(&format!("##[error]`{uses}` has no action.yml.")); | |
| 472 | + | let Some(found) = self.resolve_action(uses) else { | |
| 506 | 473 | return (false, BTreeMap::new()); | |
| 507 | 474 | }; | |
| 508 | − | let action = match std::fs::read_to_string(&manifest).ok().and_then(|text| serde_yaml::from_str::<serde_yaml::Value>(&text).ok()) { | |
| 509 | − | Some(yaml) => yaml_to_json(&yaml), | |
| 510 | − | None => { | |
| 511 | − | self.log.line(&format!("##[error]`{uses}`: its action.yml does not read.")); | |
| 512 | − | return (false, BTreeMap::new()); | |
| 513 | − | } | |
| 514 | − | }; | |
| 475 | + | let Action { dir, local, manifest: action, .. } = &found; | |
| 515 | 476 | ||
| 516 | 477 | // Inputs: what the step gives, else the action's defaults. | |
| 517 | 478 | let env_context = env.clone(); | |
| 518 | 479 | let contexts = self.contexts_for(frame, &env_context); | |
| 519 | − | let mut inputs: BTreeMap<String, String> = BTreeMap::new(); | |
| 520 | − | if let Some(Value::Object(declared)) = action.get("inputs") { | |
| 521 | − | for (input, spec) in declared { | |
| 522 | − | let given = with.iter().find(|(k, _)| k.eq_ignore_ascii_case(input)).map(|(_, v)| v.clone()); | |
| 523 | − | let value = match given { | |
| 524 | − | Some(value) => value, | |
| 525 | − | None => match spec.get("default") { | |
| 526 | − | Some(default) => { | |
| 527 | − | let default = self.with_scope(&contexts, |scope| expr::interpolate_value(default, scope)).unwrap_or(Value::Null); | |
| 528 | − | expr::to_text(&default) | |
| 529 | − | } | |
| 530 | − | None => String::new(), | |
| 531 | − | }, | |
| 532 | − | }; | |
| 533 | − | inputs.insert(input.clone(), value); | |
| 534 | − | } | |
| 535 | − | } | |
| 536 | − | for (key, value) in with { | |
| 537 | − | if !inputs.keys().any(|k| k.eq_ignore_ascii_case(key)) { | |
| 538 | − | inputs.insert(key.clone(), value.clone()); | |
| 539 | − | } | |
| 540 | − | } | |
| 480 | + | let inputs = self.action_inputs(action, with, &contexts, true); | |
| 541 | 481 | ||
| 542 | 482 | let runs = action.get("runs").cloned().unwrap_or(Value::Null); | |
| 543 | 483 | let using = runs.get("using").map(expr::to_text).unwrap_or_default().to_ascii_lowercase(); | |
| 544 | − | let mut step_env = env.clone(); | |
| 545 | − | step_env.insert("GITHUB_ACTION".into(), id.map_or_else(|| format!("__{}", repository.replace('/', "_")), str::to_owned)); | |
| 546 | − | step_env.insert("GITHUB_ACTION_REPOSITORY".into(), repository.clone()); | |
| 547 | − | step_env.insert("GITHUB_ACTION_REF".into(), git_ref.to_owned()); | |
| 548 | − | step_env.insert("GITHUB_ACTION_PATH".into(), dir.display().to_string()); | |
| 484 | + | let mut step_env = action_env(env, action_name, &found, git_ref); | |
| 549 | 485 | ||
| 550 | 486 | if using.starts_with("node") { | |
| 551 | 487 | for (input, value) in &inputs { | |
| 552 | 488 | step_env.insert(format!("INPUT_{}", input.replace(' ', "_").to_ascii_uppercase()), value.clone()); | |
| 553 | 489 | } | |
| 554 | 490 | let condition_of = |key: &str| runs.get(key).map(expr::to_text).unwrap_or_else(|| "always()".into()); | |
| 491 | + | // State an action saves, `STATE_<name>` for its later stages. | |
| 492 | + | let mut state = BTreeMap::new(); | |
| 555 | 493 | if let Some(pre) = runs.get("pre").map(expr::to_text) { | |
| 556 | − | let run_pre = self.with_scope(&contexts, |scope| expr::condition(&condition_of("pre-if"), scope)).unwrap_or(true); | |
| 557 | − | if run_pre && !self.run_node(&dir, &pre, &step_env) { | |
| 558 | − | return (false, BTreeMap::new()); | |
| 494 | + | if frame.action_path.is_some() { | |
| 495 | + | // An action inside a composite action: its `pre` right | |
| 496 | + | // before its `main`. | |
| 497 | + | let run_pre = self.with_scope(&contexts, |scope| expr::condition(&condition_of("pre-if"), scope)).unwrap_or(true); | |
| 498 | + | if run_pre { | |
| 499 | + | if !self.run_node(dir, &pre, &step_env, &using) { | |
| 500 | + | return (false, BTreeMap::new()); | |
| 501 | + | } | |
| 502 | + | state = std::mem::take(&mut self.last_node_state); | |
| 503 | + | } | |
| 504 | + | } else if *local { | |
| 505 | + | self.log.line(&format!("##[warning]`pre` execution is not supported for local action from '{uses}'")); | |
| 506 | + | } else { | |
| 507 | + | // Run before the job's first step (`pre_steps`). | |
| 508 | + | state = self.pre_state.remove(action_name).unwrap_or_default(); | |
| 559 | 509 | } | |
| 560 | 510 | } | |
| 561 | 511 | let Some(main) = runs.get("main").map(expr::to_text) else { | |
| 562 | 512 | self.log.line(&format!("##[error]`{uses}` has no `runs.main`.")); | |
| 563 | 513 | return (false, BTreeMap::new()); | |
| 564 | 514 | }; | |
| 565 | − | let ok = self.run_node(&dir, &main, &step_env); | |
| 515 | + | let mut main_env = step_env.clone(); | |
| 516 | + | for (name, value) in &state { | |
| 517 | + | main_env.insert(format!("STATE_{name}"), value.clone()); | |
| 518 | + | } | |
| 519 | + | let ok = self.run_node(dir, &main, &main_env, &using); | |
| 566 | 520 | let outputs = std::mem::take(&mut self.last_node_outputs); | |
| 567 | − | let state = std::mem::take(&mut self.last_node_state); | |
| 521 | + | state.extend(std::mem::take(&mut self.last_node_state)); | |
| 568 | 522 | if let Some(post) = runs.get("post").map(expr::to_text) { | |
| 569 | 523 | let mut post_env = step_env.clone(); | |
| 570 | 524 | for (name, value) in state { | |
| ⋯ | |||
| 574 | 528 | name: format!("Post {title}"), | |
| 575 | 529 | condition: condition_of("post-if"), | |
| 576 | 530 | env: post_env, | |
| 577 | − | run: PostRun::Node { action_dir: dir.clone(), script: post }, | |
| 531 | + | run: PostRun::Node { action_dir: dir.clone(), script: post, using: using.clone() }, | |
| 578 | 532 | }); | |
| 579 | 533 | } | |
| 580 | 534 | return (ok, outputs); | |
| ⋯ | |||
| 586 | 540 | action_path: Some(dir.display().to_string()), | |
| 587 | 541 | env: env.clone(), | |
| 588 | 542 | }; | |
| 589 | − | let steps: Vec<Map<String, Value>> = runs.get("steps").and_then(Value::as_array).map(|s| s.iter().filter_map(|s| s.as_object().cloned()).collect()).unwrap_or_default(); | |
| 543 | + | let mut steps: Vec<Map<String, Value>> = runs.get("steps").and_then(Value::as_array).map(|s| s.iter().filter_map(|s| s.as_object().cloned()).collect()).unwrap_or_default(); | |
| 544 | + | super::name_steps(&mut steps); | |
| 590 | 545 | let was_failed = self.failed; | |
| 591 | 546 | // A composite's steps see their own success, not the job's. | |
| 592 | 547 | self.failed = false; | |
| ⋯ | |||
| 610 | 565 | return (ok, outputs); | |
| 611 | 566 | } | |
| 612 | 567 | if using == "docker" { | |
| 613 | − | return self.docker_action(uses, &dir, &runs, &inputs, &step_env, frame, title); | |
| 568 | + | return self.docker_action(uses, dir, &runs, &inputs, &step_env, frame, title); | |
| 614 | 569 | } | |
| 615 | 570 | self.log.line(&format!("##[error]`{uses}` runs with `{using}`, which g1t does not know.")); | |
| 616 | 571 | (false, BTreeMap::new()) | |
| ⋯ | |||
| 692 | 647 | } | |
| 693 | 648 | } | |
| 694 | 649 | ||
| 650 | + | /// An action, read from its action.yml. | |
| 651 | + | pub(crate) struct Action { | |
| 652 | + | pub(crate) dir: PathBuf, | |
| 653 | + | /// `owner/repo`, or empty for one in the repository. | |
| 654 | + | pub(crate) repository: String, | |
| 655 | + | pub(crate) local: bool, | |
| 656 | + | pub(crate) manifest: Value, | |
| 657 | + | } | |
| 658 | + | ||
| 659 | + | /// An action's `pre`, run before the job's first step. | |
| 660 | + | pub(crate) struct PreStep { | |
| 661 | + | /// Its step's `GITHUB_ACTION`, by which the step finds the state it saved. | |
| 662 | + | pub(crate) key: String, | |
| 663 | + | /// `pre-if`, `always()` when it has none. | |
| 664 | + | pub(crate) condition: String, | |
| 665 | + | pub(crate) action_dir: PathBuf, | |
| 666 | + | pub(crate) script: String, | |
| 667 | + | pub(crate) using: String, | |
| 668 | + | /// The `env` context its condition reads. | |
| 669 | + | pub(crate) context_env: BTreeMap<String, String>, | |
| 670 | + | /// What its process is given: the step's variables and inputs. | |
| 671 | + | pub(crate) env: BTreeMap<String, String>, | |
| 672 | + | } | |
| 673 | + | ||
| 674 | + | impl Job { | |
| 675 | + | /// Finds `uses` (fetching it when it is another repository's) and reads | |
| 676 | + | /// its action.yml. A problem is told in the log. | |
| 677 | + | fn resolve_action(&mut self, uses: &str) -> Option<Action> { | |
| 678 | + | let (name, git_ref) = uses.split_once('@').unwrap_or((uses, "")); | |
| 679 | + | let source = if let Some(local) = name.strip_prefix("./") { | |
| 680 | + | // A repository moved from GitHub renamed `.github` to `.g1t`, but | |
| 681 | + | // its workflows still say `./.github/actions/…`. | |
| 682 | + | let mut dir = self.workspace.join(local); | |
| 683 | + | if let Some(rest) = local.strip_prefix(".github/") | |
| 684 | + | && !dir.exists() | |
| 685 | + | { | |
| 686 | + | dir = self.workspace.join(".g1t").join(rest); | |
| 687 | + | } | |
| 688 | + | Source::Local(dir) | |
| 689 | + | } else { | |
| 690 | + | let mut parts = name.splitn(3, '/'); | |
| 691 | + | let (Some(owner), Some(repo)) = (parts.next(), parts.next()) else { | |
| 692 | + | self.log.line(&format!("##[error]`{uses}` is not an action: use owner/repo@ref, owner/repo/path@ref, or ./path.")); | |
| 693 | + | return None; | |
| 694 | + | }; | |
| 695 | + | if git_ref.is_empty() { | |
| 696 | + | self.log.line(&format!("##[error]`{uses}` needs a version, such as @v4.")); | |
| 697 | + | return None; | |
| 698 | + | } | |
| 699 | + | Source::GitHub { | |
| 700 | + | owner: owner.to_owned(), | |
| 701 | + | repo: repo.to_owned(), | |
| 702 | + | path: parts.next().unwrap_or_default().to_owned(), | |
| 703 | + | git_ref: git_ref.to_owned(), | |
| 704 | + | } | |
| 705 | + | }; | |
| 706 | + | let (dir, repository) = match &source { | |
| 707 | + | Source::Local(dir) => (dir.clone(), String::new()), | |
| 708 | + | Source::GitHub { owner, repo, path, git_ref } => { | |
| 709 | + | let root = self.fetch_remote_action(owner, repo, git_ref)?; | |
| 710 | + | (if path.is_empty() { root } else { root.join(path) }, format!("{owner}/{repo}")) | |
| 711 | + | } | |
| 712 | + | }; | |
| 713 | + | let Some(manifest) = ["action.yml", "action.yaml"].iter().map(|f| dir.join(f)).find(|p| p.exists()) else { | |
| 714 | + | self.log.line(&format!("##[error]`{uses}` has no action.yml.")); | |
| 715 | + | return None; | |
| 716 | + | }; | |
| 717 | + | let Some(yaml) = std::fs::read_to_string(&manifest).ok().and_then(|text| serde_yaml::from_str::<serde_yaml::Value>(&text).ok()) else { | |
| 718 | + | self.log.line(&format!("##[error]`{uses}`: its action.yml does not read.")); | |
| 719 | + | return None; | |
| 720 | + | }; | |
| 721 | + | Some(Action { dir, repository, local: matches!(source, Source::Local(_)), manifest: yaml_to_json(&yaml) }) | |
| 722 | + | } | |
| 723 | + | ||
| 724 | + | /// An action's inputs: what the step gives, else the action's defaults, | |
| 725 | + | /// read with `contexts`. With `warn`, inputs the action does not declare | |
| 726 | + | /// are warned of, as GitHub's runner does. | |
| 727 | + | fn action_inputs(&mut self, action: &Value, with: &BTreeMap<String, String>, contexts: &Map<String, Value>, warn: bool) -> BTreeMap<String, String> { | |
| 728 | + | let mut inputs: BTreeMap<String, String> = BTreeMap::new(); | |
| 729 | + | let mut declared_names = Vec::new(); | |
| 730 | + | if let Some(Value::Object(declared)) = action.get("inputs") { | |
| 731 | + | for (input, spec) in declared { | |
| 732 | + | declared_names.push(input.clone()); | |
| 733 | + | let given = with.iter().find(|(k, _)| k.eq_ignore_ascii_case(input)).map(|(_, v)| v.clone()); | |
| 734 | + | let value = match given { | |
| 735 | + | Some(value) => value, | |
| 736 | + | None => match spec.get("default") { | |
| 737 | + | Some(default) => { | |
| 738 | + | let default = self.with_scope(contexts, |scope| expr::interpolate_value(default, scope)).unwrap_or(Value::Null); | |
| 739 | + | expr::to_text(&default) | |
| 740 | + | } | |
| 741 | + | None => String::new(), | |
| 742 | + | }, | |
| 743 | + | }; | |
| 744 | + | inputs.insert(input.clone(), value); | |
| 745 | + | } | |
| 746 | + | } | |
| 747 | + | for (key, value) in with { | |
| 748 | + | if !inputs.keys().any(|k| k.eq_ignore_ascii_case(key)) { | |
| 749 | + | inputs.insert(key.clone(), value.clone()); | |
| 750 | + | } | |
| 751 | + | } | |
| 752 | + | let docker = action["runs"]["using"].as_str().is_some_and(|u| u.eq_ignore_ascii_case("docker")); | |
| 753 | + | if warn && let Some(warning) = unexpected_inputs(&declared_names, &with.keys().cloned().collect::<Vec<_>>(), docker) { | |
| 754 | + | self.log.line(&format!("##[warning]{warning}")); | |
| 755 | + | } | |
| 756 | + | inputs | |
| 757 | + | } | |
| 758 | + | ||
| 759 | + | /// The `pre`s of the job's steps, in step order, each with its step's | |
| 760 | + | /// index: those of JavaScript actions from other repositories, fetched | |
| 761 | + | /// now, as GitHub's runner fetches every action while it sets up the | |
| 762 | + | /// job. An action in the repository is not there yet, so its `pre` is | |
| 763 | + | /// not run, as on GitHub. | |
| 764 | + | pub(crate) fn pre_steps(&mut self, steps: &[Map<String, Value>]) -> Vec<(usize, PreStep)> { | |
| 765 | + | let frame = Frame::default(); | |
| 766 | + | let mut pres = Vec::new(); | |
| 767 | + | for (index, step) in steps.iter().enumerate() { | |
| 768 | + | let Some(uses) = step.get("uses").map(|u| expr::to_text(u).trim().to_owned()) else { continue }; | |
| 769 | + | // Only another repository's action, which `prepare_actions` has | |
| 770 | + | // already downloaded. | |
| 771 | + | if remote_action(&uses, self.docker_hosted).is_none() { | |
| 772 | + | continue; | |
| 773 | + | } | |
| 774 | + | let Some(found) = self.resolve_action(&uses) else { continue }; | |
| 775 | + | let runs = found.manifest.get("runs").cloned().unwrap_or(Value::Null); | |
| 776 | + | let using = runs.get("using").map(expr::to_text).unwrap_or_default().to_ascii_lowercase(); | |
| 777 | + | let Some(script) = runs.get("pre").map(expr::to_text).filter(|_| using.starts_with("node")) else { continue }; | |
| 778 | + | let env_before = self.env_context(&frame); | |
| 779 | + | let contexts = self.contexts_for(&frame, &env_before); | |
| 780 | + | let with = self.step_with(step, &contexts); | |
| 781 | + | let context_env = self.step_env(step, &contexts, env_before); | |
| 782 | + | let contexts = self.contexts_for(&frame, &context_env); | |
| 783 | + | let inputs = self.action_inputs(&found.manifest, &with, &contexts, false); | |
| 784 | + | let key = step.get(super::ACTION_NAME).map(expr::to_text).unwrap_or_default(); | |
| 785 | + | let git_ref = uses.split_once('@').map_or("", |(_, r)| r); | |
| 786 | + | let mut env = action_env(&context_env, &key, &found, git_ref); | |
| 787 | + | for (input, value) in &inputs { | |
| 788 | + | env.insert(format!("INPUT_{}", input.replace(' ', "_").to_ascii_uppercase()), value.clone()); | |
| 789 | + | } | |
| 790 | + | let condition = runs.get("pre-if").map(expr::to_text).unwrap_or_else(|| "always()".into()); | |
| 791 | + | pres.push((index, PreStep { key, condition, action_dir: found.dir, script, using, context_env, env })); | |
| 792 | + | } | |
| 793 | + | pres | |
| 794 | + | } | |
| 795 | + | ||
| 796 | + | /// Whether a `pre`'s `pre-if` holds, read against the job's status. | |
| 797 | + | pub(crate) fn pre_runs(&self, pre: &PreStep) -> bool { | |
| 798 | + | let contexts = self.contexts_for(&Frame::default(), &pre.context_env); | |
| 799 | + | self.with_scope(&contexts, |scope| expr::condition(&pre.condition, scope)).unwrap_or(true) | |
| 800 | + | } | |
| 801 | + | ||
| 802 | + | /// Runs a `pre`, keeping the state it saves for its step's main and post. | |
| 803 | + | pub(crate) fn run_pre(&mut self, pre: &PreStep) -> bool { | |
| 804 | + | let ok = self.run_node(&pre.action_dir, &pre.script, &pre.env, &pre.using); | |
| 805 | + | let state = std::mem::take(&mut self.last_node_state); | |
| 806 | + | self.last_node_outputs.clear(); | |
| 807 | + | self.pre_state.insert(pre.key.clone(), state); | |
| 808 | + | ok | |
| 809 | + | } | |
| 810 | + | } | |
| 811 | + | ||
| 812 | + | /// The variables a step running an action gets over its own: which | |
| 813 | + | /// action it is and where. | |
| 814 | + | fn action_env(env: &BTreeMap<String, String>, action_name: &str, action: &Action, git_ref: &str) -> BTreeMap<String, String> { | |
| 815 | + | let mut step_env = env.clone(); | |
| 816 | + | step_env.insert("GITHUB_ACTION".into(), action_name.to_owned()); | |
| 817 | + | step_env.insert("GITHUB_ACTION_REPOSITORY".into(), action.repository.clone()); | |
| 818 | + | step_env.insert("GITHUB_ACTION_REF".into(), git_ref.to_owned()); | |
| 819 | + | step_env.insert("GITHUB_ACTION_PATH".into(), action.dir.display().to_string()); | |
| 820 | + | step_env | |
| 821 | + | } | |
| 822 | + | ||
| 823 | + | /// GitHub's warning for inputs a step gives that its action does not | |
| 824 | + | /// declare (a Docker action also takes `entryPoint` and `args`), if any. | |
| 825 | + | fn unexpected_inputs(declared: &[String], given: &[String], docker: bool) -> Option<String> { | |
| 826 | + | let mut valid: Vec<String> = declared.to_vec(); | |
| 827 | + | if docker { | |
| 828 | + | valid.extend(["entryPoint".to_owned(), "args".to_owned()]); | |
| 829 | + | } | |
| 830 | + | let unexpected: Vec<&str> = given.iter().filter(|g| !valid.iter().any(|v| v.eq_ignore_ascii_case(g))).map(String::as_str).collect(); | |
| 831 | + | if unexpected.is_empty() { | |
| 832 | + | return None; | |
| 833 | + | } | |
| 834 | + | Some(format!("Unexpected input(s) '{}', valid inputs are ['{}']", unexpected.join("', '"), valid.join("', '"))) | |
| 835 | + | } | |
| 836 | + | ||
| 837 | + | /// The Node an action's `runs.using` asks for: for `node20` (or any | |
| 838 | + | /// `nodeNN`), the binary `G1T_NODE20` names or `/opt/node20/bin/node`, | |
| 839 | + | /// when there is one; otherwise, and for anything else, the `node` on | |
| 840 | + | /// `PATH`, the runner's own (Node 24). Nothing is logged either way. | |
| 841 | + | pub(crate) fn node_program(using: &str, var: impl Fn(&str) -> Option<String>, exists: impl Fn(&Path) -> bool) -> String { | |
| 842 | + | let using = using.trim().to_ascii_lowercase(); | |
| 843 | + | let versioned = using.strip_prefix("node").is_some_and(|v| !v.is_empty() && v.chars().all(|c| c.is_ascii_digit())); | |
| 844 | + | if versioned { | |
| 845 | + | let named = var(&format!("G1T_{}", using.to_ascii_uppercase())).filter(|p| !p.is_empty() && exists(Path::new(p))); | |
| 846 | + | if let Some(path) = named { | |
| 847 | + | return path; | |
| 848 | + | } | |
| 849 | + | let standard = format!("/opt/{using}/bin/node"); | |
| 850 | + | if exists(Path::new(&standard)) { | |
| 851 | + | return standard; | |
| 852 | + | } | |
| 853 | + | } | |
| 854 | + | "node".into() | |
| 855 | + | } | |
| 856 | + | ||
| 695 | 857 | #[cfg(test)] | |
| 696 | 858 | mod tests { | |
| 697 | 859 | use super::*; | |
| ⋯ | |||
| 711 | 873 | assert_eq!(found("actions/setup-node"), None); | |
| 712 | 874 | assert_eq!(found("setup-node@v4"), None); | |
| 713 | 875 | } | |
| 876 | + | ||
| 877 | + | #[test] | |
| 878 | + | fn unexpected_inputs_are_named_with_the_valid_ones() { | |
| 879 | + | let declared = vec!["token".to_owned(), "path".to_owned()]; | |
| 880 | + | assert_eq!(unexpected_inputs(&declared, &["Token".into()], false), None); | |
| 881 | + | assert_eq!( | |
| 882 | + | unexpected_inputs(&declared, &["token".into(), "foo".into(), "bar".into()], false).as_deref(), | |
| 883 | + | Some("Unexpected input(s) 'foo', 'bar', valid inputs are ['token', 'path']") | |
| 884 | + | ); | |
| 885 | + | assert_eq!(unexpected_inputs(&[], &["x".into()], false).as_deref(), Some("Unexpected input(s) 'x', valid inputs are ['']")); | |
| 886 | + | // A Docker action takes `entrypoint` and `args` too. | |
| 887 | + | assert_eq!(unexpected_inputs(&declared, &["entrypoint".into(), "args".into()], true), None); | |
| 888 | + | assert!(unexpected_inputs(&declared, &["args".into()], false).is_some()); | |
| 889 | + | } | |
| 890 | + | ||
| 891 | + | #[test] | |
| 892 | + | fn node20_runs_with_a_node20_where_there_is_one() { | |
| 893 | + | let none = |_: &str| None; | |
| 894 | + | assert_eq!(node_program("node20", none, |_| false), "node"); | |
| 895 | + | assert_eq!(node_program("node20", none, |p| p == Path::new("/opt/node20/bin/node")), "/opt/node20/bin/node"); | |
| 896 | + | let named = |name: &str| (name == "G1T_NODE20").then(|| "/usr/local/n20/bin/node".to_owned()); | |
| 897 | + | assert_eq!(node_program("Node20", named, |_| true), "/usr/local/n20/bin/node"); | |
| 898 | + | // A named Node that is not there is passed over. | |
| 899 | + | assert_eq!(node_program("node20", named, |p| p == Path::new("/opt/node20/bin/node")), "/opt/node20/bin/node"); | |
| 900 | + | assert_eq!(node_program("node24", none, |_| false), "node"); | |
| 901 | + | assert_eq!(node_program("composite", none, |_| true), "node"); | |
| 902 | + | assert_eq!(node_program("node", none, |_| true), "node"); | |
| 903 | + | } | |
| 714 | 904 | } | |