Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.
| Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset. | 1 | /** |
| 2 | * How the outbound handler sends a sandbox's request on to an allowed | |
| 3 | * host. An upstream that resets the connection or fails outright would | |
| 4 | * otherwise throw out of the handler, and the sandbox would see its own | |
| 5 | * connection reset, which tools such as npm count as a failure rather | |
| 6 | * than something to retry. So a request that can be sent again (a GET or | |
| 7 | * HEAD with no body) is tried twice more after a short wait, and what | |
| 8 | * still fails comes back as a `502` saying which host and why, which | |
| Merge branch 'worktree-agent-af070177ded5baf00' | 9 | * those tools do retry. It never throws. |
| 10 | * | |
| 11 | * An upstream can also break off in the middle of a body, after its | |
| 12 | * headers have been handed on: npm then fails with `ECONNRESET` and | |
| 13 | * `network aborted`, and never retries it. So an answer of known, modest | |
| 14 | * size to such a request is read whole before it is handed on, and tried | |
| 15 | * again when it breaks off (`BUFFERED_MAX_BYTES`). A larger one, such as a | |
| 16 | * 50 MB package tarball, streams as it comes, and when it breaks off the | |
| 17 | * rest is asked for with a `Range` request from where it stopped and | |
| 18 | * joined on, so the sandbox sees one unbroken answer (`resuming`). | |
| Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset. | 19 | */ |
| 20 | ||
| 21 | import { normalizeHost } from "./egress.ts"; | |
| 22 | ||
| Merge branch 'worktree-agent-af070177ded5baf00' | 23 | /** The waits before the second and third tries, in milliseconds; also before each resumption of a body. */ |
| Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset. | 24 | export const RETRY_WAITS_MS: readonly number[] = [250, 1000]; |
| 25 | ||
| npm installs on g1t's cloud runners stop failing with "network aborted". The upstream sometimes broke off in the middle of a tarball's body, after its headers had already been handed to the sandbox, so the retry added for failed requests couldn't help: npm got a truncated file and gave up. Now an answer to a request that can be sent again (a GET or HEAD with no body), of known length up to 8 MB and sent uncompressed, as package tarballs are, is read whole by the runner Worker before the sandbox gets it; one whose body breaks off is tried again, twice, and still failing becomes a 502 saying why, never a truncated answer. Larger answers and ones of unknown length stream as before. And the repository's installs open at most 8 connections at once rather than npm's 15, since each passes through the Worker and bursts of parallel downloads were when it happened. | 26 | /** |
| 27 | * The largest answer to a replayable request that is read whole before the | |
| Merge branch 'worktree-agent-af070177ded5baf00' | 28 | * sandbox is answered. An answer read whole first is tried again like any |
| npm installs on g1t's cloud runners stop failing with "network aborted". The upstream sometimes broke off in the middle of a tarball's body, after its headers had already been handed to the sandbox, so the retry added for failed requests couldn't help: npm got a truncated file and gave up. Now an answer to a request that can be sent again (a GET or HEAD with no body), of known length up to 8 MB and sent uncompressed, as package tarballs are, is read whole by the runner Worker before the sandbox gets it; one whose body breaks off is tried again, twice, and still failing becomes a 502 saying why, never a truncated answer. Larger answers and ones of unknown length stream as before. And the repository's installs open at most 8 connections at once rather than npm's 15, since each passes through the Worker and bursts of parallel downloads were when it happened. | 29 | * other failure. Package tarballs and most API answers are well under this; |
| Merge branch 'worktree-agent-af070177ded5baf00' | 30 | * larger ones stream, and resume when they break off (`resuming`). |
| npm installs on g1t's cloud runners stop failing with "network aborted". The upstream sometimes broke off in the middle of a tarball's body, after its headers had already been handed to the sandbox, so the retry added for failed requests couldn't help: npm got a truncated file and gave up. Now an answer to a request that can be sent again (a GET or HEAD with no body), of known length up to 8 MB and sent uncompressed, as package tarballs are, is read whole by the runner Worker before the sandbox gets it; one whose body breaks off is tried again, twice, and still failing becomes a 502 saying why, never a truncated answer. Larger answers and ones of unknown length stream as before. And the repository's installs open at most 8 connections at once rather than npm's 15, since each passes through the Worker and bursts of parallel downloads were when it happened. | 31 | */ |
| 32 | export const BUFFERED_MAX_BYTES = 8 * 1024 * 1024; | |
| 33 | ||
| Merge branch 'worktree-agent-af070177ded5baf00' | 34 | /** The `content-length` of an answer sent as it is (no `content-encoding`), or null. */ |
| 35 | function plainLength(response: Response): number | null { | |
| 36 | const encoding = (response.headers.get("content-encoding") ?? "identity").toLowerCase(); | |
| 37 | if (encoding !== "identity") return null; | |
| 38 | const header = response.headers.get("content-length"); | |
| 39 | const length = header === null ? NaN : Number(header); | |
| 40 | return Number.isSafeInteger(length) && length >= 0 ? length : null; | |
| 41 | } | |
| 42 | ||
| npm installs on g1t's cloud runners stop failing with "network aborted". The upstream sometimes broke off in the middle of a tarball's body, after its headers had already been handed to the sandbox, so the retry added for failed requests couldn't help: npm got a truncated file and gave up. Now an answer to a request that can be sent again (a GET or HEAD with no body), of known length up to 8 MB and sent uncompressed, as package tarballs are, is read whole by the runner Worker before the sandbox gets it; one whose body breaks off is tried again, twice, and still failing becomes a 502 saying why, never a truncated answer. Larger answers and ones of unknown length stream as before. And the repository's installs open at most 8 connections at once rather than npm's 15, since each passes through the Worker and bursts of parallel downloads were when it happened. | 43 | /** Whether an answer is to be read whole: a known length within the limit, sent as it is. */ |
| 44 | export function bufferable(response: Response): boolean { | |
| 45 | if (response.body === null) return false; | |
| Merge branch 'worktree-agent-af070177ded5baf00' | 46 | const length = plainLength(response); |
| 47 | return length !== null && length <= BUFFERED_MAX_BYTES; | |
| npm installs on g1t's cloud runners stop failing with "network aborted". The upstream sometimes broke off in the middle of a tarball's body, after its headers had already been handed to the sandbox, so the retry added for failed requests couldn't help: npm got a truncated file and gave up. Now an answer to a request that can be sent again (a GET or HEAD with no body), of known length up to 8 MB and sent uncompressed, as package tarballs are, is read whole by the runner Worker before the sandbox gets it; one whose body breaks off is tried again, twice, and still failing becomes a 502 saying why, never a truncated answer. Larger answers and ones of unknown length stream as before. And the repository's installs open at most 8 connections at once rather than npm's 15, since each passes through the Worker and bursts of parallel downloads were when it happened. | 48 | } |
| 49 | ||
| 50 | /** An answer whose body is read whole; throws when the body breaks off. */ | |
| 51 | async function whole(response: Response): Promise<Response> { | |
| 52 | const body = await response.arrayBuffer(); | |
| 53 | const expected = Number(response.headers.get("content-length")); | |
| 54 | if (body.byteLength !== expected) throw new Error(`the answer broke off after ${body.byteLength} of ${expected} bytes`); | |
| 55 | return new Response(body, { status: response.status, statusText: response.statusText, headers: response.headers }); | |
| 56 | } | |
| 57 | ||
| Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset. | 58 | /** Whether a request can be sent again as it was: no body, nothing changed. */ |
| 59 | export function replayable(request: Request): boolean { | |
| 60 | return (request.method === "GET" || request.method === "HEAD") && request.body === null; | |
| 61 | } | |
| 62 | ||
| Merge branch 'worktree-agent-af070177ded5baf00' | 63 | /** |
| 64 | * What a broken-off body can be resumed against: the validator that makes | |
| 65 | * sure the rest comes from the same file (a strong `ETag`, else | |
| 66 | * `Last-Modified`), and its whole length. Null when it cannot be: only a | |
| 67 | * whole `200` to a GET that asked for no range, of known length, sent as it | |
| 68 | * is, from a host that takes byte ranges. | |
| 69 | */ | |
| 70 | export function resumeFrom(request: Request, response: Response): { validator: string; length: number } | null { | |
| 71 | if (request.method !== "GET" || request.headers.has("range") || response.status !== 200 || response.body === null) return null; | |
| 72 | if (!(response.headers.get("accept-ranges") ?? "").toLowerCase().split(/[,\s]+/).includes("bytes")) return null; | |
| 73 | const length = plainLength(response); | |
| 74 | if (length === null || length === 0) return null; | |
| 75 | const etag = response.headers.get("etag"); | |
| 76 | const validator = etag && !etag.startsWith("W/") ? etag : response.headers.get("last-modified"); | |
| 77 | return validator ? { validator, length } : null; | |
| 78 | } | |
| 79 | ||
| 80 | /** Whether a `206` carries exactly the rest of the file from `offset`. */ | |
| 81 | function restFrom(response: Response, offset: number, length: number): boolean { | |
| 82 | if (response.status !== 206 || response.body === null) return false; | |
| 83 | const range = /^bytes (\d+)-(\d+)\/(\d+)$/.exec((response.headers.get("content-range") ?? "").trim()); | |
| 84 | return range !== null && Number(range[1]) === offset && Number(range[2]) === length - 1 && Number(range[3]) === length; | |
| 85 | } | |
| 86 | ||
| 87 | /** | |
| 88 | * `response`'s body, joined across breaks: when it breaks off (an error, or | |
| 89 | * an end short of its length), the rest is asked for with | |
| 90 | * `Range: bytes=<received>-` and `If-Range`, after the waits of | |
| 91 | * `RETRY_WAITS_MS`, and read on from there. When the rest cannot be had | |
| 92 | * (no `206` for exactly that range, or every try failed), the body errors | |
| 93 | * as it would have. Each break is one line of the Worker's log. | |
| 94 | */ | |
| 95 | export function resuming( | |
| 96 | request: Request, | |
| 97 | response: Response, | |
| 98 | from: { validator: string; length: number }, | |
| 99 | send: (request: Request) => Promise<Response>, | |
| 100 | sleep: (ms: number) => Promise<void>, | |
| 101 | ): Response { | |
| 102 | const host = normalizeHost(new URL(request.url).host); | |
| 103 | const { validator, length } = from; | |
| 104 | let reader = (response.body as ReadableStream<Uint8Array>).getReader(); | |
| 105 | let received = 0; | |
| 106 | let resumes = 0; | |
| 107 | ||
| 108 | /** A reader for the rest from `received`, or the reason there is none. */ | |
| 109 | const reopen = async (broke: unknown): Promise<ReadableStreamDefaultReader<Uint8Array>> => { | |
| 110 | let failure = broke; | |
| 111 | while (resumes < RETRY_WAITS_MS.length) { | |
| 112 | const wait = RETRY_WAITS_MS[resumes++]; | |
| 113 | console.log("egress body broke off", host, `${received} of ${length} bytes`, `resuming, try ${resumes} of ${RETRY_WAITS_MS.length}`, String(failure)); | |
| 114 | await sleep(wait); | |
| 115 | const headers = new Headers(request.headers); | |
| 116 | headers.set("range", `bytes=${received}-`); | |
| 117 | headers.set("if-range", validator); | |
| 118 | let rest: Response; | |
| 119 | try { | |
| 120 | rest = await send(new Request(request.url, { method: "GET", headers, redirect: request.redirect })); | |
| 121 | } catch (error) { | |
| 122 | failure = error; | |
| 123 | continue; | |
| 124 | } | |
| 125 | if (restFrom(rest, received, length)) return (rest.body as ReadableStream<Uint8Array>).getReader(); | |
| 126 | await rest.body?.cancel().catch(() => {}); | |
| 127 | // The file changed, or the host would not send that range: there is no rest to join on. | |
| 128 | const why = `the rest could not be had (${rest.status} ${rest.headers.get("content-range") ?? "without a range"})`; | |
| 129 | console.log("egress body not resumed", host, `${received} of ${length} bytes`, why); | |
| 130 | throw new Error(`the answer broke off after ${received} of ${length} bytes, and ${why}`); | |
| 131 | } | |
| 132 | console.log("egress body not resumed", host, `${received} of ${length} bytes`, String(failure)); | |
| 133 | throw failure instanceof Error ? failure : new Error(String(failure)); | |
| 134 | }; | |
| 135 | ||
| 136 | const body = new ReadableStream<Uint8Array>({ | |
| 137 | async pull(controller) { | |
| 138 | for (;;) { | |
| 139 | let chunk: ReadableStreamReadResult<Uint8Array>; | |
| 140 | try { | |
| 141 | chunk = await reader.read(); | |
| 142 | if (chunk.done && received < length) throw new Error(`the answer ended after ${received} of ${length} bytes`); | |
| 143 | } catch (error) { | |
| 144 | try { | |
| 145 | reader = await reopen(error); | |
| 146 | } catch (final) { | |
| 147 | controller.error(final); | |
| 148 | return; | |
| 149 | } | |
| 150 | continue; | |
| 151 | } | |
| 152 | if (chunk.done) { | |
| 153 | controller.close(); | |
| 154 | return; | |
| 155 | } | |
| 156 | if (received + chunk.value.byteLength > length) { | |
| 157 | controller.error(new Error(`the answer ran past its ${length} bytes`)); | |
| 158 | return; | |
| 159 | } | |
| 160 | received += chunk.value.byteLength; | |
| 161 | controller.enqueue(chunk.value); | |
| 162 | return; | |
| 163 | } | |
| 164 | }, | |
| 165 | cancel(reason) { | |
| 166 | return reader.cancel(reason); | |
| 167 | }, | |
| 168 | }); | |
| 169 | // In the Worker, a stream of fixed length keeps the answer's | |
| 170 | // `content-length`, so the sandbox can tell a whole answer from a short one. | |
| 171 | const Fixed = (globalThis as { FixedLengthStream?: new (length: number) => TransformStream<Uint8Array, Uint8Array> }).FixedLengthStream; | |
| 172 | const sent = Fixed ? body.pipeThrough(new Fixed(length)) : body; | |
| 173 | return new Response(sent, { status: response.status, statusText: response.statusText, headers: response.headers }); | |
| 174 | } | |
| 175 | ||
| Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset. | 176 | /** What a request that could not be sent gets back. */ |
| 177 | export function upstreamFailed(host: string, error: unknown): Response { | |
| 178 | const why = error instanceof Error ? error.message : String(error); | |
| 179 | return new Response(`g1t egress: ${normalizeHost(host)} could not be reached: ${why}\n`, { | |
| 180 | status: 502, | |
| 181 | headers: { "content-type": "text/plain; charset=utf-8", "x-g1t-egress": "upstream-failed" }, | |
| 182 | }); | |
| 183 | } | |
| 184 | ||
| 185 | const wait = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)); | |
| 186 | ||
| 187 | /** | |
| 188 | * Sends `request` with `send` (`fetch`), trying a replayable one again on | |
| 189 | * a thrown error, and answering `502` when it cannot. Each failure is one | |
| 190 | * line of the Worker's log. | |
| 191 | */ | |
| 192 | export async function forward( | |
| 193 | request: Request, | |
| 194 | send: (request: Request) => Promise<Response> = fetch, | |
| 195 | sleep: (ms: number) => Promise<void> = wait, | |
| 196 | ): Promise<Response> { | |
| 197 | const host = new URL(request.url).host; | |
| 198 | const tries = replayable(request) ? RETRY_WAITS_MS.length + 1 : 1; | |
| 199 | let failure: unknown = null; | |
| 200 | for (let attempt = 1; attempt <= tries; attempt++) { | |
| 201 | if (attempt > 1) await sleep(RETRY_WAITS_MS[attempt - 2]); | |
| 202 | try { | |
| npm installs on g1t's cloud runners stop failing with "network aborted". The upstream sometimes broke off in the middle of a tarball's body, after its headers had already been handed to the sandbox, so the retry added for failed requests couldn't help: npm got a truncated file and gave up. Now an answer to a request that can be sent again (a GET or HEAD with no body), of known length up to 8 MB and sent uncompressed, as package tarballs are, is read whole by the runner Worker before the sandbox gets it; one whose body breaks off is tried again, twice, and still failing becomes a 502 saying why, never a truncated answer. Larger answers and ones of unknown length stream as before. And the repository's installs open at most 8 connections at once rather than npm's 15, since each passes through the Worker and bursts of parallel downloads were when it happened. | 203 | const response = await send(request); |
| Merge branch 'worktree-agent-af070177ded5baf00' | 204 | if (tries === 1) return response; |
| npm installs on g1t's cloud runners stop failing with "network aborted". The upstream sometimes broke off in the middle of a tarball's body, after its headers had already been handed to the sandbox, so the retry added for failed requests couldn't help: npm got a truncated file and gave up. Now an answer to a request that can be sent again (a GET or HEAD with no body), of known length up to 8 MB and sent uncompressed, as package tarballs are, is read whole by the runner Worker before the sandbox gets it; one whose body breaks off is tried again, twice, and still failing becomes a 502 saying why, never a truncated answer. Larger answers and ones of unknown length stream as before. And the repository's installs open at most 8 connections at once rather than npm's 15, since each passes through the Worker and bursts of parallel downloads were when it happened. | 205 | // Read whole first when it can be sent again, so a body that breaks off is retried too. |
| Merge branch 'worktree-agent-af070177ded5baf00' | 206 | if (bufferable(response)) return await whole(response); |
| 207 | // Too large to hold: streamed, and resumed where it breaks off. | |
| 208 | const from = resumeFrom(request, response); | |
| 209 | return from ? resuming(request, response, from, send, sleep) : response; | |
| Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset. | 210 | } catch (error) { |
| 211 | failure = error; | |
| 212 | console.log("egress upstream failed", normalizeHost(host), request.method, `try ${attempt} of ${tries}`, String(error)); | |
| 213 | } | |
| 214 | } | |
| 215 | return upstreamFailed(host, failure); | |
| 216 | } |
This file's history is long; its oldest lines are credited to the oldest commit read.