Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.
| Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset. | 1 | /** |
| 2 | * How the outbound handler sends a sandbox's request on to an allowed | |
| 3 | * host. An upstream that resets the connection or fails outright would | |
| 4 | * otherwise throw out of the handler, and the sandbox would see its own | |
| 5 | * connection reset, which tools such as npm count as a failure rather | |
| 6 | * than something to retry. So a request that can be sent again (a GET or | |
| 7 | * HEAD with no body) is tried twice more after a short wait, and what | |
| 8 | * still fails comes back as a `502` saying which host and why, which | |
| npm installs on g1t's cloud runners stop failing with "network aborted". The upstream sometimes broke off in the middle of a tarball's body, after its headers had already been handed to the sandbox, so the retry added for failed requests couldn't help: npm got a truncated file and gave up. Now an answer to a request that can be sent again (a GET or HEAD with no body), of known length up to 8 MB and sent uncompressed, as package tarballs are, is read whole by the runner Worker before the sandbox gets it; one whose body breaks off is tried again, twice, and still failing becomes a 502 saying why, never a truncated answer. Larger answers and ones of unknown length stream as before. And the repository's installs open at most 8 connections at once rather than npm's 15, since each passes through the Worker and bursts of parallel downloads were when it happened. | 9 | * those tools do retry. It never throws. An answer of known, modest size |
| 10 | * to such a request is read whole before it is handed on, so one whose | |
| 11 | * body breaks off midway is tried again too (`BUFFERED_MAX_BYTES`). | |
| Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset. | 12 | */ |
| 13 | ||
| 14 | import { normalizeHost } from "./egress.ts"; | |
| 15 | ||
| 16 | /** The waits before the second and third tries, in milliseconds. */ | |
| 17 | export const RETRY_WAITS_MS: readonly number[] = [250, 1000]; | |
| 18 | ||
| npm installs on g1t's cloud runners stop failing with "network aborted". The upstream sometimes broke off in the middle of a tarball's body, after its headers had already been handed to the sandbox, so the retry added for failed requests couldn't help: npm got a truncated file and gave up. Now an answer to a request that can be sent again (a GET or HEAD with no body), of known length up to 8 MB and sent uncompressed, as package tarballs are, is read whole by the runner Worker before the sandbox gets it; one whose body breaks off is tried again, twice, and still failing becomes a 502 saying why, never a truncated answer. Larger answers and ones of unknown length stream as before. And the repository's installs open at most 8 connections at once rather than npm's 15, since each passes through the Worker and bursts of parallel downloads were when it happened. | 19 | /** |
| 20 | * The largest answer to a replayable request that is read whole before the | |
| 21 | * sandbox is answered. An upstream can also break off in the middle of a | |
| 22 | * body, after its headers: npm then reports `network aborted` with a | |
| 23 | * truncated tarball, and by then the answer was already the sandbox's, too | |
| 24 | * late to try again. An answer read whole first is tried again like any | |
| 25 | * other failure. Package tarballs and most API answers are well under this; | |
| 26 | * larger ones, and ones of unknown length, stream as they come. | |
| 27 | */ | |
| 28 | export const BUFFERED_MAX_BYTES = 8 * 1024 * 1024; | |
| 29 | ||
| 30 | /** Whether an answer is to be read whole: a known length within the limit, sent as it is. */ | |
| 31 | export function bufferable(response: Response): boolean { | |
| 32 | if (response.body === null) return false; | |
| 33 | const length = Number(response.headers.get("content-length") ?? NaN); | |
| 34 | const encoding = (response.headers.get("content-encoding") ?? "identity").toLowerCase(); | |
| 35 | return Number.isFinite(length) && length <= BUFFERED_MAX_BYTES && encoding === "identity"; | |
| 36 | } | |
| 37 | ||
| 38 | /** An answer whose body is read whole; throws when the body breaks off. */ | |
| 39 | async function whole(response: Response): Promise<Response> { | |
| 40 | const body = await response.arrayBuffer(); | |
| 41 | const expected = Number(response.headers.get("content-length")); | |
| 42 | if (body.byteLength !== expected) throw new Error(`the answer broke off after ${body.byteLength} of ${expected} bytes`); | |
| 43 | return new Response(body, { status: response.status, statusText: response.statusText, headers: response.headers }); | |
| 44 | } | |
| 45 | ||
| Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset. | 46 | /** Whether a request can be sent again as it was: no body, nothing changed. */ |
| 47 | export function replayable(request: Request): boolean { | |
| 48 | return (request.method === "GET" || request.method === "HEAD") && request.body === null; | |
| 49 | } | |
| 50 | ||
| 51 | /** What a request that could not be sent gets back. */ | |
| 52 | export function upstreamFailed(host: string, error: unknown): Response { | |
| 53 | const why = error instanceof Error ? error.message : String(error); | |
| 54 | return new Response(`g1t egress: ${normalizeHost(host)} could not be reached: ${why}\n`, { | |
| 55 | status: 502, | |
| 56 | headers: { "content-type": "text/plain; charset=utf-8", "x-g1t-egress": "upstream-failed" }, | |
| 57 | }); | |
| 58 | } | |
| 59 | ||
| 60 | const wait = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)); | |
| 61 | ||
| 62 | /** | |
| 63 | * Sends `request` with `send` (`fetch`), trying a replayable one again on | |
| 64 | * a thrown error, and answering `502` when it cannot. Each failure is one | |
| 65 | * line of the Worker's log. | |
| 66 | */ | |
| 67 | export async function forward( | |
| 68 | request: Request, | |
| 69 | send: (request: Request) => Promise<Response> = fetch, | |
| 70 | sleep: (ms: number) => Promise<void> = wait, | |
| 71 | ): Promise<Response> { | |
| 72 | const host = new URL(request.url).host; | |
| 73 | const tries = replayable(request) ? RETRY_WAITS_MS.length + 1 : 1; | |
| 74 | let failure: unknown = null; | |
| 75 | for (let attempt = 1; attempt <= tries; attempt++) { | |
| 76 | if (attempt > 1) await sleep(RETRY_WAITS_MS[attempt - 2]); | |
| 77 | try { | |
| npm installs on g1t's cloud runners stop failing with "network aborted". The upstream sometimes broke off in the middle of a tarball's body, after its headers had already been handed to the sandbox, so the retry added for failed requests couldn't help: npm got a truncated file and gave up. Now an answer to a request that can be sent again (a GET or HEAD with no body), of known length up to 8 MB and sent uncompressed, as package tarballs are, is read whole by the runner Worker before the sandbox gets it; one whose body breaks off is tried again, twice, and still failing becomes a 502 saying why, never a truncated answer. Larger answers and ones of unknown length stream as before. And the repository's installs open at most 8 connections at once rather than npm's 15, since each passes through the Worker and bursts of parallel downloads were when it happened. | 78 | const response = await send(request); |
| 79 | // Read whole first when it can be sent again, so a body that breaks off is retried too. | |
| 80 | return tries > 1 && bufferable(response) ? await whole(response) : response; | |
| Merge brokered secrets: a secret a job uses only to call an API can be tied to the hosts it is for, and the job gets a stand-in in its place, which g1t fills in on the way out to those hosts and nowhere else, so the value never enters the sandbox; and a request to an allowed host that fails upstream is retried when it can be, or answered with a 502 that says why, rather than reset. | 81 | } catch (error) { |
| 82 | failure = error; | |
| 83 | console.log("egress upstream failed", normalizeHost(host), request.method, `try ${attempt} of ${tries}`, String(error)); | |
| 84 | } | |
| 85 | } | |
| 86 | return upstreamFailed(host, failure); | |
| 87 | } |
This file's history is long; its oldest lines are credited to the oldest commit read.