Skip to content
Sign up

flagon-io/g1t

public
00

Where people and agents ship software together. The open-source git platform for the whole job: issues, agents, checks and deploys to the edge.

g1t.sh
gitagentscloudflarerustforgemcp

Commit

A sandbox's request to an allowed host that fails upstream no longer resets the sandbox's connection. The runner Worker sent each one on with a bare fetch, so a reset or a failure upstream (as under npm ci's many tarball fetches) was thrown out of the outbound handler, and npm saw ECONNRESET and gave up. Forwarding is now one function (forward.ts) that egress() and the brokered-secret helper both use: a GET or HEAD with no body is tried twice more, after 250 ms and then a second, and what still fails comes back as a 502 whose plain-text body names the host and the error, with x-g1t-egress: upstream-failed, which npm and most tools retry. A request with a body is never sent twice. Each failure is one line of the Worker's log, with the host, the method and the error.

syntaqxcommitted Parent52cdcf0Browse files
4 files+114−20/4 viewed
+2−1
99
1010 import { type BrokeredJob, brokerFor, fill, scrub, scrubHeaders, scrubbable, standInsIn } from "./broker.ts";
1111 import { sandboxNamespace } from "./egress.ts";
12+import { forward } from "./forward.ts";
1213
1314 /**
1415 * What the outbound handler is given for brokering: `open` for a job whose
5253 }
5354 }
5455 if (filled.length === 0) return null;
55− const response = await fetch(new Request(url, { method: request.method, headers, body: request.body, redirect: "manual" }));
56+ const response = await forward(new Request(url, { method: request.method, headers, body: request.body, redirect: "manual" }));
5657 const answered = scrubHeaders(response.headers, filled);
5758 const init = { status: response.status, statusText: response.statusText, headers: answered };
5859 if (!scrubbable(response.headers)) return new Response(response.body, init);
+54−0
1+import assert from "node:assert/strict";
2+import { test } from "node:test";
3+
4+import { RETRY_WAITS_MS, forward } from "./forward.ts";
5+
6+/** A fetch that throws `failures` times, then answers 200; it counts its calls. */
7+function flaky(failures: number) {
8+ const calls: Request[] = [];
9+ const send = async (request: Request) => {
10+ calls.push(request);
11+ if (calls.length <= failures) throw new Error("Network connection lost.");
12+ return new Response("ok");
13+ };
14+ return { calls, send };
15+}
16+
17+const waits: number[] = [];
18+const sleep = async (ms: number) => {
19+ waits.push(ms);
20+};
21+
22+test("a GET that fails is tried again, and its answer comes back", async () => {
23+ waits.length = 0;
24+ const { calls, send } = flaky(2);
25+ const response = await forward(new Request("https://registry.npmjs.org/react"), send, sleep);
26+ assert.equal(response.status, 200);
27+ assert.equal(await response.text(), "ok");
28+ assert.equal(calls.length, 3);
29+ assert.deepEqual(waits, [...RETRY_WAITS_MS]);
30+});
31+
32+test("one that keeps failing gets a 502 naming the host and why, never a throw", async () => {
33+ const { calls, send } = flaky(10);
34+ const response = await forward(new Request("https://Registry.npmjs.org:443/react"), send, sleep);
35+ assert.equal(calls.length, 3);
36+ assert.equal(response.status, 502);
37+ assert.equal(response.headers.get("x-g1t-egress"), "upstream-failed");
38+ assert.match(response.headers.get("content-type") ?? "", /^text\/plain/);
39+ const body = await response.text();
40+ assert.match(body, /registry\.npmjs\.org could not be reached/);
41+ assert.match(body, /Network connection lost\./);
42+});
43+
44+test("a POST is never sent twice", async () => {
45+ const { calls, send } = flaky(1);
46+ const response = await forward(new Request("https://api.example.com/upload", { method: "POST", body: "data" }), send, sleep);
47+ assert.equal(calls.length, 1);
48+ assert.equal(response.status, 502);
49+});
50+
51+test("an answer, even an error, is passed on as it is", async () => {
52+ const response = await forward(new Request("https://example.com/"), async () => new Response("no", { status: 503 }), sleep);
53+ assert.equal(response.status, 503);
54+});
+56−0
1+/**
2+ * How the outbound handler sends a sandbox's request on to an allowed
3+ * host. An upstream that resets the connection or fails outright would
4+ * otherwise throw out of the handler, and the sandbox would see its own
5+ * connection reset, which tools such as npm count as a failure rather
6+ * than something to retry. So a request that can be sent again (a GET or
7+ * HEAD with no body) is tried twice more after a short wait, and what
8+ * still fails comes back as a `502` saying which host and why, which
9+ * those tools do retry. It never throws.
10+ */
11+
12+import { normalizeHost } from "./egress.ts";
13+
14+/** The waits before the second and third tries, in milliseconds. */
15+export const RETRY_WAITS_MS: readonly number[] = [250, 1000];
16+
17+/** Whether a request can be sent again as it was: no body, nothing changed. */
18+export function replayable(request: Request): boolean {
19+ return (request.method === "GET" || request.method === "HEAD") && request.body === null;
20+}
21+
22+/** What a request that could not be sent gets back. */
23+export function upstreamFailed(host: string, error: unknown): Response {
24+ const why = error instanceof Error ? error.message : String(error);
25+ return new Response(`g1t egress: ${normalizeHost(host)} could not be reached: ${why}\n`, {
26+ status: 502,
27+ headers: { "content-type": "text/plain; charset=utf-8", "x-g1t-egress": "upstream-failed" },
28+ });
29+}
30+
31+const wait = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));
32+
33+/**
34+ * Sends `request` with `send` (`fetch`), trying a replayable one again on
35+ * a thrown error, and answering `502` when it cannot. Each failure is one
36+ * line of the Worker's log.
37+ */
38+export async function forward(
39+ request: Request,
40+ send: (request: Request) => Promise<Response> = fetch,
41+ sleep: (ms: number) => Promise<void> = wait,
42+): Promise<Response> {
43+ const host = new URL(request.url).host;
44+ const tries = replayable(request) ? RETRY_WAITS_MS.length + 1 : 1;
45+ let failure: unknown = null;
46+ for (let attempt = 1; attempt <= tries; attempt++) {
47+ if (attempt > 1) await sleep(RETRY_WAITS_MS[attempt - 2]);
48+ try {
49+ return await send(request);
50+ } catch (error) {
51+ failure = error;
52+ console.log("egress upstream failed", normalizeHost(host), request.method, `try ${attempt} of ${tries}`, String(error));
53+ }
54+ }
55+ return upstreamFailed(host, failure);
56+}
+2−1
2727 sandboxNamespace,
2828 } from "./egress";
2929 import { type BrokeredParams, brokered } from "./broker-egress.ts";
30+import { forward } from "./forward.ts";
3031
3132 export {
3233 ABUSE_EXIT_CODE,
110111 if (host === ABUSE_HOST) return abuse(request, env, ctx);
111112 if (ctx.params?.open || allows(ctx.params?.hosts ?? [], host)) {
112113 // A workflow job's brokered secrets are filled in first, if it has any.
113− return (await brokered(request, env, ctx)) ?? fetch(request);
114+ return (await brokered(request, env, ctx)) ?? forward(request);
114115 }
115116 try {
116117 const namespace = sandboxNamespace(env, ctx.className);

Comments

Sign in to comment.