Merge g1tusercontent.com: registry answers run nothing in a browser, the site's pages run only their own scripts, repository files and avatars on their own origin, raw files rate limited per address
42 files+1116−1570/42 viewed
| 80 | 80 | files, not the history; clone for that. A repository with more than 10,000 | |
| 81 | 81 | files or over 24 MB is too large to download this way, so clone it instead. | |
| 82 | 82 | ||
| 83 | + | ## Raw files | |
| 84 | + | ||
| 85 | + | **Raw**, above a file on its page, opens the file as it is, with nothing | |
| 86 | + | around it. Any file is at: | |
| 87 | + | ||
| 88 | + | ```text | |
| 89 | + | https://g1t.sh/<workspace>/<repo>/raw/<branch, tag or commit>/<path> | |
| 90 | + | ``` | |
| 91 | + | ||
| 92 | + | That address sends you on to the commit the branch or tag names now, on | |
| 93 | + | g1t's file host: | |
| 94 | + | ||
| 95 | + | ```text | |
| 96 | + | https://g1tusercontent.com/<workspace>/<repo>/raw/<commit>/<path> | |
| 97 | + | ``` | |
| 98 | + | ||
| 99 | + | g1tusercontent.com is a site of its own so that nothing in a repository | |
| 100 | + | can reach your g1t.sh session: it never receives g1t.sh's cookies, and a | |
| 101 | + | file opened there cannot run script. Images, video, audio and PDFs are | |
| 102 | + | served as themselves; any other text, HTML, SVG source, XML and | |
| 103 | + | JavaScript included, as plain text; anything else as a download. A file | |
| 104 | + | is served up to 10 MB; clone the repository for larger ones. | |
| 105 | + | ||
| 106 | + | - **Public repositories.** The address works for anyone, and an address | |
| 107 | + | at a commit can be kept for good. | |
| 108 | + | - **Private repositories.** The address carries a `token` that g1t.sh | |
| 109 | + | makes for someone who can read the repository. It is good for that one | |
| 110 | + | file for an hour or two; after that, open the file on g1t.sh again for a | |
| 111 | + | new address. Revoking someone's access does not end an address they | |
| 112 | + | already have before then. | |
| 113 | + | ||
| 114 | + | Images in a file's page show from its raw address, and so do pictures in a | |
| 115 | + | README that name a file in the repository (``, | |
| 116 | + | relative to the README's folder, or `/docs/diagram.png` from the | |
| 117 | + | repository's root): they show the file at the commit the page shows. | |
| 118 | + | Uploaded avatars are on the same host, at | |
| 119 | + | `https://g1tusercontent.com/avatars/<sha256>`. | |
| 120 | + | ||
| 83 | 121 | ## Browsing without an account | |
| 84 | 122 | ||
| 85 | 123 | Public projects, Explore, Search and profiles are open to everyone, in the |
| 192 | 192 | tarball, a crate, a Maven artifact (not its POM or signatures), a NuGet | |
| 193 | 193 | `.nupkg`, a gem, and a Composer zip. | |
| 194 | 194 | ||
| 195 | + | A package's files are its publisher's, so the registries never let a | |
| 196 | + | browser run them. Every registry answer carries | |
| 197 | + | `X-Content-Type-Options: nosniff` and | |
| 198 | + | `Content-Security-Policy: default-src 'none'; sandbox`, and a file a | |
| 199 | + | browser would open as a page, such as a POM, a `.nuspec` or anything else | |
| 200 | + | in XML, HTML or SVG, comes with `Content-Disposition: attachment`, so it | |
| 201 | + | downloads instead. Package managers ignore these headers. | |
| 202 | + | ||
| 195 | 203 | ## The API | |
| 196 | 204 | ||
| 197 | 205 | The [REST API](/reference/api/) and the `package` tool of the |
| 153 | 153 | | `API_PORT` | `8789` | The port the API and the MCP server are published on | | |
| 154 | 154 | | `API_URL` | `PUBLIC_URL`'s host on `API_PORT` | The address of the API, as people and applications reach it. It is also the OAuth issuer. Set it when the API is behind a proxy, for example `https://api.git.example.com`. | | |
| 155 | 155 | | `MCP_URL` | `API_URL/mcp` | The address of the MCP server. | | |
| 156 | + | | `USERCONTENT_URL` | `PUBLIC_URL/-/usercontent` | Where [raw files](/guides/git/#raw-files) and avatars are served. Set it to a host of its own (another domain, not a subdomain of `PUBLIC_URL`'s, for example `https://files.example.net`) that your proxy sends to the same port as `PUBLIC_URL`, keeping the `Host` header, so a file in a repository can never reach the site's session cookie. Unset, they are served under `PUBLIC_URL`, still as plain text or images that cannot run script. | | |
| 156 | 157 | | `MAILPIT_PORT` | `8025` | The port of the Mailpit inbox | | |
| 157 | 158 | | `MAIL_FROM` | `g1t <noreply@localhost>` | The sender of g1t's email | | |
| 158 | 159 | | `MAIL_URL` | `http://mailpit:8025` | The Mailpit server g1t sends mail through | |
| 80 | 80 | ||
| 81 | 81 | The icon then shows wherever the workspace does, and on its link previews | |
| 82 | 82 | (PNG and JPEG icons only). Each image is served from | |
| 83 | − | `g1t.sh/avatars/<sha256>`, an address named after its contents, so an icon | |
| 83 | + | `g1tusercontent.com/avatars/<sha256>`, an address named after its contents, so an icon | |
| 84 | 84 | that changes gets a new address and nothing shows the old one. | |
| 85 | 85 | ||
| 86 | 86 | You can upload a picture of yourself the same way, under |
| 22 | 22 | | Pages on g1t.sh, signed in | Session | 1,200 | | |
| 23 | 23 | | Pages on g1t.sh, signed out | Client IP address | 600 | | |
| 24 | 24 | | Archive downloads, workflow run pages, logs and search, signed out | Client IP address | 30 | | |
| 25 | + | | [Raw files](/guides/git/#raw-files) on g1tusercontent.com | Client IP address, together with pages signed out | 600 | | |
| 25 | 26 | | Container and package registries | See [storage and pull limits](/guides/containers/#storage-and-pull-limits) | | | |
| 26 | 27 | ||
| 27 | 28 | The REST API and the MCP server count apart: calls to one do not use up | |
| ⋯ | |||
| 50 | 51 | | --- | --- | | |
| 51 | 52 | | REST API and MCP server | JSON in the [error shape](/reference/api/#errors) every endpoint uses, with `code` `rate_limited`. Through MCP it comes back as the HTTP answer to the request, not as a tool result. | | |
| 52 | 53 | | Git over HTTPS | Plain text, which git prints after `remote:` or in its error. | | |
| 53 | − | | Pages on g1t.sh | Plain text. | | |
| 54 | + | | Pages on g1t.sh, and raw files | Plain text. | | |
| 54 | 55 | ||
| 55 | 56 | Branch on the `429` status or the `rate_limited` code, never on the | |
| 56 | 57 | message. The API sends `Access-Control-Expose-Headers: retry-after`, so a | |
| 17 | 17 | ||
| 18 | 18 | import { Checkbox } from "./ui/checkbox"; | |
| 19 | 19 | import { type AlertKind, G1T_MENTION_HREF, type MarkdownRepo, rehypeAlerts, rehypeReferences } from "../lib/markdown-plugins"; | |
| 20 | + | import { imageSource } from "../lib/usercontent"; | |
| 20 | 21 | import { UserCard } from "./user-card"; | |
| 21 | 22 | ||
| 22 | 23 | /** The text inside a React tree, for anchors and copying. */ | |
| ⋯ | |||
| 160 | 161 | source, | |
| 161 | 162 | repo, | |
| 162 | 163 | base, | |
| 164 | + | rawBase, | |
| 163 | 165 | }: { | |
| 164 | 166 | source: string; | |
| 165 | 167 | /** The repository the text belongs to, for its references. */ | |
| 166 | 168 | repo?: MarkdownRepo; | |
| 167 | 169 | /** Where relative links point, e.g. `/acme/web/blob/main/docs` for a file's own folder. */ | |
| 168 | 170 | base?: string; | |
| 171 | + | /** | |
| 172 | + | * Where relative images point: the same folder's raw files, e.g. | |
| 173 | + | * `/acme/web/raw/<commit>/docs`, under `/acme/web/raw/<commit>`. An image | |
| 174 | + | * path starting with `/` is from the repository's root. | |
| 175 | + | */ | |
| 176 | + | rawBase?: string; | |
| 169 | 177 | }) { | |
| 170 | 178 | return ( | |
| 171 | 179 | <div className="prose"> | |
| ⋯ | |||
| 247 | 255 | ) : null; | |
| 248 | 256 | }, | |
| 249 | 257 | img({ src, alt }) { | |
| 250 | − | return <img src={typeof src === "string" ? src : undefined} alt={alt ?? ""} loading="lazy" className="inline max-w-full rounded" />; | |
| 258 | + | const at = typeof src === "string" ? imageSource(src, rawBase) : undefined; | |
| 259 | + | return <img src={at} alt={alt ?? ""} loading="lazy" className="inline max-w-full rounded" />; | |
| 251 | 260 | }, | |
| 252 | 261 | }} | |
| 253 | 262 | > | |
| 64 | 64 | </div> | |
| 65 | 65 | <div className="px-5 py-4"> | |
| 66 | 66 | {release.body.trim() ? ( | |
| 67 | − | <Markdown source={release.body} repo={repo} base={`${base}/blob/${encodeTag(release.tagName)}`} /> | |
| 67 | + | <Markdown | |
| 68 | + | source={release.body} | |
| 69 | + | repo={repo} | |
| 70 | + | base={`${base}/blob/${encodeTag(release.tagName)}`} | |
| 71 | + | rawBase={`${base}/raw/${encodeURIComponent(release.tagName)}`} | |
| 72 | + | /> | |
| 68 | 73 | ) : ( | |
| 69 | 74 | <p className="text-sm text-faint">No notes.</p> | |
| 70 | 75 | )} |
| 15 | 15 | ||
| 16 | 16 | import { AgentSetup } from "./agent-setup"; | |
| 17 | 17 | import { useAddresses } from "../lib/addresses"; | |
| 18 | + | import { isImagePath } from "../lib/usercontent"; | |
| 18 | 19 | import { CloneBox } from "./clone-box"; | |
| 19 | 20 | import { type ChecksSource, CommitChecksBadge } from "./commit-checks"; | |
| 20 | 21 | import { type AboutData, RepoAboutPanel } from "./repo-about"; | |
| ⋯ | |||
| 370 | 371 | <Markdown | |
| 371 | 372 | source={readme.text} | |
| 372 | 373 | repo={{ namespace: repo.namespace, name: repo.name }} | |
| 373 | − | // Relative links in a README point into the repository. | |
| 374 | + | // Relative links in a README point into the repository, | |
| 375 | + | // and its pictures at the files of the commit shown. | |
| 374 | 376 | base={`/${repo.namespace}/${repo.name}/blob/${ref}${path ? `/${path}` : ""}`} | |
| 377 | + | rawBase={`/${repo.namespace}/${repo.name}/raw/${head.hash}${path ? `/${encodePath(path)}` : ""}`} | |
| 375 | 378 | /> | |
| 376 | 379 | ) : ( | |
| 377 | 380 | <pre className="whitespace-pre-wrap text-sm"><code>{readme.text}</code></pre> | |
| ⋯ | |||
| 419 | 422 | const { repo, ref, path, size, text } = blob; | |
| 420 | 423 | const lines = text?.replace(/\n$/, "").split("\n"); | |
| 421 | 424 | const base = `/${repo.namespace}/${repo.name}`; | |
| 425 | + | // The file as it is, on the usercontent origin (routes/repo/raw.ts). | |
| 426 | + | const raw = `${base}/raw/${encodeURIComponent(ref)}/${encodePath(path)}`; | |
| 422 | 427 | const toggle = (label: string, on: boolean, search: string) => ( | |
| 423 | 428 | <Link | |
| 424 | 429 | to={{ search }} | |
| ⋯ | |||
| 445 | 450 | <div className="flex items-center gap-3 border-b border-line bg-surface px-4 py-2.5 text-xs text-muted"> | |
| 446 | 451 | {lines && <span>{lines.length.toLocaleString("en-US")} lines</span>} | |
| 447 | 452 | <span>{size.toLocaleString("en-US")} bytes</span> | |
| 448 | − | {lines && ( | |
| 449 | − | <span className="ml-auto flex rounded-md border border-line p-0.5"> | |
| 450 | − | {toggle("Code", !blame, "")} | |
| 451 | − | {toggle("Blame", Boolean(blame), "?blame=1")} | |
| 452 | − | </span> | |
| 453 | − | )} | |
| 453 | + | <span className="ml-auto flex items-center gap-2"> | |
| 454 | + | {lines && ( | |
| 455 | + | <span className="flex rounded-md border border-line p-0.5"> | |
| 456 | + | {toggle("Code", !blame, "")} | |
| 457 | + | {toggle("Blame", Boolean(blame), "?blame=1")} | |
| 458 | + | </span> | |
| 459 | + | )} | |
| 460 | + | <a href={raw} className="rounded-md border border-line px-2 py-1 transition-colors hover:text-fg"> | |
| 461 | + | Raw | |
| 462 | + | </a> | |
| 463 | + | </span> | |
| 454 | 464 | </div> | |
| 455 | 465 | {blame && lines ? ( | |
| 456 | 466 | <BlameView base={base} path={path} lines={lines} html={blame.lines} blame={blame.blame} /> | |
| 457 | 467 | ) : lines ? ( | |
| 458 | 468 | <CodeLines lines={lines} html={html} marked={marked} /> | |
| 469 | + | ) : isImagePath(path) ? ( | |
| 470 | + | <div className="flex justify-center bg-[repeating-conic-gradient(var(--color-raised)_0_25%,transparent_0_50%)] bg-[length:16px_16px] p-6"> | |
| 471 | + | <img src={raw} alt={path.split("/").pop() ?? path} className="max-h-[70vh] max-w-full" /> | |
| 472 | + | </div> | |
| 459 | 473 | ) : ( | |
| 460 | 474 | <p className="p-6 text-sm text-muted"> | |
| 461 | − | This file is binary or too large to show. | |
| 475 | + | This file is binary or too large to show.{" "} | |
| 476 | + | <a href={raw} className="text-accent hover:underline"> | |
| 477 | + | View it raw | |
| 478 | + | </a> | |
| 479 | + | . | |
| 462 | 480 | </p> | |
| 463 | 481 | )} | |
| 464 | 482 | </div> | |
| 1 | 1 | import { Check, Copy, LoaderCircle, User } from "lucide-react"; | |
| 2 | 2 | import { type ComponentProps, Fragment, type ReactNode, useState } from "react"; | |
| 3 | − | import { Link, type LinkProps, NavLink, useLocation, useNavigation } from "react-router"; | |
| 3 | + | import { Link, type LinkProps, NavLink, useLocation, useNavigation, useRouteLoaderData } from "react-router"; | |
| 4 | 4 | ||
| 5 | + | import { usercontentFrom } from "../../lib/addresses"; | |
| 5 | 6 | import { isWaitingMessage, linkPaths } from "../../lib/compute"; | |
| 6 | 7 | import { type Submission, isPending } from "../../lib/pending"; | |
| 7 | 8 | import { Mark } from "../logo"; | |
| ⋯ | |||
| 278 | 279 | return name === "g1t"; | |
| 279 | 280 | } | |
| 280 | 281 | ||
| 281 | − | /** Where an uploaded avatar is served, from the hash it is stored by. */ | |
| 282 | − | export function avatarUrl(avatar: string): string { | |
| 283 | − | return `/avatars/${avatar}`; | |
| 282 | + | /** | |
| 283 | + | * Where an uploaded avatar is served, from the hash it is stored by: the | |
| 284 | + | * usercontent origin, or the site's own address (which redirects there) | |
| 285 | + | * when it is not known. | |
| 286 | + | */ | |
| 287 | + | export function avatarUrl(avatar: string, usercontent = ""): string { | |
| 288 | + | return `${usercontent}/avatars/${avatar}`; | |
| 284 | 289 | } | |
| 285 | 290 | ||
| 286 | 291 | /** | |
| ⋯ | |||
| 304 | 309 | system?: boolean; | |
| 305 | 310 | }) { | |
| 306 | 311 | const [failed, setFailed] = useState<string | null>(null); | |
| 312 | + | const usercontent = usercontentFrom(useRouteLoaderData("root")); | |
| 307 | 313 | // g1t itself wears its own mark: the pixel 1 on a dark square. | |
| 308 | 314 | if (system || isSystemName(name)) { | |
| 309 | 315 | return ( | |
| ⋯ | |||
| 332 | 338 | if (image && failed !== image) { | |
| 333 | 339 | return ( | |
| 334 | 340 | <img | |
| 335 | − | src={avatarUrl(image)} | |
| 341 | + | src={avatarUrl(image, usercontent)} | |
| 336 | 342 | alt="" | |
| 337 | 343 | aria-hidden="true" | |
| 338 | 344 | width={size} | |
| 3 | 3 | import { isbot } from "isbot"; | |
| 4 | 4 | import { renderToReadableStream } from "react-dom/server"; | |
| 5 | 5 | ||
| 6 | + | import { addresses } from "./lib/addresses.server"; | |
| 7 | + | import { NonceContext } from "./lib/nonce"; | |
| 8 | + | import { makeNonce, pagePolicy } from "./lib/page-headers"; | |
| 6 | 9 | import { recordHandler } from "./lib/perf.server"; | |
| 7 | 10 | ||
| 8 | 11 | // React Router's own server entry, plus the timing of every loader and | |
| ⋯ | |||
| 47 | 50 | let shellRendered = false; | |
| 48 | 51 | const userAgent = request.headers.get("user-agent"); | |
| 49 | 52 | ||
| 50 | − | const body = await renderToReadableStream(<ServerRouter context={routerContext} url={request.url} />, { | |
| 51 | − | signal: AbortSignal.timeout(streamTimeout + 1000), | |
| 52 | − | onError(error: unknown) { | |
| 53 | − | responseStatusCode = 500; | |
| 54 | − | // Errors while streaming after the shell; those in the shell reject | |
| 55 | − | // and are logged by React Router. | |
| 56 | − | if (shellRendered) { | |
| 57 | − | console.error(error); | |
| 58 | − | } | |
| 53 | + | // The page's inline scripts carry this nonce, and its policy allows only | |
| 54 | + | // them (lib/page-headers.ts). Not in development, where Vite adds its own. | |
| 55 | + | const nonce = import.meta.env.DEV ? undefined : makeNonce(); | |
| 56 | + | const body = await renderToReadableStream( | |
| 57 | + | <NonceContext value={nonce}> | |
| 58 | + | <ServerRouter context={routerContext} url={request.url} nonce={nonce} /> | |
| 59 | + | </NonceContext>, | |
| 60 | + | { | |
| 61 | + | nonce, | |
| 62 | + | signal: AbortSignal.timeout(streamTimeout + 1000), | |
| 63 | + | onError(error: unknown) { | |
| 64 | + | responseStatusCode = 500; | |
| 65 | + | // Errors while streaming after the shell; those in the shell reject | |
| 66 | + | // and are logged by React Router. | |
| 67 | + | if (shellRendered) { | |
| 68 | + | console.error(error); | |
| 69 | + | } | |
| 70 | + | }, | |
| 59 | 71 | }, | |
| 60 | − | }); | |
| 72 | + | ); | |
| 61 | 73 | shellRendered = true; | |
| 62 | 74 | ||
| 63 | 75 | // Crawlers get the whole page at once, deferred panels included. | |
| ⋯ | |||
| 66 | 78 | } | |
| 67 | 79 | ||
| 68 | 80 | responseHeaders.set("Content-Type", "text/html"); | |
| 81 | + | if (nonce) responseHeaders.set("Content-Security-Policy", pagePolicy(nonce, addresses().usercontent)); | |
| 69 | 82 | return new Response(body, { | |
| 70 | 83 | headers: responseHeaders, | |
| 71 | 84 | status: responseStatusCode, | |
| 2 | 2 | ||
| 3 | 3 | import { type Addresses, addressesFor } from "./addresses"; | |
| 4 | 4 | ||
| 5 | − | /** This g1t's addresses, from the Worker's settings (SITE_URL, API_URL, MCP_URL, OG_URL). */ | |
| 5 | + | /** This g1t's addresses, from the Worker's settings (SITE_URL, API_URL, MCP_URL, OG_URL, USERCONTENT_URL). */ | |
| 6 | 6 | export function addresses(): Addresses { | |
| 7 | 7 | return addressesFor(env); | |
| 8 | 8 | } |
| 1 | 1 | import assert from "node:assert/strict"; | |
| 2 | 2 | import { test } from "node:test"; | |
| 3 | 3 | ||
| 4 | − | import { HOSTED_ADDRESSES, addressesFor, addressesFrom, cloneUrl } from "./addresses.ts"; | |
| 4 | + | import { HOSTED_ADDRESSES, addressesFor, addressesFrom, cloneUrl, usercontentFrom } from "./addresses.ts"; | |
| 5 | 5 | import { MCP_URL } from "./agent-setup.ts"; | |
| 6 | 6 | import { OG, SITE } from "./meta.ts"; | |
| 7 | 7 | ||
| ⋯ | |||
| 18 | 18 | MCP_URL: "http://localhost:8790/mcp/", | |
| 19 | 19 | OG_URL: "", | |
| 20 | 20 | }), | |
| 21 | − | { site: "http://localhost:8787", api: "http://localhost:8788", mcp: "http://localhost:8790/mcp", og: null }, | |
| 21 | + | { | |
| 22 | + | site: "http://localhost:8787", | |
| 23 | + | api: "http://localhost:8788", | |
| 24 | + | mcp: "http://localhost:8790/mcp", | |
| 25 | + | og: null, | |
| 26 | + | usercontent: "http://localhost:8787/-/usercontent", | |
| 27 | + | }, | |
| 22 | 28 | ); | |
| 23 | 29 | }); | |
| 24 | 30 | ||
| 31 | + | test("files people supply are served from an origin of their own", () => { | |
| 32 | + | assert.equal(addressesFor({}).usercontent, "https://g1tusercontent.com"); | |
| 33 | + | assert.equal(addressesFor({ SITE_URL: "https://git.example.com", USERCONTENT_URL: "https://files.example.net/" }).usercontent, "https://files.example.net"); | |
| 34 | + | assert.equal(addressesFor({ SITE_URL: "https://git.example.com", USERCONTENT_URL: "https://git.example.com" }).usercontent, "https://git.example.com/-/usercontent"); | |
| 35 | + | assert.equal(usercontentFrom(undefined), ""); | |
| 36 | + | assert.equal(usercontentFrom({ addresses: HOSTED_ADDRESSES }), "https://g1tusercontent.com"); | |
| 37 | + | }); | |
| 38 | + | ||
| 25 | 39 | test("pages without root data use g1t.sh's addresses", () => { | |
| 26 | 40 | assert.deepEqual(addressesFrom(undefined), HOSTED_ADDRESSES); | |
| 27 | 41 | const own = { site: "http://localhost:8787", api: "a", mcp: "m", og: null }; | |
| 15 | 15 | mcp: string; | |
| 16 | 16 | /** The social-card image service's origin, or null when there is none. */ | |
| 17 | 17 | og: string | null; | |
| 18 | + | /** | |
| 19 | + | * Where bytes people supplied are served (repository files, avatars), on | |
| 20 | + | * an origin of its own that never sees the site's cookies. Self-hosted | |
| 21 | + | * without a host of its own, a path on the site (`<site>/-/usercontent`). | |
| 22 | + | */ | |
| 23 | + | usercontent: string; | |
| 18 | 24 | }; | |
| 19 | 25 | ||
| 20 | 26 | export const HOSTED_ADDRESSES: Addresses = { | |
| ⋯ | |||
| 22 | 28 | api: "https://api.g1t.sh", | |
| 23 | 29 | mcp: "https://mcp.g1t.sh", | |
| 24 | 30 | og: "https://og.g1t.sh", | |
| 31 | + | usercontent: "https://g1tusercontent.com", | |
| 25 | 32 | }; | |
| 26 | 33 | ||
| 27 | 34 | /** The Worker settings the addresses come from; every one optional. */ | |
| 28 | − | export type AddressSettings = { SITE_URL?: string; API_URL?: string; MCP_URL?: string; OG_URL?: string }; | |
| 35 | + | export type AddressSettings = { SITE_URL?: string; API_URL?: string; MCP_URL?: string; OG_URL?: string; USERCONTENT_URL?: string }; | |
| 29 | 36 | ||
| 30 | 37 | const trim = (value: string) => value.trim().replace(/\/+$/, ""); | |
| 31 | 38 | ||
| 32 | 39 | /** | |
| 33 | 40 | * The addresses from the Worker's settings. An unset or empty setting is | |
| 34 | 41 | * g1t.sh's address, except `OG_URL`: set to an empty string, it means there | |
| 35 | − | * is no card service, and pages carry no image tags. | |
| 42 | + | * is no card service, and pages carry no image tags. `USERCONTENT_URL` | |
| 43 | + | * unset is g1tusercontent.com on g1t.sh, and a path on the site wherever | |
| 44 | + | * `SITE_URL` names another. | |
| 36 | 45 | */ | |
| 37 | 46 | export function addressesFor(settings: AddressSettings): Addresses { | |
| 38 | 47 | const pick = (value: string | undefined, fallback: string) => (value ? trim(value) : "") || fallback; | |
| 48 | + | const site = pick(settings.SITE_URL, HOSTED_ADDRESSES.site); | |
| 49 | + | const ownSite = site === HOSTED_ADDRESSES.site; | |
| 39 | 50 | return { | |
| 40 | − | site: pick(settings.SITE_URL, HOSTED_ADDRESSES.site), | |
| 51 | + | site, | |
| 52 | + | usercontent: usercontentOf(pick(settings.USERCONTENT_URL, ownSite ? HOSTED_ADDRESSES.usercontent : `${site}/-/usercontent`), site), | |
| 41 | 53 | api: pick(settings.API_URL, HOSTED_ADDRESSES.api), | |
| 42 | 54 | mcp: pick(settings.MCP_URL, HOSTED_ADDRESSES.mcp), | |
| 43 | 55 | og: settings.OG_URL === undefined ? HOSTED_ADDRESSES.og : trim(settings.OG_URL) || null, | |
| 44 | 56 | }; | |
| 45 | 57 | } | |
| 46 | 58 | ||
| 59 | + | /** | |
| 60 | + | * Where uploaded avatars and repository files are served, from the root | |
| 61 | + | * loader's data; the site's own paths when it has none (they redirect). | |
| 62 | + | */ | |
| 63 | + | export function usercontentFrom(rootData: unknown): string { | |
| 64 | + | return (rootData as { addresses?: Addresses } | null | undefined)?.addresses?.usercontent ?? ""; | |
| 65 | + | } | |
| 66 | + | ||
| 67 | + | /** The site's own origin is never the usercontent origin: a path on it is. */ | |
| 68 | + | function usercontentOf(address: string, site: string): string { | |
| 69 | + | return address === site ? `${site}/-/usercontent` : address; | |
| 70 | + | } | |
| 71 | + | ||
| 47 | 72 | /** The addresses in the root loader's data, or g1t.sh's when it has none. */ | |
| 48 | 73 | export function addressesFrom(rootData: unknown): Addresses { | |
| 49 | 74 | return (rootData as { addresses?: Addresses } | null | undefined)?.addresses ?? HOSTED_ADDRESSES; | |
| 1 | + | import assert from "node:assert/strict"; | |
| 2 | + | import { test } from "node:test"; | |
| 3 | + | ||
| 4 | + | import { contentDisposition, hardenRegistryHeaders, NOTHING_RUNS, opensAsDocument } from "./content-safety.ts"; | |
| 5 | + | ||
| 6 | + | test("publisher documents a browser would open are downloads", () => { | |
| 7 | + | for (const type of [ | |
| 8 | + | "application/xml", | |
| 9 | + | "text/xml; charset=utf-8", | |
| 10 | + | "application/xhtml+xml", | |
| 11 | + | "text/html", | |
| 12 | + | "image/svg+xml", | |
| 13 | + | "application/vnd.example+xml", | |
| 14 | + | "text/javascript", | |
| 15 | + | "", | |
| 16 | + | null, | |
| 17 | + | ]) { | |
| 18 | + | assert.equal(opensAsDocument(type), true, String(type)); | |
| 19 | + | } | |
| 20 | + | }); | |
| 21 | + | ||
| 22 | + | test("data types stay inline", () => { | |
| 23 | + | for (const type of [ | |
| 24 | + | "application/json", | |
| 25 | + | "text/plain; charset=utf-8", | |
| 26 | + | "application/vnd.oci.image.manifest.v1+json", | |
| 27 | + | "application/vnd.npm.install-v1+json", | |
| 28 | + | "application/octet-stream", | |
| 29 | + | "application/java-archive", | |
| 30 | + | "application/gzip", | |
| 31 | + | "image/png", | |
| 32 | + | ]) { | |
| 33 | + | assert.equal(opensAsDocument(type), false, type); | |
| 34 | + | } | |
| 35 | + | }); | |
| 36 | + | ||
| 37 | + | test("every registry answer runs nothing and is never sniffed", () => { | |
| 38 | + | const pom = new Headers({ "content-type": "application/xml" }); | |
| 39 | + | hardenRegistryHeaders(pom); | |
| 40 | + | assert.equal(pom.get("x-content-type-options"), "nosniff"); | |
| 41 | + | assert.equal(pom.get("content-security-policy"), NOTHING_RUNS); | |
| 42 | + | assert.equal(pom.get("content-disposition"), "attachment"); | |
| 43 | + | ||
| 44 | + | const json = new Headers({ "content-type": "application/json" }); | |
| 45 | + | hardenRegistryHeaders(json); | |
| 46 | + | assert.equal(json.get("content-security-policy"), NOTHING_RUNS); | |
| 47 | + | assert.equal(json.get("content-disposition"), null); | |
| 48 | + | ||
| 49 | + | const named = new Headers({ "content-type": "text/html", "content-disposition": 'attachment; filename="a.html"' }); | |
| 50 | + | hardenRegistryHeaders(named); | |
| 51 | + | assert.equal(named.get("content-disposition"), 'attachment; filename="a.html"'); | |
| 52 | + | }); | |
| 53 | + | ||
| 54 | + | test("download names keep quotes and control characters out of the header", () => { | |
| 55 | + | assert.equal(contentDisposition("web-main.zip"), `attachment; filename="web-main.zip"; filename*=UTF-8''web-main.zip`); | |
| 56 | + | const sly = contentDisposition('web-a"b\r\nSet-Cookie: x.zip'); | |
| 57 | + | assert.ok(!/[\r\n]/.test(sly)); | |
| 58 | + | assert.match(sly, /^attachment; filename="web-a_b__Set-Cookie: x.zip"; filename\*=UTF-8''web-a%22b__Set-Cookie%3A%20x.zip$/); | |
| 59 | + | assert.match(contentDisposition("café.zip"), /filename="caf_.zip"; filename\*=UTF-8''caf%C3%A9.zip$/); | |
| 60 | + | }); |
| 1 | + | /** | |
| 2 | + | * Headers that keep bytes someone else wrote from running as a page. | |
| 3 | + | * | |
| 4 | + | * The package registries answer on the site's own origin, and what they | |
| 5 | + | * serve (a POM, a nuspec, a manifest) is the publisher's. Every registry | |
| 6 | + | * answer is told never to be sniffed, to run nothing and to load nothing, | |
| 7 | + | * and one a browser would open as a document is a download instead. The | |
| 8 | + | * clients the registries serve ignore all three headers. | |
| 9 | + | */ | |
| 10 | + | ||
| 11 | + | /** Nothing loads and nothing runs: the policy for bytes that are only ever data. */ | |
| 12 | + | export const NOTHING_RUNS = "default-src 'none'; sandbox"; | |
| 13 | + | ||
| 14 | + | /** | |
| 15 | + | * Types a browser shows as data, never as a page: JSON, plain text, | |
| 16 | + | * archives and checked images. Anything else (HTML, SVG, any XML, an | |
| 17 | + | * unknown or missing type) could become a page, so it is a download. | |
| 18 | + | */ | |
| 19 | + | const SHOWN_AS_DATA = [ | |
| 20 | + | /^text\/plain$/, | |
| 21 | + | /^application\/json$/, | |
| 22 | + | /^application\/[a-z0-9.+-]+\+json$/, | |
| 23 | + | /^application\/(?:octet-stream|gzip|x-gzip|zip|x-tar|java-archive|pgp-signature)$/, | |
| 24 | + | /^application\/vnd\.[a-z0-9.+-]+$/, | |
| 25 | + | /^image\/(?:png|jpeg|gif|webp|avif)$/, | |
| 26 | + | ]; | |
| 27 | + | ||
| 28 | + | /** The media type alone: lowercase, without parameters. */ | |
| 29 | + | export function mediaType(contentType: string | null | undefined): string { | |
| 30 | + | return (contentType ?? "").split(";")[0]!.trim().toLowerCase(); | |
| 31 | + | } | |
| 32 | + | ||
| 33 | + | /** Whether a browser could open a body of this type as a document. */ | |
| 34 | + | export function opensAsDocument(contentType: string | null | undefined): boolean { | |
| 35 | + | const type = mediaType(contentType); | |
| 36 | + | if (/\+xml$|\/xml$|xml-|html|svg|xsl/.test(type)) return true; | |
| 37 | + | return !SHOWN_AS_DATA.some((pattern) => pattern.test(type)); | |
| 38 | + | } | |
| 39 | + | ||
| 40 | + | /** | |
| 41 | + | * The headers a registry answer gains: no sniffing, a policy that runs | |
| 42 | + | * nothing, and, for a type a browser would open as a document, an | |
| 43 | + | * attachment. A disposition the service already set is kept. | |
| 44 | + | */ | |
| 45 | + | export function hardenRegistryHeaders(headers: Headers): void { | |
| 46 | + | headers.set("x-content-type-options", "nosniff"); | |
| 47 | + | headers.set("content-security-policy", NOTHING_RUNS); | |
| 48 | + | if (!headers.has("content-disposition") && opensAsDocument(headers.get("content-type"))) { | |
| 49 | + | headers.set("content-disposition", "attachment"); | |
| 50 | + | } | |
| 51 | + | } | |
| 52 | + | ||
| 53 | + | /** | |
| 54 | + | * A `Content-Disposition` for a download named `filename`: an ASCII | |
| 55 | + | * fallback with anything unsafe replaced, and the exact name as | |
| 56 | + | * RFC 6266's `filename*`. | |
| 57 | + | */ | |
| 58 | + | export function contentDisposition(filename: string): string { | |
| 59 | + | const fallback = filename.replace(/[^\x20-\x7e]|["\\%;]/g, "_") || "download"; | |
| 60 | + | const exact = encodeURIComponent(filename.replace(/[\x00-\x1f\x7f]/g, "_")).replace(/['()*]/g, (c) => `%${c.charCodeAt(0).toString(16).toUpperCase()}`); | |
| 61 | + | return `attachment; filename="${fallback}"; filename*=UTF-8''${exact}`; | |
| 62 | + | } |
| 11 | 11 | secretKey, | |
| 12 | 12 | } from "@g1t/contracts/rate-limits"; | |
| 13 | 13 | ||
| 14 | − | import { gitLimited, heavy, pageLimited, sessionCookie, unlimited } from "./front-door-limits.ts"; | |
| 14 | + | import { gitLimited, heavy, pageLimited, sessionCookie, unlimited, usercontentLimited } from "./front-door-limits.ts"; | |
| 15 | 15 | ||
| 16 | 16 | /** A binding that lets `allow` requests through per key, recording each key it was asked. */ | |
| 17 | 17 | function binding(allow: number): RateLimitBinding & { keys: string[] } { | |
| ⋯ | |||
| 176 | 176 | ids.add(spec.namespaceId); | |
| 177 | 177 | } | |
| 178 | 178 | }); | |
| 179 | + | ||
| 180 | + | test("repository files on the usercontent origin count per address; avatars do not", async () => { | |
| 181 | + | const env = { WEB_ANONYMOUS_LIMIT: binding(1) }; | |
| 182 | + | const raw = "/acme/rocket/raw/main/logo.png"; | |
| 183 | + | assert.equal(await usercontentLimited(env, request(raw), raw), null); | |
| 184 | + | assert.equal((await usercontentLimited(env, request(raw), raw))?.status, 429); | |
| 185 | + | assert.equal(await usercontentLimited(env, request("/avatars/" + "a".repeat(64)), "/avatars/" + "a".repeat(64)), null); | |
| 186 | + | assert.deepEqual(env.WEB_ANONYMOUS_LIMIT.keys, ["ip:203.0.113.9", "ip:203.0.113.9"]); | |
| 187 | + | }); | |
| 89 | 89 | if (!verdicts.includes("limited")) return null; | |
| 90 | 90 | return tooManyRequests(session ? PAGE_MESSAGE : `${PAGE_MESSAGE.trimEnd()} Signed-in accounts have a higher limit.\n`); | |
| 91 | 91 | } | |
| 92 | + | ||
| 93 | + | /** | |
| 94 | + | * The 429 for a repository file on the usercontent origin past its limit, | |
| 95 | + | * or null to go on. Nothing there is signed in (it never sees the session | |
| 96 | + | * cookie), so a file counts as a signed-out page from its address. Avatars | |
| 97 | + | * are answered from cache and are not limited. | |
| 98 | + | */ | |
| 99 | + | export async function usercontentLimited(env: FrontDoorLimits, request: Request, path: string): Promise<Response | null> { | |
| 100 | + | if (path.startsWith("/avatars/")) return null; | |
| 101 | + | const verdict = await checkLimit(env.WEB_ANONYMOUS_LIMIT, `ip:${clientAddress(request)}`); | |
| 102 | + | return verdict === "limited" ? tooManyRequests(PAGE_MESSAGE) : null; | |
| 103 | + | } |
| 1 | 1 | /** Answers for the log downloads: a file to save, or why there is none. */ | |
| 2 | 2 | ||
| 3 | + | import { contentDisposition } from "./content-safety"; | |
| 4 | + | ||
| 3 | 5 | export function refused(status: number, message: string): Response { | |
| 4 | 6 | return new Response(`${message}\n`, { status, headers: { "content-type": "text/plain; charset=utf-8", "cache-control": "no-store" } }); | |
| 5 | 7 | } | |
| ⋯ | |||
| 8 | 10 | return new Response(body, { | |
| 9 | 11 | headers: { | |
| 10 | 12 | "content-type": type, | |
| 11 | − | "content-disposition": `attachment; filename="${file.replace(/["\\\r\n]/g, "")}"`, | |
| 13 | + | "content-disposition": contentDisposition(file), | |
| 12 | 14 | "cache-control": "no-store", | |
| 13 | 15 | }, | |
| 14 | 16 | }); | |
| 1 | + | import { createContext, useContext } from "react"; | |
| 2 | + | ||
| 3 | + | /** | |
| 4 | + | * The nonce the page's inline scripts carry, which its Content-Security-Policy | |
| 5 | + | * names (lib/page-headers.ts). Given by entry.server.tsx; in the browser | |
| 6 | + | * there is none, and none is needed: the scripts have already run. | |
| 7 | + | */ | |
| 8 | + | export const NonceContext = createContext<string | undefined>(undefined); | |
| 9 | + | ||
| 10 | + | export function useNonce(): string | undefined { | |
| 11 | + | return useContext(NonceContext); | |
| 12 | + | } |
| 1 | + | import assert from "node:assert/strict"; | |
| 2 | + | import { test } from "node:test"; | |
| 3 | + | ||
| 4 | + | import { makeNonce, pagePolicy, withSiteHeaders } from "./page-headers.ts"; | |
| 5 | + | ||
| 6 | + | test("a page runs only its own scripts and is never framed", () => { | |
| 7 | + | const nonce = makeNonce(); | |
| 8 | + | assert.match(nonce, /^[A-Za-z0-9+/]{22}==$/); | |
| 9 | + | assert.notEqual(nonce, makeNonce()); | |
| 10 | + | const policy = pagePolicy(nonce); | |
| 11 | + | assert.ok(policy.includes(`script-src 'self' 'nonce-${nonce}'`)); | |
| 12 | + | assert.doesNotMatch(policy, /script-src[^;]*'unsafe-inline'/); | |
| 13 | + | assert.match(policy, /frame-ancestors 'none'/); | |
| 14 | + | assert.match(policy, /object-src 'none'/); | |
| 15 | + | assert.match(policy, /base-uri 'self'/); | |
| 16 | + | // Pictures in a README come from anywhere on HTTPS. | |
| 17 | + | assert.match(policy, /img-src 'self' https: data: blob:;/); | |
| 18 | + | // An installation serving its files over plain HTTP names that origin. | |
| 19 | + | assert.match(pagePolicy(nonce, "http://files.local:8787"), /img-src 'self' https: data: blob: http:\/\/files\.local:8787;/); | |
| 20 | + | assert.match(pagePolicy(nonce, "https://g1tusercontent.com"), /img-src 'self' https: data: blob:;/); | |
| 21 | + | }); | |
| 22 | + | ||
| 23 | + | test("every answer gains nosniff and a referrer policy; pages are not framed", () => { | |
| 24 | + | const page = withSiteHeaders(new Response("<p>hi</p>", { headers: { "content-type": "text/html" } })); | |
| 25 | + | assert.equal(page.headers.get("x-content-type-options"), "nosniff"); | |
| 26 | + | assert.equal(page.headers.get("referrer-policy"), "strict-origin-when-cross-origin"); | |
| 27 | + | assert.equal(page.headers.get("x-frame-options"), "DENY"); | |
| 28 | + | ||
| 29 | + | const data = withSiteHeaders(new Response("{}", { headers: { "content-type": "application/json" } })); | |
| 30 | + | assert.equal(data.headers.get("x-content-type-options"), "nosniff"); | |
| 31 | + | assert.equal(data.headers.get("x-frame-options"), null); | |
| 32 | + | ||
| 33 | + | // A redirect's headers are fixed; the answer is copied, status and all. | |
| 34 | + | const moved = withSiteHeaders(Response.redirect("https://docs.g1t.sh/", 301)); | |
| 35 | + | assert.equal(moved.status, 301); | |
| 36 | + | assert.equal(moved.headers.get("location"), "https://docs.g1t.sh/"); | |
| 37 | + | assert.equal(moved.headers.get("referrer-policy"), "strict-origin-when-cross-origin"); | |
| 38 | + | ||
| 39 | + | const own = withSiteHeaders(new Response("", { headers: { "referrer-policy": "no-referrer" } })); | |
| 40 | + | assert.equal(own.headers.get("referrer-policy"), "no-referrer"); | |
| 41 | + | }); |
| 1 | + | /** | |
| 2 | + | * The headers every answer from the site carries, and the policy its pages | |
| 3 | + | * run under. No Workers imports, so it can be tested under Node. | |
| 4 | + | * | |
| 5 | + | * - Nothing is sniffed: a download or a data request is only the type it says. | |
| 6 | + | * - A link to another site sends the origin, never the path. | |
| 7 | + | * - No other site may put g1t's pages in a frame. | |
| 8 | + | * - A page runs only the scripts the site served it: its own files, and the | |
| 9 | + | * inline scripts React and React Router write, each carrying the page's | |
| 10 | + | * nonce. Styles may be inline (highlighting and layout set them); images | |
| 11 | + | * may come from any HTTPS address (pictures in a README); requests may go | |
| 12 | + | * to any HTTPS address (the status page's summary). | |
| 13 | + | */ | |
| 14 | + | ||
| 15 | + | /** A fresh nonce for one page: 128 random bits, base64. */ | |
| 16 | + | export function makeNonce(): string { | |
| 17 | + | const bytes = crypto.getRandomValues(new Uint8Array(16)); | |
| 18 | + | return btoa(String.fromCharCode(...bytes)); | |
| 19 | + | } | |
| 20 | + | ||
| 21 | + | /** | |
| 22 | + | * The Content-Security-Policy of a page rendered with `nonce`. `usercontent` | |
| 23 | + | * is where repository files and avatars are served (lib/usercontent.ts), | |
| 24 | + | * named for an installation that serves them over plain HTTP. | |
| 25 | + | */ | |
| 26 | + | export function pagePolicy(nonce: string, usercontent?: string): string { | |
| 27 | + | const files = usercontent && /^http:/.test(usercontent) ? ` ${new URL(usercontent).origin}` : ""; | |
| 28 | + | return [ | |
| 29 | + | "default-src 'self'", | |
| 30 | + | // Cloudflare's Web Analytics beacon, when the zone turns it on. | |
| 31 | + | `script-src 'self' 'nonce-${nonce}' https://static.cloudflareinsights.com`, | |
| 32 | + | "style-src 'self' 'unsafe-inline'", | |
| 33 | + | `img-src 'self' https: data: blob:${files}`, | |
| 34 | + | `media-src 'self' https:${files}`, | |
| 35 | + | "font-src 'self' data:", | |
| 36 | + | "connect-src 'self' https:", | |
| 37 | + | "frame-src 'none'", | |
| 38 | + | "object-src 'none'", | |
| 39 | + | "base-uri 'self'", | |
| 40 | + | "frame-ancestors 'none'", | |
| 41 | + | ].join("; "); | |
| 42 | + | } | |
| 43 | + | ||
| 44 | + | /** | |
| 45 | + | * The answer with the site's headers added. A header the answer already | |
| 46 | + | * has is kept. An upgrade to a WebSocket is passed on as it is. | |
| 47 | + | */ | |
| 48 | + | export function withSiteHeaders(response: Response): Response { | |
| 49 | + | if (response.status === 101 || (response as Response & { webSocket?: unknown }).webSocket) return response; | |
| 50 | + | // A redirect's headers cannot be changed, so the answer is copied. | |
| 51 | + | const answer = new Response(response.body, response); | |
| 52 | + | const headers = answer.headers; | |
| 53 | + | if (!headers.has("x-content-type-options")) headers.set("x-content-type-options", "nosniff"); | |
| 54 | + | if (!headers.has("referrer-policy")) headers.set("referrer-policy", "strict-origin-when-cross-origin"); | |
| 55 | + | if (/^text\/html\b/i.test(headers.get("content-type") ?? "") && !headers.has("x-frame-options")) { | |
| 56 | + | headers.set("x-frame-options", "DENY"); | |
| 57 | + | } | |
| 58 | + | return answer; | |
| 59 | + | } |
| 1 | + | import assert from "node:assert/strict"; | |
| 2 | + | import { test } from "node:test"; | |
| 3 | + | ||
| 4 | + | import { | |
| 5 | + | PDF_POLICY, | |
| 6 | + | USERCONTENT_POLICY, | |
| 7 | + | imageSource, | |
| 8 | + | isCommit, | |
| 9 | + | parseRawPath, | |
| 10 | + | rawHeaders, | |
| 11 | + | rawPath, | |
| 12 | + | signRaw, | |
| 13 | + | usercontentPath, | |
| 14 | + | verifyRaw, | |
| 15 | + | } from "./usercontent.ts"; | |
| 16 | + | ||
| 17 | + | const FILE = { owner: "acme", repo: "web", ref: "feat/new", path: "docs/a b.png" }; | |
| 18 | + | ||
| 19 | + | test("a raw file's path keeps a ref with slashes in one segment", () => { | |
| 20 | + | const path = rawPath(FILE); | |
| 21 | + | assert.equal(path, "/acme/web/raw/feat%2Fnew/docs/a%20b.png"); | |
| 22 | + | assert.deepEqual(parseRawPath(path), FILE); | |
| 23 | + | }); | |
| 24 | + | ||
| 25 | + | test("paths that are not a file, or climb out of the repository, are refused", () => { | |
| 26 | + | for (const path of ["/acme/web/raw/main", "/acme/web/blob/main/a.png", "/acme/web/raw/main/../x", "/acme/web/raw/main/a//b", "/acme/web/raw/main/%E0%A4%A"]) { | |
| 27 | + | assert.equal(parseRawPath(path), null, path); | |
| 28 | + | } | |
| 29 | + | }); | |
| 30 | + | ||
| 31 | + | test("usercontent is its own host, or a path on the site", () => { | |
| 32 | + | const hosted = "https://g1tusercontent.com"; | |
| 33 | + | assert.equal(usercontentPath(new URL("https://g1tusercontent.com/acme/web/raw/main/a.png"), hosted), "/acme/web/raw/main/a.png"); | |
| 34 | + | assert.equal(usercontentPath(new URL("https://g1t.sh/acme/web/raw/main/a.png"), hosted), null); | |
| 35 | + | const own = "https://git.example.com/-/usercontent"; | |
| 36 | + | assert.equal(usercontentPath(new URL("http://localhost:8787/-/usercontent/avatars/ab"), own), "/avatars/ab"); | |
| 37 | + | assert.equal(usercontentPath(new URL("http://localhost:8787/-/usercontentx"), own), null); | |
| 38 | + | assert.equal(usercontentPath(new URL("http://localhost:8787/acme/web"), own), null); | |
| 39 | + | }); | |
| 40 | + | ||
| 41 | + | test("a token is good for its file alone, until it ends", async () => { | |
| 42 | + | const now = Date.UTC(2026, 9, 8, 12, 30); | |
| 43 | + | const token = await signRaw("secret", FILE, "repo_1", now); | |
| 44 | + | assert.match(token, /^\d+\.repo_1\.[A-Za-z0-9_-]{43}$/); | |
| 45 | + | // The same within the hour, so a page's addresses are kept by the browser. | |
| 46 | + | assert.equal(await signRaw("secret", FILE, "repo_1", now + 20 * 60_000), token); | |
| 47 | + | assert.equal(await verifyRaw("secret", FILE, token, now), "repo_1"); | |
| 48 | + | assert.equal(await verifyRaw("secret", { ...FILE, owner: "ACME" }, token, now), "repo_1"); | |
| 49 | + | assert.equal(await verifyRaw("secret", { ...FILE, path: "docs/other.png" }, token, now), null); | |
| 50 | + | assert.equal(await verifyRaw("secret", { ...FILE, repo: "api" }, token, now), null); | |
| 51 | + | assert.equal(await verifyRaw("secret", { ...FILE, ref: "main" }, token, now), null); | |
| 52 | + | assert.equal(await verifyRaw("other", FILE, token, now), null); | |
| 53 | + | assert.equal(await verifyRaw("secret", FILE, token.replace("repo_1", "repo_2"), now), null); | |
| 54 | + | assert.equal(await verifyRaw("secret", FILE, token, now + 2 * 3600_000), null); | |
| 55 | + | assert.equal(await verifyRaw("secret", FILE, "nonsense", now), null); | |
| 56 | + | }); | |
| 57 | + | ||
| 58 | + | test("files are served as data that cannot run", () => { | |
| 59 | + | const text = new TextEncoder().encode("<script>alert(1)</script>"); | |
| 60 | + | for (const name of ["index.html", "page.xhtml", "data.xml", "app.js", "README.md"]) { | |
| 61 | + | const headers = rawHeaders(name, text); | |
| 62 | + | assert.equal(headers.get("content-type"), "text/plain; charset=utf-8", name); | |
| 63 | + | assert.equal(headers.get("x-content-type-options"), "nosniff"); | |
| 64 | + | assert.equal(headers.get("content-security-policy"), USERCONTENT_POLICY); | |
| 65 | + | } | |
| 66 | + | assert.equal(rawHeaders("logo.png", new Uint8Array([137, 80, 78, 71])).get("content-type"), "image/png"); | |
| 67 | + | // An SVG shows as an image; opened on its own, its scripts are sandboxed. | |
| 68 | + | const svg = rawHeaders("logo.svg", text); | |
| 69 | + | assert.equal(svg.get("content-type"), "image/svg+xml"); | |
| 70 | + | assert.match(svg.get("content-security-policy")!, /sandbox/); | |
| 71 | + | assert.equal(rawHeaders("paper.pdf", new Uint8Array([37, 80])).get("content-security-policy"), PDF_POLICY); | |
| 72 | + | const binary = rawHeaders("tool.bin", new Uint8Array([0, 1, 2])); | |
| 73 | + | assert.equal(binary.get("content-type"), "application/octet-stream"); | |
| 74 | + | assert.match(binary.get("content-disposition")!, /^attachment; filename="tool.bin"/); | |
| 75 | + | }); | |
| 76 | + | ||
| 77 | + | test("a README's relative pictures are the repository's files at the same commit", () => { | |
| 78 | + | const root = "/acme/web/raw/abc123"; | |
| 79 | + | const docs = `${root}/docs`; | |
| 80 | + | assert.equal(imageSource("logo.png", root), `${root}/logo.png`); | |
| 81 | + | assert.equal(imageSource("./img/a b.png", docs), `${root}/docs/img/a%20b.png`); | |
| 82 | + | assert.equal(imageSource("../logo.png?raw=true", docs), `${root}/logo.png`); | |
| 83 | + | // From the repository's root, as people write them. | |
| 84 | + | assert.equal(imageSource("/assets/x.svg", docs), `${root}/assets/x.svg`); | |
| 85 | + | // Never out of the repository. | |
| 86 | + | assert.equal(imageSource("../../../../other/repo/raw/main/x.png", docs), undefined); | |
| 87 | + | // External pictures, and those with no repository, are as written. | |
| 88 | + | assert.equal(imageSource("https://example.com/x.png", docs), "https://example.com/x.png"); | |
| 89 | + | assert.equal(imageSource("data:image/png;base64,AA", docs), "data:image/png;base64,AA"); | |
| 90 | + | assert.equal(imageSource("logo.png", undefined), "logo.png"); | |
| 91 | + | }); | |
| 92 | + | ||
| 93 | + | test("commits are full hashes", () => { | |
| 94 | + | assert.equal(isCommit("a".repeat(40)), true); | |
| 95 | + | assert.equal(isCommit("main"), false); | |
| 96 | + | }); |
| 1 | + | /** | |
| 2 | + | * Files people supply, served from an origin of their own: a repository's | |
| 3 | + | * files and uploaded avatars at `USERCONTENT_URL` (g1tusercontent.com on | |
| 4 | + | * g1t.sh). The site's session cookie is never sent there, and nothing | |
| 5 | + | * served there can run script. | |
| 6 | + | * | |
| 7 | + | * <usercontent>/<owner>/<repo>/raw/<ref>/<path> a file at a branch, tag or commit | |
| 8 | + | * <usercontent>/avatars/<sha256> an uploaded avatar | |
| 9 | + | * | |
| 10 | + | * A public repository's files are there for anyone. A private one's carry | |
| 11 | + | * `?token=`, a signature the site makes for someone who can read the | |
| 12 | + | * repository (routes/repo/raw.ts), good for one file for an hour or two. | |
| 13 | + | * No Workers imports, so it can be tested under Node. | |
| 14 | + | */ | |
| 15 | + | ||
| 16 | + | /** What every file served there runs under: nothing runs, images and inline styles of its own only. */ | |
| 17 | + | export const USERCONTENT_POLICY = "default-src 'none'; img-src 'self'; style-src 'unsafe-inline'; sandbox"; | |
| 18 | + | /** A PDF: the same, but not sandboxed, which browsers' PDF viewers refuse to open under. */ | |
| 19 | + | export const PDF_POLICY = "default-src 'none'; img-src 'self'; style-src 'unsafe-inline'; object-src 'none'; base-uri 'none'; form-action 'none'"; | |
| 20 | + | ||
| 21 | + | /** The largest file served, in bytes. */ | |
| 22 | + | export const MAX_RAW_BYTES = 10 * 1024 * 1024; | |
| 23 | + | ||
| 24 | + | /** A signed address lasts until the end of the next whole hour, so a page's addresses stay the same for an hour. */ | |
| 25 | + | const TOKEN_HOURS = 2; | |
| 26 | + | ||
| 27 | + | export type RawFile = { owner: string; repo: string; ref: string; path: string }; | |
| 28 | + | ||
| 29 | + | const segment = (value: string) => encodeURIComponent(value); | |
| 30 | + | ||
| 31 | + | /** `/<owner>/<repo>/raw/<ref>/<path>`, each part encoded; a ref's slashes too, so it stays one segment. */ | |
| 32 | + | export function rawPath(file: RawFile): string { | |
| 33 | + | const path = file.path.split("/").filter(Boolean).map(segment).join("/"); | |
| 34 | + | return `/${segment(file.owner)}/${segment(file.repo)}/raw/${segment(file.ref)}/${path}`; | |
| 35 | + | } | |
| 36 | + | ||
| 37 | + | /** The parts of a raw file's path, decoded; null for any other path. */ | |
| 38 | + | export function parseRawPath(pathname: string): RawFile | null { | |
| 39 | + | const parts = pathname.split("/").slice(1); | |
| 40 | + | if (parts.length < 5 || parts[2] !== "raw") return null; | |
| 41 | + | try { | |
| 42 | + | const [owner, repo, , ref, ...rest] = parts.map(decodeURIComponent); | |
| 43 | + | const path = rest.join("/"); | |
| 44 | + | if (!owner || !repo || !ref || !path || rest.some((part) => !part || part === "." || part === "..")) return null; | |
| 45 | + | return { owner, repo, ref, path }; | |
| 46 | + | } catch { | |
| 47 | + | return null; | |
| 48 | + | } | |
| 49 | + | } | |
| 50 | + | ||
| 51 | + | /** | |
| 52 | + | * The part of `url` under the usercontent address `base`, or null when it | |
| 53 | + | * is not there: on its own host, any path; as a path on the site | |
| 54 | + | * (`<site>/-/usercontent`), what follows that path, whatever the host the | |
| 55 | + | * request came in on (a proxy may change it). | |
| 56 | + | */ | |
| 57 | + | export function usercontentPath(url: URL, base: string): string | null { | |
| 58 | + | const at = new URL(base); | |
| 59 | + | const prefix = at.pathname.replace(/\/+$/, ""); | |
| 60 | + | if (!prefix) return url.host === at.host ? url.pathname : null; | |
| 61 | + | if (url.pathname === prefix || url.pathname.startsWith(`${prefix}/`)) return url.pathname.slice(prefix.length) || "/"; | |
| 62 | + | return null; | |
| 63 | + | } | |
| 64 | + | ||
| 65 | + | /** Whether a ref names a commit, whose files never change. */ | |
| 66 | + | export function isCommit(ref: string): boolean { | |
| 67 | + | return /^[0-9a-f]{40}$/.test(ref); | |
| 68 | + | } | |
| 69 | + | ||
| 70 | + | const encoder = new TextEncoder(); | |
| 71 | + | ||
| 72 | + | function base64url(bytes: ArrayBuffer): string { | |
| 73 | + | return btoa(String.fromCharCode(...new Uint8Array(bytes))).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, ""); | |
| 74 | + | } | |
| 75 | + | ||
| 76 | + | function fromBase64url(text: string): Uint8Array<ArrayBuffer> | null { | |
| 77 | + | try { | |
| 78 | + | const plain = atob(text.replace(/-/g, "+").replace(/_/g, "/")); | |
| 79 | + | return Uint8Array.from(plain, (c) => c.charCodeAt(0)); | |
| 80 | + | } catch { | |
| 81 | + | return null; | |
| 82 | + | } | |
| 83 | + | } | |
| 84 | + | ||
| 85 | + | function hmacKey(secret: string, use: KeyUsage): Promise<CryptoKey> { | |
| 86 | + | return crypto.subtle.importKey("raw", encoder.encode(secret), { name: "HMAC", hash: "SHA-256" }, false, [use]); | |
| 87 | + | } | |
| 88 | + | ||
| 89 | + | /** What a token signs: the file, by the repository's path and id, and when it ends. */ | |
| 90 | + | function signed(file: RawFile, repoId: string, expires: number): Uint8Array<ArrayBuffer> { | |
| 91 | + | return encoder.encode(["raw", file.owner.toLowerCase(), file.repo.toLowerCase(), repoId, file.ref, file.path, String(expires)].join("\n")); | |
| 92 | + | } | |
| 93 | + | ||
| 94 | + | /** A token for one file of a private repository: `<expires>.<repoId>.<signature>`. */ | |
| 95 | + | export async function signRaw(secret: string, file: RawFile, repoId: string, nowMs = Date.now()): Promise<string> { | |
| 96 | + | const hour = 3600; | |
| 97 | + | const expires = (Math.floor(nowMs / 1000 / hour) + TOKEN_HOURS) * hour; | |
| 98 | + | const signature = await crypto.subtle.sign("HMAC", await hmacKey(secret, "sign"), signed(file, repoId, expires)); | |
| 99 | + | return `${expires}.${repoId}.${base64url(signature)}`; | |
| 100 | + | } | |
| 101 | + | ||
| 102 | + | /** The repository id a token is good for, when it is for this file and has not ended; else null. */ | |
| 103 | + | export async function verifyRaw(secret: string, file: RawFile, token: string, nowMs = Date.now()): Promise<string | null> { | |
| 104 | + | const match = /^(\d{1,12})\.([A-Za-z0-9_-]{1,64})\.([A-Za-z0-9_-]{43})$/.exec(token); | |
| 105 | + | if (!match) return null; | |
| 106 | + | const [, at, repoId, signature] = match; | |
| 107 | + | const expires = Number(at); | |
| 108 | + | if (expires * 1000 <= nowMs) return null; | |
| 109 | + | const bytes = fromBase64url(signature!); | |
| 110 | + | if (!bytes) return null; | |
| 111 | + | const ok = await crypto.subtle.verify("HMAC", await hmacKey(secret, "verify"), bytes, signed(file, repoId!, expires)); | |
| 112 | + | return ok ? repoId! : null; | |
| 113 | + | } | |
| 114 | + | ||
| 115 | + | const IMAGES: Record<string, string> = { | |
| 116 | + | png: "image/png", | |
| 117 | + | jpg: "image/jpeg", | |
| 118 | + | jpeg: "image/jpeg", | |
| 119 | + | gif: "image/gif", | |
| 120 | + | webp: "image/webp", | |
| 121 | + | avif: "image/avif", | |
| 122 | + | ico: "image/x-icon", | |
| 123 | + | bmp: "image/bmp", | |
| 124 | + | svg: "image/svg+xml", | |
| 125 | + | }; | |
| 126 | + | ||
| 127 | + | const MEDIA: Record<string, string> = { | |
| 128 | + | mp4: "video/mp4", | |
| 129 | + | webm: "video/webm", | |
| 130 | + | mov: "video/quicktime", | |
| 131 | + | mp3: "audio/mpeg", | |
| 132 | + | ogg: "audio/ogg", | |
| 133 | + | wav: "audio/wav", | |
| 134 | + | woff: "font/woff", | |
| 135 | + | woff2: "font/woff2", | |
| 136 | + | pdf: "application/pdf", | |
| 137 | + | }; | |
| 138 | + | ||
| 139 | + | function extension(path: string): string { | |
| 140 | + | const name = path.split("/").pop() ?? ""; | |
| 141 | + | return name.includes(".") ? name.split(".").pop()!.toLowerCase() : ""; | |
| 142 | + | } | |
| 143 | + | ||
| 144 | + | /** Whether a file shows as an image in a page, by its name. */ | |
| 145 | + | export function isImagePath(path: string): boolean { | |
| 146 | + | return extension(path) in IMAGES; | |
| 147 | + | } | |
| 148 | + | ||
| 149 | + | /** Whether the bytes look like text: no NUL in the first 8,000. */ | |
| 150 | + | function looksLikeText(bytes: Uint8Array): boolean { | |
| 151 | + | return !bytes.subarray(0, 8000).includes(0); | |
| 152 | + | } | |
| 153 | + | ||
| 154 | + | /** | |
| 155 | + | * The headers a file is served with. Images, media and PDFs as | |
| 156 | + | * themselves; any other text (HTML, SVG's script, XML, JavaScript | |
| 157 | + | * included) as plain text; anything else as bytes to save. Never sniffed, | |
| 158 | + | * and nothing in it runs. | |
| 159 | + | */ | |
| 160 | + | export function rawHeaders(path: string, bytes: Uint8Array): Headers { | |
| 161 | + | const ext = extension(path); | |
| 162 | + | const type = IMAGES[ext] ?? MEDIA[ext] ?? (looksLikeText(bytes) ? "text/plain; charset=utf-8" : "application/octet-stream"); | |
| 163 | + | const headers = new Headers({ | |
| 164 | + | "content-type": type, | |
| 165 | + | "content-length": String(bytes.byteLength), | |
| 166 | + | "x-content-type-options": "nosniff", | |
| 167 | + | "content-security-policy": type === "application/pdf" ? PDF_POLICY : USERCONTENT_POLICY, | |
| 168 | + | "cross-origin-resource-policy": "cross-origin", | |
| 169 | + | "referrer-policy": "no-referrer", | |
| 170 | + | }); | |
| 171 | + | if (type === "application/octet-stream") { | |
| 172 | + | const name = path.split("/").pop() ?? "file"; | |
| 173 | + | headers.set("content-disposition", `attachment; filename="${name.replace(/[^\x20-\x7e]|["\\%;]/g, "_")}"; filename*=UTF-8''${encodeURIComponent(name)}`); | |
| 174 | + | } | |
| 175 | + | return headers; | |
| 176 | + | } | |
| 177 | + | ||
| 178 | + | /** | |
| 179 | + | * An image's address: an external one as written; a relative one as the | |
| 180 | + | * repository's raw file at the same commit, or nothing when it climbs out | |
| 181 | + | * of the repository. `rawBase` is the document's folder under | |
| 182 | + | * `/<owner>/<repo>/raw/<ref>`. | |
| 183 | + | */ | |
| 184 | + | export function imageSource(src: string, rawBase: string | undefined): string | undefined { | |
| 185 | + | if (/^[a-z][a-z0-9+.-]*:/i.test(src) || src.startsWith("//") || !rawBase || src.startsWith("#")) return src; | |
| 186 | + | const root = /^\/[^/]+\/[^/]+\/raw\/[^/]+/.exec(rawBase)?.[0]; | |
| 187 | + | if (!root) return src; | |
| 188 | + | const path = src.split(/[?#]/)[0]!; | |
| 189 | + | if (!path) return undefined; | |
| 190 | + | const from = path.startsWith("/") ? `${root}/` : `${rawBase.replace(/\/+$/, "")}/`; | |
| 191 | + | const resolved = new URL(path.replace(/^\/+/, ""), `https://g1t.invalid${from}`).pathname; | |
| 192 | + | return resolved.startsWith(`${root}/`) ? resolved : undefined; | |
| 193 | + | } |
| 66 | 66 | import { addresses } from "./lib/addresses.server"; | |
| 67 | 67 | import { useSignUpCopy } from "./lib/registration"; | |
| 68 | 68 | import { RELOADED_KEY, reloadFixes } from "./lib/stale-build"; | |
| 69 | + | import { useNonce } from "./lib/nonce"; | |
| 69 | 70 | ||
| 70 | 71 | ||
| 71 | 72 | export const links: Route.LinksFunction = () => [ | |
| ⋯ | |||
| 463 | 464 | let lastRoot: Awaited<ReturnType<typeof loader>> | undefined; | |
| 464 | 465 | ||
| 465 | 466 | export function Layout({ children }: { children: React.ReactNode }) { | |
| 467 | + | const nonce = useNonce(); | |
| 466 | 468 | // Undefined when the root loader itself failed. | |
| 467 | 469 | const loaded = useRouteLoaderData<typeof loader>("root"); | |
| 468 | 470 | const inBrowser = typeof document !== "undefined"; | |
| ⋯ | |||
| 518 | 520 | connection paints sooner (docs/research/css-shipping.md). */} | |
| 519 | 521 | <link rel="stylesheet" href={appCss} precedence="default" /> | |
| 520 | 522 | <Meta /> | |
| 521 | − | <Links /> | |
| 523 | + | <Links nonce={nonce} /> | |
| 522 | 524 | </head> | |
| 523 | 525 | <body className="flex min-h-screen flex-col"> | |
| 524 | 526 | {/* The first stop for the keyboard: past the menus, to the page. */} | |
| ⋯ | |||
| 543 | 545 | <SiteFooter user={user} /> | |
| 544 | 546 | </> | |
| 545 | 547 | )} | |
| 546 | − | <ScrollRestoration /> | |
| 547 | − | <Scripts /> | |
| 548 | + | <ScrollRestoration nonce={nonce} /> | |
| 549 | + | <Scripts nonce={nonce} /> | |
| 548 | 550 | </body> | |
| 549 | 551 | </html> | |
| 550 | 552 | ); | |
| 142 | 142 | route(":owner/:repo/add-ci", "routes/repo/add-ci.ts"), | |
| 143 | 143 | // A screenshot of a project's production, for its overview. | |
| 144 | 144 | route(":owner/:repo/production.jpg", "routes/repo/production-screenshot.ts"), | |
| 145 | + | // A file as it is, sent on to the usercontent origin (lib/usercontent.ts). | |
| 146 | + | route(":owner/:repo/raw/:ref/*", "routes/repo/raw.ts"), | |
| 145 | 147 | // A project: its overview first, its repository's code under Code. The | |
| 146 | 148 | // 1:1 project of a repository has the repository's name, so every | |
| 147 | 149 | // repository address below keeps working. |
| 2 | 2 | ||
| 3 | 3 | import type { Route } from "./+types/actions-artifact"; | |
| 4 | 4 | import { addresses } from "../../lib/addresses.server"; | |
| 5 | + | import { contentDisposition } from "../../lib/content-safety"; | |
| 5 | 6 | import { readArtifact } from "../../lib/artifacts.server"; | |
| 6 | 7 | import { actions } from "../../lib/services.server"; | |
| 7 | 8 | import { getViewer } from "../../lib/session.server"; | |
| ⋯ | |||
| 25 | 26 | return new Response(bytes.buffer as ArrayBuffer, { | |
| 26 | 27 | headers: { | |
| 27 | 28 | "content-type": "application/gzip", | |
| 28 | − | "content-disposition": `attachment; filename="${params.name.replace(/"/g, "")}.tar.gz"`, | |
| 29 | + | "content-disposition": contentDisposition(`${params.name}.tar.gz`), | |
| 29 | 30 | }, | |
| 30 | 31 | }); | |
| 31 | 32 | } | |
| 7 | 7 | import type { Route } from "./+types/archive"; | |
| 8 | 8 | import { cloneUrl } from "../../lib/addresses"; | |
| 9 | 9 | import { addresses } from "../../lib/addresses.server"; | |
| 10 | + | import { contentDisposition } from "../../lib/content-safety"; | |
| 10 | 11 | import { repos } from "../../lib/services.server"; | |
| 11 | 12 | import { getViewer } from "../../lib/session.server"; | |
| 12 | 13 | import { zip } from "../../lib/zip"; | |
| ⋯ | |||
| 61 | 62 | return new Response(archive, { | |
| 62 | 63 | headers: { | |
| 63 | 64 | "content-type": "application/zip", | |
| 64 | − | "content-disposition": `attachment; filename="${label}.zip"`, | |
| 65 | + | // A ref may hold quotes; the header never does. | |
| 66 | + | "content-disposition": contentDisposition(`${label}.zip`), | |
| 65 | 67 | // A commit's files never change; a branch's do. | |
| 66 | 68 | "cache-control": /^[0-9a-f]{40}$/.test(ref) ? "private, max-age=31536000, immutable" : "private, no-cache", | |
| 67 | 69 | }, | |
| 45 | 45 | "content-type": shot.contentType, | |
| 46 | 46 | "cache-control": current ? LONG : BRIEF, | |
| 47 | 47 | "x-content-type-options": "nosniff", | |
| 48 | + | // A picture of someone's app: shown, never run. | |
| 49 | + | "content-security-policy": "default-src 'none'; sandbox", | |
| 48 | 50 | "last-modified": new Date(shot.capturedAt).toUTCString(), | |
| 49 | 51 | }, | |
| 50 | 52 | }); |
| 1 | + | /** | |
| 2 | + | * A file of a repository as it is, for the Raw button, images on a file's | |
| 3 | + | * page and pictures in a README: `/<owner>/<repo>/raw/<ref>/<path>`. Sends | |
| 4 | + | * the viewer on to the file at the commit the ref names, on the usercontent | |
| 5 | + | * origin (lib/usercontent.ts). A public repository's address is the same | |
| 6 | + | * for everyone; a private one's carries a token for this file alone, made | |
| 7 | + | * here for someone who can read the repository, good for an hour or two. | |
| 8 | + | * Without USERCONTENT_KEY a private file is served from here instead, | |
| 9 | + | * under the same policy. | |
| 10 | + | */ | |
| 11 | + | import { env } from "cloudflare:workers"; | |
| 12 | + | ||
| 13 | + | import type { Route } from "./+types/raw"; | |
| 14 | + | import { addresses } from "../../lib/addresses.server"; | |
| 15 | + | import { repos } from "../../lib/services.server"; | |
| 16 | + | import { getViewer } from "../../lib/session.server"; | |
| 17 | + | import { MAX_RAW_BYTES, isCommit, rawHeaders, rawPath, signRaw } from "../../lib/usercontent"; | |
| 18 | + | ||
| 19 | + | function refused(status: number, message: string): Response { | |
| 20 | + | return new Response(`${message}\n`, { | |
| 21 | + | status, | |
| 22 | + | headers: { "content-type": "text/plain; charset=utf-8", "cache-control": "no-store", "x-content-type-options": "nosniff" }, | |
| 23 | + | }); | |
| 24 | + | } | |
| 25 | + | ||
| 26 | + | export async function loader({ params, context }: Route.LoaderArgs) { | |
| 27 | + | const viewer = getViewer(context); | |
| 28 | + | const path = params["*"] ?? ""; | |
| 29 | + | if (!path) return refused(404, "Ask for /<owner>/<repo>/raw/<branch, tag or commit>/<path>."); | |
| 30 | + | const named = { namespace: params.owner, name: params.repo }; | |
| 31 | + | const found = await repos.get(named, viewer).catch(() => null); | |
| 32 | + | if (!found?.ok) return refused(404, "There is no such repository, or you cannot see it."); | |
| 33 | + | const repo = found.value; | |
| 34 | + | // The commit the ref names now, so the file's address never changes. | |
| 35 | + | let commit = isCommit(params.ref) ? params.ref : null; | |
| 36 | + | if (!commit) { | |
| 37 | + | const log = await repos.log(named, viewer, params.ref, 1).catch(() => null); | |
| 38 | + | commit = log?.ok ? (log.value[0]?.hash ?? null) : null; | |
| 39 | + | } | |
| 40 | + | if (!commit) return refused(404, `There is no branch, tag or commit named ${params.ref}.`); | |
| 41 | + | const file = { owner: repo.namespace, repo: repo.name, ref: commit, path }; | |
| 42 | + | const target = `${addresses().usercontent}${rawPath(file)}`; | |
| 43 | + | // Kept briefly when it followed a branch, which moves. | |
| 44 | + | const cache = isCommit(params.ref) ? "private, max-age=86400" : "private, max-age=60"; | |
| 45 | + | if (!repo.isPrivate) return redirect(target, cache); | |
| 46 | + | if (env.USERCONTENT_KEY) { | |
| 47 | + | const token = await signRaw(env.USERCONTENT_KEY, file, repo.id); | |
| 48 | + | return redirect(`${target}?token=${encodeURIComponent(token)}`, "private, max-age=600"); | |
| 49 | + | } | |
| 50 | + | const raw = await repos.rawFile(repo.id, commit, path, MAX_RAW_BYTES).catch(() => null); | |
| 51 | + | if (!raw) return refused(404, `There is no such file, or it is over ${MAX_RAW_BYTES / 1024 / 1024} MB. Clone the repository for it.`); | |
| 52 | + | const bytes = Uint8Array.from(atob(raw.data), (c) => c.charCodeAt(0)); | |
| 53 | + | const headers = rawHeaders(path, bytes); | |
| 54 | + | headers.set("cache-control", "private, max-age=60"); | |
| 55 | + | return new Response(bytes, { headers }); | |
| 56 | + | } | |
| 57 | + | ||
| 58 | + | function redirect(location: string, cache: string): Response { | |
| 59 | + | return new Response(null, { status: 302, headers: { location, "cache-control": cache } }); | |
| 60 | + | } |
| 4 | 4 | # kept for good. That includes the typefaces. | |
| 5 | 5 | /assets/* | |
| 6 | 6 | Cache-Control: public, max-age=31536000, immutable | |
| 7 | + | ||
| 8 | + | # Every static file: only the type it says, and a referrer of the origin | |
| 9 | + | # alone, as the Worker's own answers (app/lib/page-headers.ts). | |
| 10 | + | /* | |
| 11 | + | X-Content-Type-Options: nosniff | |
| 12 | + | Referrer-Policy: strict-origin-when-cross-origin |
| 2 | 2 | ||
| 3 | 3 | import { identityClient, isNamespaceShaped } from "@g1t/contracts"; | |
| 4 | 4 | ||
| 5 | + | import { addressesFor } from "../app/lib/addresses"; | |
| 6 | + | import { hardenRegistryHeaders } from "../app/lib/content-safety"; | |
| 7 | + | import { withSiteHeaders } from "../app/lib/page-headers"; | |
| 5 | 8 | import { finishResponse, withRequestPerf } from "../app/lib/perf.server"; | |
| 6 | − | import { gitLimited, pageLimited } from "../app/lib/front-door-limits"; | |
| 9 | + | import { gitLimited, pageLimited, usercontentLimited } from "../app/lib/front-door-limits"; | |
| 7 | 10 | import { goImport } from "../app/lib/go-get"; | |
| 8 | 11 | import { repositoryOfPage, stillPublic } from "../app/lib/public-cache"; | |
| 9 | 12 | import { registryWorkspace, servicePath } from "../app/lib/registry-paths"; | |
| 13 | + | import { usercontentPath } from "../app/lib/usercontent"; | |
| 14 | + | import { serveUsercontent } from "./usercontent"; | |
| 10 | 15 | ||
| 11 | 16 | const requestHandler = createRequestHandler( | |
| 12 | 17 | () => import("virtual:react-router/server-build"), | |
| 13 | 18 | import.meta.env.MODE, | |
| 14 | 19 | ); | |
| 15 | 20 | ||
| 16 | − | /** An uploaded avatar, by the SHA-256 of its bytes. */ | |
| 17 | − | const AVATAR_PATH = /^\/avatars\/([0-9a-f]{64})$/; | |
| 18 | − | /** The only types identity stores, having checked each image's bytes. */ | |
| 19 | − | const AVATAR_TYPES = new Set(["image/png", "image/jpeg", "image/webp", "image/gif"]); | |
| 20 | 21 | const DOCS = "https://docs.g1t.sh"; | |
| 21 | 22 | ||
| 22 | 23 | /** Where the documentation pages that used to live under /docs are now. */ | |
| ⋯ | |||
| 33 | 34 | ||
| 34 | 35 | export default { | |
| 35 | 36 | async fetch(request, env, ctx) { | |
| 36 | − | const { pathname } = new URL(request.url); | |
| 37 | − | // Git over HTTPS shares this hostname but belongs to the repos service. | |
| 38 | − | // Its answer goes back to the git client as it is: a repository under a | |
| 39 | − | // renamed workspace's old name answers with a 301, which git follows and | |
| 40 | − | // must see, so the redirect is never followed here. | |
| 41 | − | // The container registry (`docker login g1t.sh`) and the npm registry | |
| 42 | − | // (`g1t.sh/-/npm/`) are the packages | |
| 43 | − | // service's, handed over the same way. | |
| 44 | − | // `go get g1t.sh/<workspace>/<repo>`: where its code is, from the | |
| 45 | − | // address alone, so it costs nothing and caches. | |
| 46 | − | const go = request.method === "GET" ? goImport(new URL(request.url)) : null; | |
| 47 | − | if (go) { | |
| 48 | − | return new Response(go, { | |
| 49 | − | headers: { "content-type": "text/html; charset=utf-8", "cache-control": "public, max-age=3600" }, | |
| 50 | − | }); | |
| 37 | + | // Repository files and avatars, on their own origin | |
| 38 | + | // (g1tusercontent.com): answered before anything of the site's runs, | |
| 39 | + | // so nothing there reads or sets the session cookie. | |
| 40 | + | const usercontent = usercontentPath(new URL(request.url), addressesFor(env).usercontent); | |
| 41 | + | if (usercontent !== null) { | |
| 42 | + | return (await usercontentLimited(env, request, usercontent)) ?? serveUsercontent(env, ctx, request, usercontent); | |
| 51 | 43 | } | |
| 52 | − | const service = servicePath(pathname); | |
| 53 | − | if (service === "git") { | |
| 54 | − | // Per address without credentials, per credential with them | |
| 55 | − | // (app/lib/front-door-limits.ts): anonymous clones are not free to | |
| 56 | − | // the repository's owner. | |
| 57 | − | return (await gitLimited(env, request)) ?? proxyGit(env, request); | |
| 58 | − | } | |
| 59 | − | if (service === "packages") { | |
| 60 | − | return proxyPackages(env, request); | |
| 61 | − | } | |
| 62 | − | const avatar = AVATAR_PATH.exec(pathname); | |
| 63 | − | if (avatar) { | |
| 64 | − | return serveAvatar(env, ctx, request, avatar[1]); | |
| 65 | − | } | |
| 66 | − | // The documentation is its own site. | |
| 67 | − | if (pathname === "/docs" || pathname.startsWith("/docs/")) { | |
| 68 | − | const page = pathname.endsWith("/") ? pathname.slice(0, -1) : pathname; | |
| 69 | − | const target = MOVED_DOCS[page] ?? "/"; | |
| 70 | − | return Response.redirect(DOCS + target, 301); | |
| 71 | − | } | |
| 72 | − | // Pages and data requests, limited per address signed out and per | |
| 73 | − | // session signed in (app/lib/front-door-limits.ts). | |
| 74 | − | const limited = await pageLimited(env, request, pathname); | |
| 75 | − | if (limited) return limited; | |
| 76 | − | // Every page and data request says where its time went (Server-Timing) | |
| 77 | − | // and keeps the reader's D1 bookmarks (app/lib/perf.server.ts). | |
| 78 | − | const render = () => withRequestPerf(request, async () => finishResponse(request, await requestHandler(request))); | |
| 79 | − | if (anonymousPage(request, pathname)) return servePublic(env, request, ctx, render); | |
| 80 | − | return render(); | |
| 44 | + | // Every answer: no sniffing, a referrer of the origin alone, and no | |
| 45 | + | // framing of pages (app/lib/page-headers.ts). | |
| 46 | + | return withSiteHeaders(await site(request, env, ctx)); | |
| 81 | 47 | }, | |
| 82 | 48 | } satisfies ExportedHandler<Env>; | |
| 83 | 49 | ||
| 50 | + | async function site(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> { | |
| 51 | + | const { pathname } = new URL(request.url); | |
| 52 | + | // Git over HTTPS shares this hostname but belongs to the repos service. | |
| 53 | + | // Its answer goes back to the git client as it is: a repository under a | |
| 54 | + | // renamed workspace's old name answers with a 301, which git follows and | |
| 55 | + | // must see, so the redirect is never followed here. | |
| 56 | + | // The container registry (`docker login g1t.sh`) and the npm registry | |
| 57 | + | // (`g1t.sh/-/npm/`) are the packages | |
| 58 | + | // service's, handed over the same way. | |
| 59 | + | // `go get g1t.sh/<workspace>/<repo>`: where its code is, from the | |
| 60 | + | // address alone, so it costs nothing and caches. | |
| 61 | + | const go = request.method === "GET" ? goImport(new URL(request.url)) : null; | |
| 62 | + | if (go) { | |
| 63 | + | return new Response(go, { | |
| 64 | + | headers: { "content-type": "text/html; charset=utf-8", "cache-control": "public, max-age=3600" }, | |
| 65 | + | }); | |
| 66 | + | } | |
| 67 | + | const service = servicePath(pathname); | |
| 68 | + | if (service === "git") { | |
| 69 | + | // Per address without credentials, per credential with them | |
| 70 | + | // (app/lib/front-door-limits.ts): anonymous clones are not free to | |
| 71 | + | // the repository's owner. | |
| 72 | + | return (await gitLimited(env, request)) ?? proxyGit(env, request); | |
| 73 | + | } | |
| 74 | + | if (service === "packages") { | |
| 75 | + | return proxyPackages(env, request); | |
| 76 | + | } | |
| 77 | + | // The documentation is its own site. | |
| 78 | + | if (pathname === "/docs" || pathname.startsWith("/docs/")) { | |
| 79 | + | const page = pathname.endsWith("/") ? pathname.slice(0, -1) : pathname; | |
| 80 | + | const target = MOVED_DOCS[page] ?? "/"; | |
| 81 | + | return Response.redirect(DOCS + target, 301); | |
| 82 | + | } | |
| 83 | + | // Pages and data requests, limited per address signed out and per | |
| 84 | + | // session signed in (app/lib/front-door-limits.ts). | |
| 85 | + | const limited = await pageLimited(env, request, pathname); | |
| 86 | + | if (limited) return limited; | |
| 87 | + | // Every page and data request says where its time went (Server-Timing) | |
| 88 | + | // and keeps the reader's D1 bookmarks (app/lib/perf.server.ts). | |
| 89 | + | const render = () => withRequestPerf(request, async () => finishResponse(request, await requestHandler(request))); | |
| 90 | + | if (anonymousPage(request, pathname)) return servePublic(env, request, ctx, render); | |
| 91 | + | return render(); | |
| 92 | + | } | |
| 93 | + | ||
| 84 | 94 | /** | |
| 85 | 95 | * Public pages as someone signed out sees them: the same for every such | |
| 86 | 96 | * visitor, so kept in this data centre's cache. Reserved first segments | |
| ⋯ | |||
| 175 | 185 | * follows them itself. One that found nothing under a workspace's old name | |
| 176 | 186 | * or an alias staff set (`g1t` for `flagon-io`) is sent to the same path | |
| 177 | 187 | * under the workspace's name: only the not-found answer pays for the lookup. | |
| 188 | + | * Every answer runs nothing in a browser (app/lib/content-safety.ts): what | |
| 189 | + | * a registry serves is its publisher's, on this origin. | |
| 178 | 190 | */ | |
| 179 | 191 | async function proxyPackages(env: Env, request: Request): Promise<Response> { | |
| 180 | 192 | const started = Date.now(); | |
| 181 | 193 | const answer = await env.PACKAGES.fetch(new Request(request, { redirect: "manual" })); | |
| 182 | 194 | const moved = answer.status === 404 ? await registryMoved(env, request) : null; | |
| 183 | 195 | const response = moved ?? new Response(answer.body, answer); | |
| 196 | + | hardenRegistryHeaders(response.headers); | |
| 184 | 197 | response.headers.append("server-timing", `packages;dur=${Date.now() - started}`); | |
| 185 | 198 | return response; | |
| 186 | 199 | } | |
| ⋯ | |||
| 200 | 213 | const get = request.method === "GET" || request.method === "HEAD"; | |
| 201 | 214 | // 308 keeps a publish a PUT, for the clients that follow it. | |
| 202 | 215 | return new Response(null, { status: get ? 301 : 308, headers: { location: named.under(current) + url.search } }); | |
| 203 | − | } | |
| 204 | − | ||
| 205 | − | /** | |
| 206 | − | * An uploaded avatar. Its address is its hash, so it never changes and is | |
| 207 | − | * kept for good. It is served as nothing but an image: the stored type, | |
| 208 | − | * no sniffing, and a policy that lets nothing in it run. | |
| 209 | − | */ | |
| 210 | − | /** | |
| 211 | − | * An uploaded icon. Its address is its content's hash, so it never changes: | |
| 212 | − | * each data centre keeps it in its cache after the first view, and storage | |
| 213 | − | * is read about once per place, not once per visitor. | |
| 214 | − | */ | |
| 215 | − | async function serveAvatar(env: Env, ctx: ExecutionContext, request: Request, hash: string): Promise<Response> { | |
| 216 | − | const method = request.method; | |
| 217 | − | if (method !== "GET" && method !== "HEAD") { | |
| 218 | − | return new Response("Method not allowed", { status: 405, headers: { allow: "GET, HEAD" } }); | |
| 219 | − | } | |
| 220 | − | // The Workers runtime's own cache, which the DOM types do not know. | |
| 221 | − | const cache = (caches as unknown as { default: Cache }).default; | |
| 222 | − | const key = new Request(new URL(`/avatars/${hash}`, request.url).toString(), { method: "GET" }); | |
| 223 | − | const cached = await cache.match(key); | |
| 224 | − | if (cached) { | |
| 225 | − | return method === "HEAD" ? new Response(null, { headers: cached.headers }) : cached; | |
| 226 | − | } | |
| 227 | − | const { value, metadata } = await env.AVATARS.getWithMetadata<{ contentType?: string }>(hash, { | |
| 228 | − | type: "arrayBuffer", | |
| 229 | − | cacheTtl: 86400, | |
| 230 | − | }); | |
| 231 | − | const contentType = metadata?.contentType; | |
| 232 | − | if (!value || !contentType || !AVATAR_TYPES.has(contentType)) { | |
| 233 | − | return new Response("Not found", { | |
| 234 | − | status: 404, | |
| 235 | − | headers: { "cache-control": "public, max-age=60" }, | |
| 236 | − | }); | |
| 237 | − | } | |
| 238 | − | const headers = { | |
| 239 | − | "content-type": contentType, | |
| 240 | − | "content-length": String(value.byteLength), | |
| 241 | − | "cache-control": "public, max-age=31536000, immutable", | |
| 242 | − | "x-content-type-options": "nosniff", | |
| 243 | − | "content-security-policy": "default-src 'none'; sandbox", | |
| 244 | − | "cross-origin-resource-policy": "cross-origin", | |
| 245 | − | }; | |
| 246 | − | ctx.waitUntil(cache.put(key, new Response(value, { headers }))); | |
| 247 | − | return new Response(method === "HEAD" ? null : value, { headers }); | |
| 248 | 216 | } | |
| 51 | 51 | WEB_ADDRESS_LIMIT?: RateLimitBinding; | |
| 52 | 52 | GIT_ANONYMOUS_LIMIT?: RateLimitBinding; | |
| 53 | 53 | GIT_SIGNED_LIMIT?: RateLimitBinding; | |
| 54 | + | /** | |
| 55 | + | * Where repository files and avatars are served (app/lib/usercontent.ts). | |
| 56 | + | * Unset on g1t.sh, which is https://g1tusercontent.com; unset on another | |
| 57 | + | * site, `<SITE_URL>/-/usercontent`. | |
| 58 | + | */ | |
| 59 | + | USERCONTENT_URL?: string; | |
| 60 | + | /** Signs the short-lived addresses of private repositories' files. A secret. */ | |
| 61 | + | USERCONTENT_KEY?: string; | |
| 54 | 62 | } | |
| 55 | 63 | } | |
| 56 | 64 | interface Env extends Cloudflare.Env {} |
| 1 | + | /** | |
| 2 | + | * The usercontent origin (app/lib/usercontent.ts): repository files and | |
| 3 | + | * uploaded avatars, on g1tusercontent.com for g1t.sh. This answers before | |
| 4 | + | * anything of the site's runs, and reads no cookie and sets none: nothing | |
| 5 | + | * here knows who is asking, only what the address and its token say. | |
| 6 | + | */ | |
| 7 | + | import { reposClient } from "@g1t/contracts"; | |
| 8 | + | ||
| 9 | + | import { MAX_RAW_BYTES, isCommit, parseRawPath, rawHeaders, verifyRaw } from "../app/lib/usercontent"; | |
| 10 | + | ||
| 11 | + | /** An uploaded avatar, by the SHA-256 of its bytes. */ | |
| 12 | + | export const AVATAR_PATH = /^\/avatars\/([0-9a-f]{64})$/; | |
| 13 | + | /** The only types identity stores, having checked each image's bytes. */ | |
| 14 | + | const AVATAR_TYPES = new Set(["image/png", "image/jpeg", "image/webp", "image/gif"]); | |
| 15 | + | ||
| 16 | + | function plain(status: number, message: string, cache = "no-store"): Response { | |
| 17 | + | return new Response(`${message}\n`, { | |
| 18 | + | status, | |
| 19 | + | headers: { | |
| 20 | + | "content-type": "text/plain; charset=utf-8", | |
| 21 | + | "cache-control": cache, | |
| 22 | + | "x-content-type-options": "nosniff", | |
| 23 | + | "content-security-policy": "default-src 'none'; sandbox", | |
| 24 | + | }, | |
| 25 | + | }); | |
| 26 | + | } | |
| 27 | + | ||
| 28 | + | /** Answers a request for `path`, the part of its address under the usercontent origin. */ | |
| 29 | + | export async function serveUsercontent(env: Env, ctx: ExecutionContext, request: Request, path: string): Promise<Response> { | |
| 30 | + | const method = request.method; | |
| 31 | + | if (method !== "GET" && method !== "HEAD") { | |
| 32 | + | return new Response("Method not allowed\n", { status: 405, headers: { allow: "GET, HEAD" } }); | |
| 33 | + | } | |
| 34 | + | if (path === "/robots.txt") { | |
| 35 | + | return new Response("User-agent: *\nDisallow: /\n", { headers: { "content-type": "text/plain; charset=utf-8", "cache-control": "public, max-age=86400" } }); | |
| 36 | + | } | |
| 37 | + | const avatar = AVATAR_PATH.exec(path); | |
| 38 | + | if (avatar) return serveAvatar(env, ctx, method, avatar[1]!, new URL(request.url).origin); | |
| 39 | + | const file = parseRawPath(path); | |
| 40 | + | if (file) return serveRaw(env, request, method, file); | |
| 41 | + | return plain(404, "Not found", "public, max-age=300"); | |
| 42 | + | } | |
| 43 | + | ||
| 44 | + | /** | |
| 45 | + | * A repository's file. A public repository's to anyone; a private one's | |
| 46 | + | * only with a token for this very file (routes/repo/raw.ts makes them). | |
| 47 | + | */ | |
| 48 | + | async function serveRaw(env: Env, request: Request, method: string, file: NonNullable<ReturnType<typeof parseRawPath>>): Promise<Response> { | |
| 49 | + | const repos = reposClient(env.REPOS); | |
| 50 | + | const token = new URL(request.url).searchParams.get("token"); | |
| 51 | + | let repoId: string | null = null; | |
| 52 | + | let isPublic = false; | |
| 53 | + | if (token) { | |
| 54 | + | if (!env.USERCONTENT_KEY) return plain(404, "Not found"); | |
| 55 | + | repoId = await verifyRaw(env.USERCONTENT_KEY, file, token); | |
| 56 | + | if (!repoId) return plain(403, "This address has expired. Open the file on g1t again for a new one."); | |
| 57 | + | } else { | |
| 58 | + | // No viewer: only a public repository answers. | |
| 59 | + | const found = await repos.get({ namespace: file.owner, name: file.repo }, null).catch(() => null); | |
| 60 | + | if (!found?.ok) return plain(404, "There is no such file, or it is not public.", "public, max-age=60"); | |
| 61 | + | repoId = found.value.id; | |
| 62 | + | isPublic = true; | |
| 63 | + | } | |
| 64 | + | const raw = await repos.rawFile(repoId, file.ref, file.path, MAX_RAW_BYTES).catch(() => null); | |
| 65 | + | if (!raw) return plain(404, `There is no such file, or it is over ${MAX_RAW_BYTES / 1024 / 1024} MB. Clone the repository for it.`, "public, max-age=60"); | |
| 66 | + | const bytes = Uint8Array.from(atob(raw.data), (c) => c.charCodeAt(0)); | |
| 67 | + | const headers = rawHeaders(file.path, bytes); | |
| 68 | + | // A commit's files never change; a branch's or tag's may. | |
| 69 | + | const lasting = isCommit(file.ref); | |
| 70 | + | headers.set( | |
| 71 | + | "cache-control", | |
| 72 | + | isPublic ? (lasting ? "public, max-age=31536000, immutable" : "public, max-age=60") : lasting ? "private, max-age=3600" : "private, max-age=60", | |
| 73 | + | ); | |
| 74 | + | if (lasting) headers.set("etag", `"${file.ref}"`); | |
| 75 | + | return new Response(method === "HEAD" ? null : bytes, { headers }); | |
| 76 | + | } | |
| 77 | + | ||
| 78 | + | /** | |
| 79 | + | * An uploaded avatar. Its address is its hash, so it never changes and is | |
| 80 | + | * kept for good: each data centre keeps it in its cache after the first | |
| 81 | + | * view, and storage is read about once per place, not once per visitor. | |
| 82 | + | * It is served as nothing but an image: the stored type, no sniffing, and | |
| 83 | + | * a policy that lets nothing in it run. | |
| 84 | + | */ | |
| 85 | + | async function serveAvatar(env: Env, ctx: ExecutionContext, method: string, hash: string, origin: string): Promise<Response> { | |
| 86 | + | // The Workers runtime's own cache, which the DOM types do not know. | |
| 87 | + | const cache = (caches as unknown as { default: Cache }).default; | |
| 88 | + | const key = new Request(`${origin}/avatars/${hash}`, { method: "GET" }); | |
| 89 | + | const cached = await cache.match(key); | |
| 90 | + | if (cached) { | |
| 91 | + | return method === "HEAD" ? new Response(null, { headers: cached.headers }) : cached; | |
| 92 | + | } | |
| 93 | + | const { value, metadata } = await env.AVATARS.getWithMetadata<{ contentType?: string }>(hash, { | |
| 94 | + | type: "arrayBuffer", | |
| 95 | + | cacheTtl: 86400, | |
| 96 | + | }); | |
| 97 | + | const contentType = metadata?.contentType; | |
| 98 | + | if (!value || !contentType || !AVATAR_TYPES.has(contentType)) { | |
| 99 | + | return plain(404, "Not found", "public, max-age=60"); | |
| 100 | + | } | |
| 101 | + | const headers = { | |
| 102 | + | "content-type": contentType, | |
| 103 | + | "content-length": String(value.byteLength), | |
| 104 | + | "cache-control": "public, max-age=31536000, immutable", | |
| 105 | + | "x-content-type-options": "nosniff", | |
| 106 | + | "content-security-policy": "default-src 'none'; sandbox", | |
| 107 | + | "cross-origin-resource-policy": "cross-origin", | |
| 108 | + | }; | |
| 109 | + | ctx.waitUntil(cache.put(key, new Response(value, { headers }))); | |
| 110 | + | return new Response(method === "HEAD" ? null : value, { headers }); | |
| 111 | + | } |
| 11 | 11 | // alternatives and `apply` changes every Worker's at once. | |
| 12 | 12 | "placement": { "mode": "off" }, | |
| 13 | 13 | "main": "./workers/app.ts", | |
| 14 | − | "routes": [{ "pattern": "g1t.sh", "custom_domain": true }], | |
| 14 | + | // g1tusercontent.com: repository files and avatars, on an origin of | |
| 15 | + | // their own that never sees g1t.sh's cookies (workers/usercontent.ts). | |
| 16 | + | "routes": [ | |
| 17 | + | { "pattern": "g1t.sh", "custom_domain": true }, | |
| 18 | + | { "pattern": "g1tusercontent.com", "custom_domain": true } | |
| 19 | + | ], | |
| 15 | 20 | // The site holds no data of its own; everything goes through services. | |
| 16 | 21 | // GitHub Actions artifacts, as the API keeps them, for download from a run. | |
| 17 | 22 | // Uploaded avatars (g1t-avatars), served at /avatars/<sha256>. Read |
| 19 | 19 | // | |
| 20 | 20 | // Usage: node configs.mjs [outDir] | |
| 21 | 21 | // Environment: PUBLIC_URL, GITSTORE_URL, GITSTORE_SECRET, MAIL_URL, | |
| 22 | − | // ACTIONS_KEY, INTEGRATIONS_KEY, WEBHOOKS_KEY, IDENTITY_KEY, | |
| 22 | + | // ACTIONS_KEY, INTEGRATIONS_KEY, WEBHOOKS_KEY, IDENTITY_KEY, USERCONTENT_KEY, | |
| 23 | + | // USERCONTENT_URL, | |
| 23 | 24 | // PACKAGES_TOKEN_SECRET, S3_ENDPOINT, S3_BUCKET, BACKUP_S3_BUCKET, S3_REGION, | |
| 24 | 25 | // S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_PUBLIC_ENDPOINT, and optionally | |
| 25 | 26 | // your own GitHub App: GITHUB_APP_ID, GITHUB_APP_SLUG, GITHUB_APP_CLIENT_ID, | |
| ⋯ | |||
| 76 | 77 | ||
| 77 | 78 | /** Sealing keys, by the service that holds each (hosted: Wrangler secrets). */ | |
| 78 | 79 | const SECRETS = { | |
| 80 | + | g1t: "USERCONTENT_KEY", | |
| 79 | 81 | "g1t-actions": "ACTIONS_KEY", | |
| 80 | 82 | "g1t-integrations": "INTEGRATIONS_KEY", | |
| 81 | 83 | "g1t-webhooks": "WEBHOOKS_KEY", | |
| ⋯ | |||
| 240 | 242 | // shows: the site's clone URLs, meta tags and agent setup, the API's | |
| 241 | 243 | // OAuth issuer and MCP server, and identity's mail. No social cards: the | |
| 242 | 244 | // card service (services/og) is not run here. | |
| 243 | − | if (service.web) Object.assign(config.vars, { SITE_URL: PUBLIC_URL, API_URL, MCP_URL, OG_URL: "" }); | |
| 245 | + | // Repository files and avatars: USERCONTENT_URL, a host of its own that | |
| 246 | + | // reaches this same site, or, empty, a path on it (PUBLIC_URL/-/usercontent). | |
| 247 | + | if (service.web) { | |
| 248 | + | Object.assign(config.vars, { SITE_URL: PUBLIC_URL, API_URL, MCP_URL, OG_URL: "", USERCONTENT_URL: (process.env.USERCONTENT_URL ?? "").trim() }); | |
| 249 | + | } | |
| 244 | 250 | if (hosted.name === "g1t-api") Object.assign(config.vars, { SITE_URL: PUBLIC_URL, API_URL, MCP_URL }); | |
| 245 | 251 | if (hosted.name === "g1t-identity") config.vars.SITE_URL = PUBLIC_URL; | |
| 246 | 252 | // Nightly backups' bundles go to a bucket of their own on the same | |
| 30 | 30 | # means PUBLIC_URL's host on API_PORT. MCP_URL, empty, is API_URL/mcp. | |
| 31 | 31 | API_URL: ${API_URL:-} | |
| 32 | 32 | MCP_URL: ${MCP_URL:-} | |
| 33 | + | # Where repository files and avatars are served: a host of its own | |
| 34 | + | # that reaches this container, so they never share the site's | |
| 35 | + | # cookies; empty serves them under PUBLIC_URL/-/usercontent. | |
| 36 | + | USERCONTENT_URL: ${USERCONTENT_URL:-} | |
| 33 | 37 | API_PORT: ${API_PORT:-8789} | |
| 34 | 38 | GITSTORE_URL: http://gitstore:8080 | |
| 35 | 39 | GITSTORE_SECRET_FILE: /secrets/gitstore |
| 28 | 28 | if ! grep -q '^IDENTITY_KEY=' "$KEYS"; then | |
| 29 | 29 | echo "IDENTITY_KEY=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')" >> "$KEYS" | |
| 30 | 30 | fi | |
| 31 | + | # The site's key, which signs the short-lived addresses of private | |
| 32 | + | # repositories' files. | |
| 33 | + | if ! grep -q '^USERCONTENT_KEY=' "$KEYS"; then | |
| 34 | + | echo "USERCONTENT_KEY=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')" >> "$KEYS" | |
| 35 | + | fi | |
| 31 | 36 | # The packages service's key, which signs registry tokens. | |
| 32 | 37 | if ! grep -q '^PACKAGES_TOKEN_SECRET=' "$KEYS"; then | |
| 33 | 38 | echo "PACKAGES_TOKEN_SECRET=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')" >> "$KEYS" |
| 277 | 277 | "kind": "react-router", | |
| 278 | 278 | "worker": "g1t", | |
| 279 | 279 | "stage": "front", | |
| 280 | − | "secrets": [], | |
| 281 | − | "setup": ["The R2 bucket g1t-downloads, for the self-hosted runner's releases: npx wrangler r2 bucket create g1t-downloads"], | |
| 280 | + | // USERCONTENT_KEY signs the short-lived addresses of private | |
| 281 | + | // repositories' files on g1tusercontent.com; without it they are | |
| 282 | + | // served from g1t.sh instead. | |
| 283 | + | "secrets": ["USERCONTENT_KEY"], | |
| 284 | + | "setup": [ | |
| 285 | + | "The R2 bucket g1t-downloads, for the self-hosted runner's releases: npx wrangler r2 bucket create g1t-downloads", | |
| 286 | + | "The zone g1tusercontent.com on the account; the Worker's custom domain on it is made by the deploy", | |
| 287 | + | "The key for private files' addresses: `node -e \"console.log(require('crypto').randomBytes(32).toString('hex'))\" | npx wrangler secret put USERCONTENT_KEY` in apps/web" | |
| 288 | + | ], | |
| 282 | 289 | "self_host": "run" | |
| 283 | 290 | }, | |
| 284 | 291 | "sudo": { |
| 137 | 137 | `MCP_URL` and `OG_URL` (empty: no social card tags). The root loader | |
| 138 | 138 | hands them to the page; `meta.ts`, the clone box, agent setup, the | |
| 139 | 139 | pull request and merge box remotes, the tokens page and the OAuth | |
| 140 | − | consent's `iss` read them. | |
| 140 | + | consent's `iss` read them. `USERCONTENT_URL` (raw files and avatars, | |
| 141 | + | `apps/web/workers/usercontent.ts`): g1tusercontent.com hosted, else | |
| 142 | + | `<SITE_URL>/-/usercontent` unless set to a host of its own; | |
| 143 | + | `USERCONTENT_KEY` (made by `start.sh`) signs private files' addresses. | |
| 141 | 144 | - The API (`apps/api/src/addresses.rs`): `SITE_URL`, `API_URL` (the OAuth | |
| 142 | 145 | issuer) and `MCP_URL` (the protected resource; a path on the API's host | |
| 143 | 146 | self-hosted). |
| 394 | 394 | deleteRelease: (actor, path, id) => call("delete_release", { path, actor, id }), | |
| 395 | 395 | listFiles: (repoId, ref, limit) => call("list_files", { repoId, ref, skipDirs: [], limit }), | |
| 396 | 396 | rawBlobs: (repoId, hashes, maxBytes) => call("raw_blobs", { repoId, hashes, maxBytes }), | |
| 397 | + | rawFile: (repoId, ref, path, maxBytes) => call("raw_file", { repoId, ref, path, maxBytes }), | |
| 397 | 398 | commitFile: (repo, actor, file) => call("commit_file", { repo, actor, ...file }), | |
| 398 | 399 | land: (sourceId, actor, branch) => call("land", { sourceId, actor, branch }), | |
| 399 | 400 | compare: (repoId, viewer, base, head, baseBranch) => call("compare", { repoId, viewer, base, head, baseBranch }), |
| 294 | 294 | /** Blobs' bytes as standard base64, at most 100; `data` is null for one missing or over `maxBytes`. No viewer. */ | |
| 295 | 295 | rawBlobs(repoId: string, hashes: string[], maxBytes: number): Promise<RawBlob[]>; | |
| 296 | 296 | ||
| 297 | + | /** One file's bytes at a branch, tag or commit; null when missing or over `maxBytes`. No viewer: check access first. */ | |
| 298 | + | rawFile(repoId: string, ref: string, path: string, maxBytes: number): Promise<RawFile | null>; | |
| 299 | + | ||
| 297 | 300 | /** | |
| 298 | 301 | * Writes one file on a new branch made from the default branch's head, as | |
| 299 | 302 | * one commit by `actor`, for a change g1t proposes on their behalf (a | |
| ⋯ | |||
| 400 | 403 | /** One blob's bytes, standard base64; null when missing or too large. */ | |
| 401 | 404 | export type RawBlob = { hash: string; size: number; data: string | null }; | |
| 402 | 405 | ||
| 406 | + | /** One file's bytes, standard base64. */ | |
| 407 | + | export type RawFile = { size: number; data: string }; | |
| 408 | + | ||
| 403 | 409 | /** | |
| 404 | 410 | * What came of bringing a pull request up to date with the default branch | |
| 405 | 411 | * without a sandbox. `needs_agent` pushed nothing: the runner's `update` | |
| 852 | 852 | } | |
| 853 | 853 | } | |
| 854 | 854 | ||
| 855 | − | #[event(fetch)] | |
| 856 | − | async fn fetch(mut request: Request, env: Env, ctx: Context) -> Result<Response> { | |
| 857 | − | let packages = Packages::from_env(&env)?; | |
| 858 | − | let Some(method) = rpc_method(&request) else { | |
| 859 | − | if request.path().starts_with("/-/npm/") || request.path() == "/-/npm" { | |
| 860 | − | return packages.npm(request, &ctx).await; | |
| 855 | + | impl Packages { | |
| 856 | + | /// A registry request, to the registry its path names. | |
| 857 | + | async fn serve_registry(&self, request: Request, ctx: &Context) -> Result<Response> { | |
| 858 | + | let path = request.path(); | |
| 859 | + | if path.starts_with("/-/npm/") || path == "/-/npm" { | |
| 860 | + | return self.npm(request, ctx).await; | |
| 861 | 861 | } | |
| 862 | − | if request.path().starts_with("/-/composer/") { | |
| 863 | − | return packages.composer(request, &ctx).await; | |
| 862 | + | if path.starts_with("/-/composer/") { | |
| 863 | + | return self.composer(request, ctx).await; | |
| 864 | 864 | } | |
| 865 | − | if request.path().starts_with("/-/cargo/") { | |
| 866 | − | return packages.cargo(request, &ctx).await; | |
| 865 | + | if path.starts_with("/-/cargo/") { | |
| 866 | + | return self.cargo(request, ctx).await; | |
| 867 | 867 | } | |
| 868 | − | if request.path().starts_with("/-/maven/") { | |
| 869 | − | return packages.maven(request, &ctx).await; | |
| 868 | + | if path.starts_with("/-/maven/") { | |
| 869 | + | return self.maven(request, ctx).await; | |
| 870 | 870 | } | |
| 871 | − | if request.path().starts_with("/-/nuget/") { | |
| 872 | − | return packages.nuget(request, &ctx).await; | |
| 871 | + | if path.starts_with("/-/nuget/") { | |
| 872 | + | return self.nuget(request, ctx).await; | |
| 873 | 873 | } | |
| 874 | − | if request.path().starts_with("/-/rubygems/") { | |
| 875 | − | return packages.rubygems(request, &ctx).await; | |
| 874 | + | if path.starts_with("/-/rubygems/") { | |
| 875 | + | return self.rubygems(request, ctx).await; | |
| 876 | 876 | } | |
| 877 | − | return packages.registry(request, &ctx).await; | |
| 877 | + | self.registry(request, ctx).await | |
| 878 | + | } | |
| 879 | + | } | |
| 880 | + | ||
| 881 | + | /// The policy for registry answers: what they serve is a publisher's bytes, | |
| 882 | + | /// on the site's origin, so nothing in them may load or run. | |
| 883 | + | const NOTHING_RUNS: &str = "default-src 'none'; sandbox"; | |
| 884 | + | ||
| 885 | + | /// Whether a browser could open a body of this type as a page: HTML, SVG, | |
| 886 | + | /// any XML, or a type it does not know as data. Such a body is a download. | |
| 887 | + | fn opens_as_document(content_type: Option<&str>) -> bool { | |
| 888 | + | let kind = content_type.unwrap_or("").split(';').next().unwrap_or("").trim().to_ascii_lowercase(); | |
| 889 | + | if kind.ends_with("+xml") || kind.ends_with("/xml") || kind.contains("html") || kind.contains("svg") || kind.contains("xsl") { | |
| 890 | + | return true; | |
| 891 | + | } | |
| 892 | + | let data = kind == "text/plain" | |
| 893 | + | || kind == "application/json" | |
| 894 | + | || (kind.starts_with("application/") && kind.ends_with("+json")) | |
| 895 | + | || matches!( | |
| 896 | + | kind.as_str(), | |
| 897 | + | "application/octet-stream" | |
| 898 | + | | "application/gzip" | |
| 899 | + | | "application/x-gzip" | |
| 900 | + | | "application/zip" | |
| 901 | + | | "application/x-tar" | |
| 902 | + | | "application/java-archive" | |
| 903 | + | | "application/pgp-signature" | |
| 904 | + | | "image/png" | |
| 905 | + | | "image/jpeg" | |
| 906 | + | | "image/gif" | |
| 907 | + | | "image/webp" | |
| 908 | + | | "image/avif" | |
| 909 | + | ) | |
| 910 | + | || kind.starts_with("application/vnd."); | |
| 911 | + | !data | |
| 912 | + | } | |
| 913 | + | ||
| 914 | + | /// The headers every registry answer carries: no sniffing, nothing runs, | |
| 915 | + | /// and a type a browser would open is an attachment. The site's Worker | |
| 916 | + | /// sets the same (apps/web/app/lib/content-safety.ts); this keeps the | |
| 917 | + | /// service safe on its own. | |
| 918 | + | fn harden(mut response: Response) -> Result<Response> { | |
| 919 | + | let headers = response.headers_mut(); | |
| 920 | + | headers.set("x-content-type-options", "nosniff")?; | |
| 921 | + | headers.set("content-security-policy", NOTHING_RUNS)?; | |
| 922 | + | if headers.get("content-disposition")?.is_none() && opens_as_document(headers.get("content-type")?.as_deref()) { | |
| 923 | + | headers.set("content-disposition", "attachment")?; | |
| 924 | + | } | |
| 925 | + | Ok(response) | |
| 926 | + | } | |
| 927 | + | ||
| 928 | + | #[event(fetch)] | |
| 929 | + | async fn fetch(mut request: Request, env: Env, ctx: Context) -> Result<Response> { | |
| 930 | + | let packages = Packages::from_env(&env)?; | |
| 931 | + | let Some(method) = rpc_method(&request) else { | |
| 932 | + | let answer = packages.serve_registry(request, &ctx).await?; | |
| 933 | + | return harden(answer); | |
| 878 | 934 | }; | |
| 879 | 935 | let body: serde_json::Value = request.json().await?; | |
| 880 | 936 | match method.as_str() { | |
| ⋯ | |||
| 963 | 1019 | use super::*; | |
| 964 | 1020 | use serde_json::json; | |
| 965 | 1021 | ||
| 1022 | + | #[test] | |
| 1023 | + | fn publisher_documents_are_downloads() { | |
| 1024 | + | for kind in ["application/xml", "text/xml; charset=utf-8", "application/xhtml+xml", "text/html", "image/svg+xml", "application/vnd.foo+xml", "", "text/javascript"] { | |
| 1025 | + | assert!(opens_as_document(Some(kind)), "{kind}"); | |
| 1026 | + | } | |
| 1027 | + | assert!(opens_as_document(None)); | |
| 1028 | + | for kind in [ | |
| 1029 | + | "application/json", | |
| 1030 | + | "text/plain; charset=utf-8", | |
| 1031 | + | "application/vnd.oci.image.manifest.v1+json", | |
| 1032 | + | "application/vnd.npm.install-v1+json", | |
| 1033 | + | "application/octet-stream", | |
| 1034 | + | "application/java-archive", | |
| 1035 | + | "application/gzip", | |
| 1036 | + | "application/pgp-signature", | |
| 1037 | + | ] { | |
| 1038 | + | assert!(!opens_as_document(Some(kind)), "{kind}"); | |
| 1039 | + | } | |
| 1040 | + | } | |
| 1041 | + | ||
| 966 | 1042 | fn event(kind: &str, data: serde_json::Value) -> Event { | |
| 967 | 1043 | Event { | |
| 968 | 1044 | id: "evt_1".into(), | |