Sign in

flagon-io/g1t

public

Commit

Composer from the workspace's own repositories, and go get from g1t.sh

Composer - /-/composer/<workspace>/: packages.json, p2 metadata and ~dev, with notify-batch counting installs. Any repository with a root composer.json is a package under its name: semver tags are versions, branches dev- versions. Each version's composer.json, a git source and a zip dist, built from the commit on first request (export-ignore respected) and kept by digest. Pushing a tag publishes. - Kept current from git.push and repository events, with an hourly backfill and a sync_composer call. Access is the repository's: Basic (composer's http-basic with a g1t token) or Bearer. - repos: refs (branches and tags, annotated tags peeled), raw_file and raw_blobs for the packages service. - The site lists Composer packages with their README and install commands. Docs: the Composer guide. Go - Repository paths answer ?go-get=1 with go-import and go-source, so go get g1t.sh/<workspace>/<repo>/... works from git; private ones with GOPRIVATE and a token in .netrc. Docs: the Go guide. - Tree and blob pages take HEAD as the default branch (a 302), which go-source links use. Checked locally with the composer CLI: require, autoload, and update to a new tag after a push.

syntaqxcommitted Parent91d61caBrowse files
29 files+1812−130/29 viewed
+2−0
10131013 "g1t-kit",
10141014 "hex",
10151015 "hmac 0.12.1",
1016+ "miniz_oxide",
10161017 "serde",
10171018 "serde_json",
10181019 "sha1 0.10.7",
10241025 name = "g1t-repos"
10251026 version = "0.1.0"
10261027 dependencies = [
1028+ "base64 0.22.1",
10271029 "futures-util",
10281030 "g1t-contracts",
10291031 "g1t-kit",
+2−0
7676 { label: 'Packages', slug: 'guides/packages' },
7777 { label: 'Container images', slug: 'guides/containers' },
7878 { label: 'npm', slug: 'guides/npm' },
79+ { label: 'Composer', slug: 'guides/composer' },
80+ { label: 'Go modules', slug: 'guides/go' },
7981 { label: 'Secrets and variables', slug: 'guides/secrets-and-variables' },
8082 { label: 'Security', slug: 'guides/security' },
8183 ],
+110−0
1+---
2+title: Composer
3+description: Install a workspace's PHP packages with Composer, straight from its repositories on g1t.sh. Push a tag and it is released.
4+---
5+
6+Every workspace has a Composer repository of its own, made from its
7+repositories: one with a `composer.json` at its root is a package. There
8+is nothing to upload and nothing to keep in step. Push a tag and Composer
9+can install it; push to a branch and its `dev-` version follows.
10+
11+```text
12+https://g1t.sh/-/composer/<workspace>/
13+```
14+
15+## Make a repository a package
16+
17+Give the repository a `composer.json` at its root with a `name`:
18+
19+```json
20+{
21+ "name": "acme/http-client",
22+ "description": "Our HTTP client",
23+ "type": "library",
24+ "require": { "php": ">=8.1" },
25+ "autoload": { "psr-4": { "Acme\Http\\": "src/" } }
26+}
27+```
28+
29+Push it to the default branch, and the package appears in the workspace's
30+**Packages**, linked to the repository, with its README. Any vendor name
31+works; each name is one package in a workspace.
32+
33+## Versions
34+
35+| In git | Version |
36+| --- | --- |
37+| A tag that reads as a version: `v1.2.0`, `1.2.0`, `2.0.0-RC1`, `1.0.0-beta.2` | That version (`v1.2.0`) |
38+| A branch | `dev-<branch>`; a branch named like a version, `1.x`, is `1.x-dev` |
39+| The default branch | Its `dev-` version, marked as the default branch |
40+
41+Each version's requirements, autoloading and the rest are read from the
42+`composer.json` at that tag or branch, so a version installs exactly what
43+it was tagged with. Tags that do not read as versions, such as `nightly`,
44+are left out. `extra.branch-alias` in the default branch's `composer.json`
45+aliases it as usual (`"dev-main": "1.x-dev"`).
46+
47+To release, tag and push:
48+
49+```sh
50+git tag v1.3.0
51+git push origin v1.3.0
52+```
53+
54+Deleting a tag or branch takes its version away.
55+
56+## Install
57+
58+Add the workspace's repository to your project, then require the package:
59+
60+```sh
61+composer config repositories.acme composer https://g1t.sh/-/composer/acme/
62+composer require acme/http-client
63+```
64+
65+Packages install from a zip of the tag's commit, made the first time
66+anyone asks for it and kept from then on. Files the repository's
67+`.gitattributes` marks `export-ignore`, such as tests, are left out of it,
68+as `git archive` leaves them out. `--prefer-source` clones from g1t.sh
69+instead.
70+
71+## Private packages
72+
73+A package has its repository's visibility and
74+[roles](/guides/access-and-roles/): Read installs it. For a private one,
75+give Composer your username and an
76+[access token](https://g1t.sh/settings/tokens) (one with full access, or
77+with `packages:read` and `code:read`, so that `--prefer-source` can clone
78+too):
79+
80+```sh
81+composer config --global --auth http-basic.g1t.sh <you> <token>
82+```
83+
84+That writes `~/.composer/auth.json`, which stays out of the project. A
85+`Bearer` token works as well (`bearer.g1t.sh`). Without credentials, a
86+workspace's repository lists only its public packages.
87+
88+## In workflows
89+
90+A workflow's `G1T_TOKEN` can install the workspace's private packages:
91+
92+```yaml
93+jobs:
94+ test:
95+ runs-on: ubuntu-latest
96+ steps:
97+ - uses: actions/checkout@v4
98+ - run: composer config --global --auth http-basic.g1t.sh g1t "$G1T_TOKEN"
99+ env:
100+ G1T_TOKEN: ${{ secrets.G1T_TOKEN }}
101+ - run: composer install --no-interaction
102+```
103+
104+## Limits
105+
106+A zip is made of at most 10,000 files and 64 MB; a larger commit installs
107+from source (`composer install --prefer-source`). A package lists its
108+newest 300 version tags and 50 branches. Zips count toward the workspace's
109+package storage, as other packages do (see
110+[storage and pull limits](/guides/containers/#storage-and-pull-limits)).
+79−0
1+---
2+title: Go modules
3+description: go get a workspace's Go modules straight from its repositories on g1t.sh, public and private.
4+---
5+
6+A repository on g1t.sh is a Go module at its own address. `go get` finds
7+its code from the address, and fetches it with git:
8+
9+```sh
10+go get g1t.sh/acme/tools
11+go get g1t.sh/acme/tools/cmd/lint@v1.4.0
12+```
13+
14+The repository's `go.mod` names the module:
15+
16+```text
17+module g1t.sh/acme/tools
18+```
19+
20+Versions are its tags (`v1.4.0`); a pseudo-version names any other commit.
21+Packages in subdirectories are imported by their path, as above.
22+
23+## Public modules
24+
25+Public repositories need nothing: `go get` works as is, and the public Go
26+module proxy and checksum database serve them like any other public module.
27+
28+## Private modules
29+
30+Tell Go which modules are private, so it fetches them from g1t.sh directly
31+and does not ask the public proxy or checksum database about them:
32+
33+```sh
34+go env -w GOPRIVATE=g1t.sh/acme
35+```
36+
37+Then give git credentials for g1t.sh: your username and an
38+[access token](https://g1t.sh/settings/tokens) (full access, or with
39+`code:read`) in `~/.netrc` (`_netrc` on Windows):
40+
41+```text
42+machine g1t.sh
43+login <you>
44+password <token>
45+```
46+
47+or with a git credential helper, as for any clone (see
48+[Git](/guides/git/#authentication)). Reading a private module needs the
49+Read role on its repository.
50+
51+Keep `GOINSECURE` and `GOFLAGS=-insecure` unset: g1t.sh is served over
52+HTTPS, and neither is ever needed.
53+
54+## In workflows
55+
56+```yaml
57+jobs:
58+ build:
59+ runs-on: ubuntu-latest
60+ env:
61+ GOPRIVATE: g1t.sh/acme
62+ steps:
63+ - uses: actions/checkout@v4
64+ - uses: actions/setup-go@v5
65+ with:
66+ go-version: stable
67+ - run: git config --global url."https://g1t:${G1T_TOKEN}@g1t.sh/".insteadOf "https://g1t.sh/"
68+ env:
69+ G1T_TOKEN: ${{ secrets.G1T_TOKEN }}
70+ - run: go build ./...
71+```
72+
73+## How it works
74+
75+Go asks `https://g1t.sh/<workspace>/<repo>?go-get=1` and reads a
76+`go-import` tag pointing at `https://g1t.sh/<workspace>/<repo>.git`. Every
77+repository address answers, private ones included, and the answer names
78+nothing but that address; whether anything can be fetched is up to git and
79+your credentials.
+7−4
44 ---
55
66 A workspace can publish packages to g1t and install them from it, beside
7−the code they are built from: container images and npm packages, with
8−Composer, Cargo and Go to follow. Each registry speaks its tool's own
9−protocol, so `docker` and `npm` work with nothing but a login and an
10−address.
7+the code they are built from: container images, npm packages, Composer
8+packages and Go modules, with Cargo to follow. Each registry speaks its
9+tool's own protocol, so `docker`, `npm`, `composer` and `go` work with
10+nothing but a login and an address. Composer packages and Go modules are
11+read from the workspace's repositories: there is nothing to upload.
1112
1213 | Registry | Address | Guide |
1314 | --- | --- | --- |
1415 | Container images | `g1t.sh/<workspace>/<name>` | [Container images](/guides/containers/) |
1516 | npm | `https://g1t.sh/-/npm/`, for the scope `@<workspace>` | [npm](/guides/npm/) |
17+| Composer | `https://g1t.sh/-/composer/<workspace>/`, from the workspace's repositories | [Composer](/guides/composer/) |
18+| Go | `g1t.sh/<workspace>/<repo>`, straight from git | [Go modules](/guides/go/) |
1619
1720 ## Names
1821
+4−2
2222 try {
2323 const repo = await repos.get(path, viewer);
2424 if (!repo.ok) return;
25− const now = await repos.resolveBranch(repo.value.id, ref);
25+ // `HEAD` is the default branch, as git means it (Go's go-source links use it).
26+ const now = ref === "HEAD" ? repo.value.defaultBranch : await repos.resolveBranch(repo.value.id, ref);
2627 if (!now || now === ref) return;
2728 const url = new URL(request.url);
2829 to = renamedBranchPath(url.pathname, url.search, now);
3031 // A lookup that fails leaves the page a 404, never a 500.
3132 return;
3233 }
33− if (to) throw redirect(to, 301);
34+ // A rename is for good; what HEAD names can change.
35+ if (to) throw redirect(to, ref === "HEAD" ? 302 : 301);
3436 }
+22−0
1+import assert from "node:assert/strict";
2+import { test } from "node:test";
3+
4+import { goImport } from "./go-get.ts";
5+
6+test("go get finds a repository's code from its address and any package in it", () => {
7+ for (const path of ["/acme/lib", "/acme/lib/", "/acme/lib/pkg/sub", "/acme/lib.git"]) {
8+ const page = goImport(new URL(`https://g1t.sh${path}?go-get=1`));
9+ assert.ok(page, path);
10+ assert.match(page, /<meta name="go-import" content="g1t\.sh\/acme\/lib git https:\/\/g1t\.sh\/acme\/lib\.git">/);
11+ assert.match(page, /<meta name="go-source" content="g1t\.sh\/acme\/lib https:\/\/g1t\.sh\/acme\/lib /);
12+ }
13+});
14+
15+test("only go-get requests for a repository's address are answered", () => {
16+ assert.equal(goImport(new URL("https://g1t.sh/acme/lib")), null, "no go-get");
17+ assert.equal(goImport(new URL("https://g1t.sh/acme/lib?go-get=0")), null);
18+ assert.equal(goImport(new URL("https://g1t.sh/acme?go-get=1")), null, "a workspace is not a module");
19+ assert.equal(goImport(new URL("https://g1t.sh/acme/-/packages?go-get=1")), null, "workspace pages");
20+ assert.equal(goImport(new URL("https://g1t.sh/Acme/lib?go-get=1")), null);
21+ assert.equal(goImport(new URL("https://g1t.sh/acme/%22%3E?go-get=1")), null, "nothing is put in the page unchecked");
22+});
+32−0
1+/**
2+ * `go get g1t.sh/<workspace>/<repo>[/<package>]`: Go asks the address with
3+ * `?go-get=1` and reads where the module's code is from a `go-import` meta
4+ * tag. Every repository answers, public or private, without looking it up:
5+ * the answer only says where git would find it, and git then asks for
6+ * credentials (GOPRIVATE and .netrc) as it would for a clone.
7+ */
8+
9+const WORKSPACE = /^[a-z0-9](?:[a-z0-9]|-(?=[a-z0-9])){0,38}$/;
10+const REPO = /^[a-z0-9._-]{1,100}$/;
11+
12+/** The page Go reads for `url`, or null when it is not a go-get request for a repository. */
13+export function goImport(url: URL): string | null {
14+ if (url.searchParams.get("go-get") !== "1") return null;
15+ const [workspace, rawRepo] = url.pathname.split("/").filter(Boolean);
16+ if (!workspace || !rawRepo) return null;
17+ const repo = rawRepo.replace(/\.git$/, "");
18+ if (!WORKSPACE.test(workspace) || !REPO.test(repo) || repo.startsWith(".") || workspace === "-" || repo === "-") return null;
19+ const host = url.host;
20+ const prefix = `${host}/${workspace}/${repo}`;
21+ const home = `${url.protocol}//${prefix}`;
22+ return [
23+ "<!doctype html>",
24+ "<html><head>",
25+ `<meta name="go-import" content="${prefix} git ${home}.git">`,
26+ `<meta name="go-source" content="${prefix} ${home} ${home}/tree/HEAD{/dir} ${home}/blob/HEAD{/dir}/{file}#L{line}">`,
27+ "</head><body>",
28+ `go get ${prefix}`,
29+ "</body></html>",
30+ "",
31+ ].join("\n");
32+}
+9−0
2525 assert.equal(installCommands(pkg, null, "ada").install, "npm install @acme/ui");
2626 });
2727
28+test("a Composer package is required after its workspace's repository is added, with credentials for private ones", () => {
29+ const pkg = { ecosystem: "composer" as const, address: "g1t.sh/-/composer/acme/acme/lib", name: "acme/lib", workspace: "acme" };
30+ assert.deepEqual(installCommands(pkg, "v1.1.0", "ada"), {
31+ registry: "composer config repositories.acme composer https://g1t.sh/-/composer/acme/",
32+ login: "composer config --global --auth http-basic.g1t.sh ada YOUR_TOKEN",
33+ install: "composer require acme/lib:v1.1.0",
34+ });
35+});
36+
2837 test("a container image is pulled by its address and tag", () => {
2938 const pkg = { ecosystem: "container" as const, address: "g1t.sh/acme/web", name: "web", workspace: "acme" };
3039 assert.deepEqual(installCommands(pkg, "latest", "ada"), {
+4−1
6262 install: `npm install @${pkg.workspace}/${pkg.name}${version ? `@${version}` : ""}`,
6363 };
6464 case "composer":
65+ // The workspace's repository (in composer.json, needed for any
66+ // install), then the credentials a private package also needs.
6567 return {
66− login: `composer config repositories.${pkg.workspace} composer https://${host}/-/composer/${pkg.workspace}/`,
68+ registry: `composer config repositories.${pkg.workspace} composer https://${host}/-/composer/${pkg.workspace}/`,
69+ login: `composer config --global --auth http-basic.${host} ${you} YOUR_TOKEN`,
6770 install: `composer require ${pkg.name}${version ? `:${version}` : ""}`,
6871 };
6972 case "cargo":
+12−0
3535 }
3636 });
3737
38+test("the Composer registries go to the packages service", () => {
39+ for (const path of [
40+ "/-/composer/acme/packages.json",
41+ "/-/composer/acme/p2/acme/lib.json",
42+ "/-/composer/acme/p2/acme/lib~dev.json",
43+ `/-/composer/acme/dist/acme/lib/${"a".repeat(40)}.zip`,
44+ ]) {
45+ assert.equal(servicePath(path), "packages", path);
46+ }
47+ assert.equal(servicePath("/-/composer"), null);
48+});
49+
3850 test("git goes to repos, and everything else is the site's", () => {
3951 assert.equal(servicePath("/acme/web.git/info/refs"), "git");
4052 assert.equal(servicePath("/acme/web/git-receive-pack"), "git");
+3−1
99 const REGISTRY_PATH = /^\/v2(?:\/|$)/;
1010 /** The npm registry: `/-/npm/`, which `.npmrc` names for a workspace's scope. */
1111 const NPM_PATH = /^\/-\/npm(?:\/|$)/;
12+/** The Composer registries: `/-/composer/<workspace>/`, one per workspace. */
13+const COMPOSER_PATH = /^\/-\/composer\//;
1214
1315 export type ServicePath = "git" | "packages" | null;
1416
1517 export function servicePath(pathname: string): ServicePath {
16− if (REGISTRY_PATH.test(pathname) || NPM_PATH.test(pathname)) return "packages";
18+ if (REGISTRY_PATH.test(pathname) || NPM_PATH.test(pathname) || COMPOSER_PATH.test(pathname)) return "packages";
1719 if (GIT_PATH.test(pathname)) return "git";
1820 return null;
1921 }
+19−3
7373 const outcome = actionData as Outcome | undefined;
7474 const latest = tags.find((tag) => tag.tag === "latest")?.tag ?? tags[0]?.tag ?? null;
7575 const commands = installCommands(pkg, latest, username);
76− const npm = pkg.ecosystem === "npm";
76+ // npm and Composer versions are numbers; images are digests and tags.
77+ const npm = pkg.ecosystem !== "container";
78+ // Composer's versions are the repository's tags and branches: they
79+ // change in git, not here.
80+ const fromGit = pkg.ecosystem === "composer";
7781 // Signatures and attestations hang off the images they describe.
7882 const images = versions.filter((version) => !version.subject);
7983 const attached = (digest: string) => versions.filter((version) => version.subject === digest);
116120 {commands.registry && <CopyLine prompt text={commands.registry} />}
117121 {pkg.visibility === "private" && <CopyLine prompt text={commands.login} />}
118122 <CopyLine prompt text={commands.install} />
119− {npm && pkg.visibility === "private" && (
123+ {commands.registry && pkg.visibility === "private" && (
120124 <p className="text-xs text-faint">
121125 Put an{" "}
122126 <Link to="/settings/tokens" className="text-muted hover:text-fg">
146150 <h2 className="text-sm font-semibold">
147151 Versions <span className="font-normal text-faint">{images.length}</span>
148152 </h2>
153+ {fromGit && (
154+ <p className="text-xs text-faint">
155+ Each tag of {pkg.repo ? `${pkg.repo.namespace}/${pkg.repo.name}` : "its repository"} that reads as a version, and each
156+ branch as <code className="font-mono">dev-</code>, from its composer.json. Push a tag to publish one; delete it to take it away.
157+ </p>
158+ )}
149159 {images.length === 0 ? (
150160 <p className="text-sm text-muted">No versions are left.</p>
151161 ) : (
152162 <ul className="divide-y divide-line overflow-hidden rounded-xl border border-line bg-surface">
153163 {images.map((version) => (
154− <VersionRow key={version.id} version={version} attached={attached(version.digest)} canDelete={permissions.delete} npm={npm} />
164+ <VersionRow
165+ key={version.id}
166+ version={version}
167+ attached={attached(version.digest)}
168+ canDelete={permissions.delete && !fromGit}
169+ npm={npm}
170+ />
155171 ))}
156172 </ul>
157173 )}
+9−0
11 import { createRequestHandler } from "react-router";
22
33 import { finishResponse, withRequestPerf } from "../app/lib/perf.server";
4+import { goImport } from "../app/lib/go-get";
45 import { servicePath } from "../app/lib/registry-paths";
56
67 const requestHandler = createRequestHandler(
3637 // The container registry (`docker login g1t.sh`) and the npm registry
3738 // (`g1t.sh/-/npm/`) are the packages
3839 // service's, handed over the same way.
40+ // `go get g1t.sh/<workspace>/<repo>`: where its code is, from the
41+ // address alone, so it costs nothing and caches.
42+ const go = request.method === "GET" ? goImport(new URL(request.url)) : null;
43+ if (go) {
44+ return new Response(go, {
45+ headers: { "content-type": "text/html; charset=utf-8", "cache-control": "public, max-age=3600" },
46+ });
47+ }
3948 const service = servicePath(pathname);
4049 if (service === "git") {
4150 return proxyGit(env, request);
+9−1
243243 pub workspace: String,
244244 }
245245
246+/// `sync_composer`: read a repository's Composer package again now, as a
247+/// push would: made, updated or deleted from its branches, tags and
248+/// `composer.json`. Returns `bool`: whether it is a package.
249+#[derive(Clone, Debug, Serialize, Deserialize)]
250+pub struct SyncComposerArgs {
251+ pub repo_id: String,
252+}
253+
246254 #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
247255 pub struct PackageStorage {
248256 pub public_bytes: u64,
281289 for ecosystem in Ecosystem::ALL {
282290 assert!(ts.contains(&format!("\"{}\"", ecosystem.as_str())), "{}", ecosystem.as_str());
283291 }
284− for method in ["list_packages", "get_package", "delete_version", "delete_package", "set_package", "storage", "storage_all"] {
292+ for method in ["list_packages", "get_package", "delete_version", "delete_package", "set_package", "storage", "storage_all", "sync_composer"] {
285293 assert!(ts.contains(&format!("\"{method}\"")), "{method}");
286294 }
287295 }
+61−0
808808 /// The most blobs one `read_blobs` call reads.
809809 pub const MAX_READ_BLOBS: usize = 100;
810810
811+/// `refs`: a repository's branches and tags with the commit each points to
812+/// (annotated tags peeled), for services that follow them, such as the
813+/// packages service's Composer registry. No viewer: g1t's own services
814+/// only. Returns `Option<RepoRefs>`, null for a fork or an unknown id.
815+#[derive(Debug, Default, Serialize, Deserialize)]
816+#[serde(rename_all = "camelCase")]
817+pub struct RefsArgs {
818+ pub repo_id: String,
819+}
820+
821+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
822+pub struct GitRefEntry {
823+ /// The full ref: `refs/heads/main`, `refs/tags/v1.0.0`.
824+ pub name: String,
825+ pub commit: String,
826+}
827+
828+#[derive(Clone, Debug, Serialize, Deserialize)]
829+pub struct RepoRefs {
830+ pub repo: Repo,
831+ pub refs: Vec<GitRefEntry>,
832+}
833+
834+/// `raw_file`: one file's bytes at a ref or commit, base64, for g1t's own
835+/// services (no viewer). Returns `Option<RawFile>`: null when the file is
836+/// missing or larger than `max_bytes`.
837+#[derive(Debug, Default, Serialize, Deserialize)]
838+#[serde(rename_all = "camelCase")]
839+pub struct RawFileArgs {
840+ pub repo_id: String,
841+ #[serde(rename = "ref")]
842+ pub git_ref: String,
843+ pub path: String,
844+ pub max_bytes: u32,
845+}
846+
847+#[derive(Clone, Debug, Serialize, Deserialize)]
848+pub struct RawFile {
849+ pub size: u64,
850+ /// Standard base64.
851+ pub data: String,
852+}
853+
854+/// `raw_blobs`: blobs' bytes, base64, in the order asked, at most
855+/// [`MAX_READ_BLOBS`]; `data` is null for one missing or larger than
856+/// `max_bytes`. For g1t's own services. Returns `Vec<RawBlob>`.
857+#[derive(Debug, Default, Serialize, Deserialize)]
858+#[serde(rename_all = "camelCase")]
859+pub struct RawBlobsArgs {
860+ pub repo_id: String,
861+ pub hashes: Vec<String>,
862+ pub max_bytes: u32,
863+}
864+
865+#[derive(Clone, Debug, Serialize, Deserialize)]
866+pub struct RawBlob {
867+ pub hash: String,
868+ pub size: u64,
869+ pub data: Option<String>,
870+}
871+
811872 #[derive(Clone, Debug, Serialize, Deserialize)]
812873 pub struct BlobText {
813874 pub hash: String,
+1−0
610610 }),
611611 storage: (workspace) => call("storage", { workspace }),
612612 storageAll: () => call("storage_all", {}),
613+ syncComposer: (repoId) => call("sync_composer", { repo_id: repoId }),
613614 };
614615 }
615616
+3−1
109109 storage(workspace: string): Promise<PackageStorage>;
110110 /** For billing: every workspace with packages, from one query. */
111111 storageAll(): Promise<WorkspacePackageStorage[]>;
112+ /** Read a repository's Composer package again now, as a push would. Whether it is one. */
113+ syncComposer(repoId: string): Promise<boolean>;
112114 };
113115
114116 /** The RPC method behind each call, as the Rust service names them. */
115−export const PACKAGES_METHODS = ["list_packages", "get_package", "delete_version", "delete_package", "set_package", "storage", "storage_all"] as const;
117+export const PACKAGES_METHODS = ["list_packages", "get_package", "delete_version", "delete_package", "set_package", "storage", "storage_all", "sync_composer"] as const;
+1−0
2020 hmac = "0.12"
2121 sha2 = { version = "0.10", features = ["compress"] }
2222 sha1 = "0.10"
23+miniz_oxide = "0.9"
2324
2425 # wasm-opt at -O1: about the same gzipped size as -O in a tenth of the
2526 # time (docs/DEPLOYING.md, "Build speed").
+110−0
66 // outsider, `bo` (OUTSIDER_TOKEN).
77 // - Repos knows two repositories of acme: `web` (private) and `site`
88 // (public).
9+// - Repos also keeps a small git store for `lib` (rep_lib), a private PHP
10+// library: tags v1.0.0 and v1.1.0, branches main and feature, for the
11+// Composer registry (refs, raw_file, raw_blobs, list_files, all_ids).
12+// Signing in with the password `g1t_push` (any request with Basic
13+// credentials) tags v1.2.0 on a new commit, as a push would; then call
14+// `sync_composer` as the git.push event would.
915 // - Events takes every event and audit entry and logs them.
1016 // - Billing says acme is free, with FREE_PRIVATE_BYTES of private package
1117 // storage and 10 GB public.
1319 const REPOS = {
1420 web: { isPrivate: true },
1521 site: { isPrivate: false },
22+ lib: { isPrivate: true },
1623 };
1724
25+// The library's commits: each its files. Hashes are made up, 40 hex each.
26+const composerJson = JSON.stringify(
27+ {
28+ name: "acme/lib",
29+ description: "Greets people",
30+ type: "library",
31+ license: "MIT",
32+ require: { php: ">=8.1" },
33+ autoload: { "psr-4": { "Acme\\Lib\\": "src/" } },
34+ "require-dev": { "phpunit/phpunit": "^11" },
35+ },
36+ null,
37+ 2,
38+);
39+const greeter = (body) => `<?php\n\nnamespace Acme\\Lib;\n\nfinal class Greeter\n{\n${body}\n}\n`;
40+const COMMITS = {
41+ ["1".repeat(40)]: {
42+ "composer.json": composerJson,
43+ "src/Greeter.php": greeter(' public function hello(string $name): string { return "Hello, $name!"; }'),
44+ "README.md": "# acme/lib\n\nGreets people.\n",
45+ ".gitattributes": "/tests export-ignore\n",
46+ "tests/GreeterTest.php": "<?php // not in the archive\n",
47+ },
48+ ["2".repeat(40)]: {
49+ "composer.json": composerJson,
50+ "src/Greeter.php": greeter(
51+ ' public function hello(string $name): string { return "Hello, $name!"; }\n public function bye(string $name): string { return "Bye, $name."; }',
52+ ),
53+ "README.md": "# acme/lib\n\nGreets people, and says goodbye.\n",
54+ ".gitattributes": "/tests export-ignore\n",
55+ "tests/GreeterTest.php": "<?php // not in the archive\n",
56+ },
57+ ["3".repeat(40)]: { "composer.json": composerJson, "src/Greeter.php": greeter(" // a feature in progress") },
58+ ["4".repeat(40)]: {
59+ "composer.json": composerJson,
60+ "src/Greeter.php": greeter(
61+ ' public function hello(string $name): string { return "Hi, $name!"; }\n public function bye(string $name): string { return "Bye, $name."; }',
62+ ),
63+ "README.md": "# acme/lib\n\nv1.2.\n",
64+ },
65+};
66+const REFS = [
67+ { name: "refs/heads/main", commit: "2".repeat(40) },
68+ { name: "refs/heads/feature", commit: "3".repeat(40) },
69+ { name: "refs/tags/v1.0.0", commit: "1".repeat(40) },
70+ { name: "refs/tags/v1.1.0", commit: "2".repeat(40) },
71+];
72+const blobHash = (commit, path) => `${commit.slice(0, 8)}${Buffer.from(path).toString("hex")}`;
73+const base64 = (text) => Buffer.from(text).toString("base64");
74+function blob(hash) {
75+ for (const [commit, files] of Object.entries(COMMITS)) {
76+ for (const [path, text] of Object.entries(files)) if (blobHash(commit, path) === hash) return text;
77+ }
78+ return null;
79+}
80+const libRepo = () => ({
81+ id: "rep_lib",
82+ namespace: "acme",
83+ name: "lib",
84+ description: null,
85+ isPrivate: true,
86+ ownerId: "usr_dev",
87+ defaultBranch: "main",
88+ forkOf: null,
89+ createdAt: "2026-10-01T00:00:00.000Z",
90+});
91+
92+function push() {
93+ if (!REFS.some((r) => r.name === "refs/tags/v1.2.0")) {
94+ REFS.push({ name: "refs/tags/v1.2.0", commit: "4".repeat(40) });
95+ REFS[0].commit = "4".repeat(40);
96+ }
97+}
98+
1899 function user(env, secret) {
100+ if (secret === "g1t_push") {
101+ push();
102+ return null;
103+ }
19104 if (secret === env.DEV_TOKEN) {
20105 return {
21106 id: "usr_dev",
47132 const method = new URL(request.url).pathname.replace(/^\/rpc\//, "");
48133 const args = await request.json().catch(() => ({}));
49134 const json = (value) => Response.json(value);
135+
50136 switch (method) {
137+ case "refs":
138+ return json(args.repoId === "rep_lib" ? { repo: libRepo(), refs: REFS } : null);
139+ case "raw_file": {
140+ const commit = REFS.find((r) => r.name === args.ref || r.name === `refs/heads/${args.ref}`)?.commit ?? args.ref;
141+ const text = args.repoId === "rep_lib" ? COMMITS[commit]?.[args.path] : undefined;
142+ return json(text === undefined ? null : { size: text.length, data: base64(text) });
143+ }
144+ case "list_files": {
145+ const files = args.repoId === "rep_lib" ? COMMITS[args.ref] : undefined;
146+ return json({
147+ commit: files ? args.ref : null,
148+ files: Object.keys(files ?? {}).map((path) => ({ path, hash: blobHash(args.ref, path) })),
149+ truncated: false,
150+ });
151+ }
152+ case "raw_blobs":
153+ return json(
154+ (args.hashes ?? []).map((hash) => {
155+ const text = blob(hash);
156+ return { hash, size: text?.length ?? 0, data: text === null ? null : base64(text) };
157+ }),
158+ );
159+ case "all_ids":
160+ return json(args.after ? { ids: [], next: null } : { ids: ["rep_web", "rep_lib"], next: null });
51161 case "user_for_git_credentials":
52162 return json(user(env, args.secret));
53163 case "get": {
+1−0
33 "name": "g1t-packages-stubs",
44 "main": "stubs.js",
55 "compatibility_date": "2026-09-26",
6+ "compatibility_flags": ["nodejs_compat"],
67 "vars": { "DEV_TOKEN": "g1t_devtoken", "MEMBER_TOKEN": "g1t_member", "OUTSIDER_TOKEN": "g1t_outsider", "FREE_PRIVATE_BYTES": "50000000" }
78 }
+11−0
1+-- Composer packages are found in the workspace's repositories, not
2+-- uploaded: the backfill walks every repository once, a page an hour, so
3+-- repositories that had a composer.json before the registry existed are
4+-- found without waiting for their next push.
5+CREATE TABLE composer_backfill (
6+ key TEXT PRIMARY KEY,
7+ -- The last repository id done.
8+ after TEXT,
9+ finished_at TEXT
10+);
11+CREATE INDEX versions_commit ON versions (package_id, digest);
+471−0
1+//! What the Composer registry needs that does not touch the network:
2+//! package names, versions read from tags and branches as Composer reads
3+//! them, the metadata Composer installs from, `export-ignore`, and zips.
4+//!
5+//! A Composer package is not uploaded: it is a repository of the workspace
6+//! with a `composer.json` at its root. Each tag that reads as a version is a
7+//! version, and each branch a `dev-` one, with the metadata of that ref's
8+//! `composer.json`, a git source on g1t.sh, and a zip of the commit as its
9+//! dist, made when it is first asked for.
10+
11+use serde_json::{Map, Value, json};
12+
13+/// Composer's rule for a package name: `vendor/name`, lowercase, words
14+/// joined by `.`, `_` or `-`.
15+pub fn valid_name(name: &str) -> bool {
16+ let Some((vendor, project)) = name.split_once('/') else {
17+ return false;
18+ };
19+ let part = |text: &str, double_dash: bool| {
20+ let bytes = text.as_bytes();
21+ if bytes.is_empty() || !bytes[0].is_ascii_alphanumeric() || !bytes[bytes.len() - 1].is_ascii_alphanumeric() {
22+ return false;
23+ }
24+ let mut run = String::new();
25+ for byte in bytes {
26+ if byte.is_ascii_lowercase() || byte.is_ascii_digit() {
27+ if !run.is_empty() && !(run == "." || run == "_" || run == "-" || (double_dash && run == "--")) {
28+ return false;
29+ }
30+ run.clear();
31+ } else if matches!(byte, b'.' | b'_' | b'-') {
32+ run.push(*byte as char);
33+ } else {
34+ return false;
35+ }
36+ }
37+ true
38+ };
39+ !project.contains('/') && part(vendor, false) && part(project, true) && name.len() <= 200
40+}
41+
42+/// A version Composer can install, as Packagist names it: the tag or
43+/// branch's own `version`, and `version_normalized` to compare by.
44+#[derive(Clone, Debug, PartialEq, Eq)]
45+pub struct Version {
46+ pub version: String,
47+ pub normalized: String,
48+}
49+
50+impl Version {
51+ pub fn is_dev(&self) -> bool {
52+ self.normalized.starts_with("dev-") || self.normalized.ends_with("-dev")
53+ }
54+}
55+
56+/// The stability words Composer reads after a version, as it writes them.
57+fn stability(word: &str) -> Option<&'static str> {
58+ match word.to_ascii_lowercase().as_str() {
59+ "stable" => Some(""),
60+ "beta" | "b" => Some("beta"),
61+ "rc" => Some("RC"),
62+ "alpha" | "a" => Some("alpha"),
63+ "patch" | "pl" | "p" => Some("patch"),
64+ _ => None,
65+ }
66+}
67+
68+/// A tag's version, the way Composer's version parser reads it:
69+/// `v1.2.3`, `1.2`, `1.0.0-beta.2`, `2.0.0-RC1`. Anything else is not a
70+/// version, and its tag is left out.
71+pub fn tag_version(tag: &str) -> Option<Version> {
72+ let text = tag.strip_prefix('v').or_else(|| tag.strip_prefix('V')).unwrap_or(tag);
73+ let digits_end = text.find(|c: char| !(c.is_ascii_digit() || c == '.')).unwrap_or(text.len());
74+ let (numbers, rest) = text.split_at(digits_end);
75+ let numbers = numbers.trim_end_matches('.');
76+ let parts: Vec<&str> = numbers.split('.').collect();
77+ if parts.is_empty() || parts.len() > 4 || parts.iter().any(|p| p.is_empty() || p.len() > 9) {
78+ return None;
79+ }
80+ if numbers.len() != text[..digits_end].len() && !rest.is_empty() {
81+ // `1.2.` followed by more is not a version.
82+ return None;
83+ }
84+ let mut normalized: Vec<String> = parts.iter().map(|p| p.trim_start_matches('0').to_owned()).map(|p| if p.is_empty() { "0".into() } else { p }).collect();
85+ while normalized.len() < 4 {
86+ normalized.push("0".into());
87+ }
88+ let mut normalized = normalized.join(".");
89+ let rest = rest.trim_start_matches(['-', '_', '.']);
90+ if !rest.is_empty() {
91+ let (word, tail) = rest.split_at(rest.find(|c: char| !c.is_ascii_alphabetic()).unwrap_or(rest.len()));
92+ let word = stability(word)?;
93+ let number = tail.trim_start_matches(['-', '.']);
94+ let (number, dev) = match number.strip_suffix("dev") {
95+ Some(n) => (n.trim_end_matches(['-', '.']), true),
96+ None => (number, false),
97+ };
98+ if !number.bytes().all(|b| b.is_ascii_digit() || b == b'.') {
99+ return None;
100+ }
101+ if !word.is_empty() {
102+ normalized.push('-');
103+ normalized.push_str(word);
104+ normalized.push_str(&number.replace('.', ""));
105+ }
106+ if dev {
107+ normalized.push_str("-dev");
108+ }
109+ }
110+ Some(Version { version: tag.to_owned(), normalized })
111+}
112+
113+/// A branch's version: `dev-<branch>`, or for a branch named like a
114+/// version (`1.x`, `2.1`) that version's `-dev`, as Composer reads them.
115+pub fn branch_version(branch: &str) -> Version {
116+ let text = branch.strip_prefix('v').unwrap_or(branch);
117+ let parts: Vec<&str> = text.split('.').collect();
118+ let numeric = !text.is_empty()
119+ && parts.len() <= 4
120+ && parts[0].bytes().all(|b| b.is_ascii_digit())
121+ && !parts[0].is_empty()
122+ && parts.iter().all(|p| !p.is_empty() && (p.bytes().all(|b| b.is_ascii_digit()) || matches!(*p, "x" | "X" | "*")));
123+ if !numeric {
124+ return Version { version: format!("dev-{branch}"), normalized: format!("dev-{branch}") };
125+ }
126+ let mut nines: Vec<String> = parts
127+ .iter()
128+ .map(|p| if p.bytes().all(|b| b.is_ascii_digit()) { (*p).to_owned() } else { "9999999".to_owned() })
129+ .collect();
130+ while nines.len() < 4 {
131+ nines.push("9999999".to_owned());
132+ }
133+ let version = if parts.last().is_some_and(|p| matches!(*p, "x" | "X" | "*")) {
134+ format!("{branch}-dev")
135+ } else {
136+ format!("{branch}.x-dev")
137+ };
138+ Version { version, normalized: format!("{}-dev", nines.join(".")) }
139+}
140+
141+/// The fields of `composer.json` a version's metadata keeps, as Packagist
142+/// serves them.
143+const KEPT: [&str; 25] = [
144+ "name",
145+ "description",
146+ "keywords",
147+ "homepage",
148+ "readme",
149+ "license",
150+ "authors",
151+ "type",
152+ "support",
153+ "funding",
154+ "autoload",
155+ "autoload-dev",
156+ "extra",
157+ "bin",
158+ "include-path",
159+ "target-dir",
160+ "require",
161+ "require-dev",
162+ "suggest",
163+ "provide",
164+ "replace",
165+ "conflict",
166+ "archive",
167+ "abandoned",
168+ "notification-url",
169+];
170+
171+/// Where one version installs from.
172+pub struct Origin<'a> {
173+ /// `https://g1t.sh/acme/lib.git`.
174+ pub git_url: &'a str,
175+ /// `https://g1t.sh/-/composer/acme/dist/acme/lib/<commit>.zip`.
176+ pub dist_url: &'a str,
177+ pub commit: &'a str,
178+ /// Set on the default branch's version, as Composer 2 marks it.
179+ pub default_branch: bool,
180+}
181+
182+/// A version's metadata: its `composer.json`, kept fields only, named
183+/// `name`, with its version, source and dist.
184+pub fn version_entry(composer: &Value, name: &str, version: &Version, origin: &Origin<'_>) -> Value {
185+ let mut entry = Map::new();
186+ if let Value::Object(fields) = composer {
187+ for key in KEPT {
188+ if let Some(value) = fields.get(key) {
189+ entry.insert(key.to_owned(), value.clone());
190+ }
191+ }
192+ }
193+ // A single license is a list of one, as Packagist writes it.
194+ if let Some(Value::String(license)) = entry.get("license") {
195+ let license = license.clone();
196+ entry.insert("license".into(), json!([license]));
197+ }
198+ entry.entry("type").or_insert(json!("library"));
199+ entry.insert("name".into(), json!(name));
200+ entry.insert("version".into(), json!(version.version));
201+ entry.insert("version_normalized".into(), json!(version.normalized));
202+ entry.insert("source".into(), json!({ "type": "git", "url": origin.git_url, "reference": origin.commit }));
203+ entry.insert(
204+ "dist".into(),
205+ json!({ "type": "zip", "url": origin.dist_url, "reference": origin.commit, "shasum": "" }),
206+ );
207+ if origin.default_branch {
208+ entry.insert("default-branch".into(), json!(true));
209+ }
210+ Value::Object(entry)
211+}
212+
213+/// `p2/<vendor>/<name>.json`: every version, as Composer reads them
214+/// (not minified: each entry whole).
215+pub fn p2(name: &str, versions: &[Value]) -> Value {
216+ json!({ "packages": { name: versions } })
217+}
218+
219+/// `packages.json` of a workspace.
220+pub fn root(workspace: &str, available: &[String]) -> Value {
221+ json!({
222+ "packages": [],
223+ "metadata-url": format!("/-/composer/{workspace}/p2/%package%.json"),
224+ "available-packages": available,
225+ "notify-batch": format!("/-/composer/{workspace}/downloads"),
226+ })
227+}
228+
229+/// Whether `pattern` matches `text`, `*` any run of characters but `/`,
230+/// `?` any one.
231+fn glob(pattern: &[u8], text: &[u8]) -> bool {
232+ match (pattern.first(), text.first()) {
233+ (None, None) => true,
234+ (Some(b'*'), _) => {
235+ glob(&pattern[1..], text) || (!text.is_empty() && text[0] != b'/' && glob(pattern, &text[1..]))
236+ }
237+ (Some(b'?'), Some(c)) if *c != b'/' => glob(&pattern[1..], &text[1..]),
238+ (Some(p), Some(c)) if p == c => glob(&pattern[1..], &text[1..]),
239+ _ => false,
240+ }
241+}
242+
243+/// The patterns a `.gitattributes` marks `export-ignore`.
244+pub fn export_ignores(gitattributes: &str) -> Vec<String> {
245+ gitattributes
246+ .lines()
247+ .filter_map(|line| {
248+ let line = line.trim();
249+ if line.starts_with('#') {
250+ return None;
251+ }
252+ let mut words = line.split_whitespace();
253+ let pattern = words.next()?;
254+ words.any(|attr| attr == "export-ignore").then(|| pattern.to_owned())
255+ })
256+ .collect()
257+}
258+
259+/// Whether a file is left out of an archive by an `export-ignore`
260+/// pattern: a pattern without `/` matches a name at any depth, one with a
261+/// `/` matches from the root, and a directory's pattern everything in it.
262+pub fn ignored(patterns: &[String], path: &str) -> bool {
263+ patterns.iter().any(|pattern| {
264+ let (pattern, dir_only) = match pattern.strip_suffix('/') {
265+ Some(p) => (p, true),
266+ None => (pattern.as_str(), false),
267+ };
268+ let anchored = pattern.contains('/');
269+ let pattern = pattern.trim_start_matches('/');
270+ let parts: Vec<&str> = path.split('/').collect();
271+ // Each directory the file is in, and (unless only directories
272+ // match) the file itself.
273+ let candidates = (1..=parts.len()).filter(|n| !dir_only || *n < parts.len());
274+ for n in candidates {
275+ let prefix = parts[..n].join("/");
276+ let subject = if anchored { prefix.as_str() } else { parts[n - 1] };
277+ if glob(pattern.as_bytes(), subject.as_bytes()) {
278+ return true;
279+ }
280+ }
281+ false
282+ })
283+}
284+
285+/// CRC-32, as zip records each file's.
286+fn crc32(bytes: &[u8]) -> u32 {
287+ let mut table = [0u32; 256];
288+ for (i, entry) in table.iter_mut().enumerate() {
289+ let mut c = i as u32;
290+ for _ in 0..8 {
291+ c = if c & 1 != 0 { 0xEDB8_8320 ^ (c >> 1) } else { c >> 1 };
292+ }
293+ *entry = c;
294+ }
295+ let mut crc = 0xFFFF_FFFFu32;
296+ for byte in bytes {
297+ crc = table[((crc ^ u32::from(*byte)) & 0xFF) as usize] ^ (crc >> 8);
298+ }
299+ !crc
300+}
301+
302+/// A zip of `files` (path and bytes), deflated where that is smaller. No
303+/// directory entries, a fixed time, so the same files make the same zip.
304+pub fn zip(files: &[(String, Vec<u8>)]) -> Vec<u8> {
305+ let mut out = Vec::new();
306+ let mut central = Vec::new();
307+ for (path, data) in files {
308+ let crc = crc32(data);
309+ let deflated = miniz_oxide::deflate::compress_to_vec(data, 6);
310+ let (method, body): (u16, &[u8]) = if deflated.len() < data.len() { (8, &deflated) } else { (0, data) };
311+ let offset = out.len() as u32;
312+ let name = path.as_bytes();
313+ let header = |sig: u32, central_entry: bool| {
314+ let mut h = Vec::with_capacity(46 + name.len());
315+ h.extend_from_slice(&sig.to_le_bytes());
316+ if central_entry {
317+ h.extend_from_slice(&0x031Eu16.to_le_bytes()); // made by: Unix, 3.0
318+ }
319+ h.extend_from_slice(&20u16.to_le_bytes()); // version needed
320+ h.extend_from_slice(&0x0800u16.to_le_bytes()); // UTF-8 names
321+ h.extend_from_slice(&method.to_le_bytes());
322+ h.extend_from_slice(&0u16.to_le_bytes()); // time
323+ h.extend_from_slice(&0x0021u16.to_le_bytes()); // date: 1980-01-01
324+ h.extend_from_slice(&crc.to_le_bytes());
325+ h.extend_from_slice(&(body.len() as u32).to_le_bytes());
326+ h.extend_from_slice(&(data.len() as u32).to_le_bytes());
327+ h.extend_from_slice(&(name.len() as u16).to_le_bytes());
328+ h.extend_from_slice(&0u16.to_le_bytes()); // extra
329+ if central_entry {
330+ h.extend_from_slice(&0u16.to_le_bytes()); // comment
331+ h.extend_from_slice(&0u16.to_le_bytes()); // disk
332+ h.extend_from_slice(&0u16.to_le_bytes()); // internal attributes
333+ h.extend_from_slice(&(0o100644u32 << 16).to_le_bytes());
334+ h.extend_from_slice(&offset.to_le_bytes());
335+ }
336+ h.extend_from_slice(name);
337+ h
338+ };
339+ out.extend_from_slice(&header(0x0403_4b50, false));
340+ out.extend_from_slice(body);
341+ central.extend_from_slice(&header(0x0201_4b50, true));
342+ }
343+ let central_offset = out.len() as u32;
344+ out.extend_from_slice(&central);
345+ out.extend_from_slice(&0x0605_4b50u32.to_le_bytes());
346+ out.extend_from_slice(&[0, 0, 0, 0]); // disks
347+ out.extend_from_slice(&(files.len() as u16).to_le_bytes());
348+ out.extend_from_slice(&(files.len() as u16).to_le_bytes());
349+ out.extend_from_slice(&(central.len() as u32).to_le_bytes());
350+ out.extend_from_slice(&central_offset.to_le_bytes());
351+ out.extend_from_slice(&0u16.to_le_bytes());
352+ out
353+}
354+
355+#[cfg(test)]
356+mod tests {
357+ use super::*;
358+
359+ #[test]
360+ fn names_follow_composers_rule() {
361+ for good in ["acme/lib", "acme-co/http.client", "a1/b_2", "acme/my--lib"] {
362+ assert!(valid_name(good), "{good}");
363+ }
364+ for bad in ["acme", "Acme/lib", "acme/lib/x", "-acme/lib", "acme/lib-", "acme/l b", "acme/a..b", "ac--me/lib"] {
365+ assert!(!valid_name(bad), "{bad}");
366+ }
367+ }
368+
369+ #[test]
370+ fn tags_read_as_composer_reads_them() {
371+ let v = |tag: &str| tag_version(tag).map(|v| v.normalized);
372+ assert_eq!(v("v1.2.3").as_deref(), Some("1.2.3.0"));
373+ assert_eq!(v("1.2").as_deref(), Some("1.2.0.0"));
374+ assert_eq!(v("2.0.0-RC1").as_deref(), Some("2.0.0.0-RC1"));
375+ assert_eq!(v("1.0.0-beta.2").as_deref(), Some("1.0.0.0-beta2"));
376+ assert_eq!(v("1.0.0-alpha").as_deref(), Some("1.0.0.0-alpha"));
377+ assert_eq!(v("1.0.0-p1").as_deref(), Some("1.0.0.0-patch1"));
378+ assert_eq!(v("1.0.0-stable").as_deref(), Some("1.0.0.0"));
379+ assert_eq!(v("01.02.003").as_deref(), Some("1.2.3.0"));
380+ assert_eq!(tag_version("v1.0.0").unwrap().version, "v1.0.0", "the tag's own name is the version");
381+ for bad in ["latest", "release-1", "1.0.0-nope", "1.2.3.4.5", "v", ""] {
382+ assert_eq!(tag_version(bad), None, "{bad}");
383+ }
384+ assert!(!tag_version("1.0.0").unwrap().is_dev());
385+ }
386+
387+ #[test]
388+ fn branches_are_dev_versions() {
389+ assert_eq!(branch_version("main"), Version { version: "dev-main".into(), normalized: "dev-main".into() });
390+ assert_eq!(
391+ branch_version("1.x"),
392+ Version { version: "1.x-dev".into(), normalized: "1.9999999.9999999.9999999-dev".into() }
393+ );
394+ assert_eq!(
395+ branch_version("2.1"),
396+ Version { version: "2.1.x-dev".into(), normalized: "2.1.9999999.9999999-dev".into() }
397+ );
398+ assert_eq!(branch_version("feature/x").version, "dev-feature/x");
399+ assert!(branch_version("main").is_dev());
400+ }
401+
402+ #[test]
403+ fn a_versions_metadata_is_its_composer_json_with_where_it_installs_from() {
404+ let composer = json!({
405+ "name": "acme/lib",
406+ "description": "A library",
407+ "license": "MIT",
408+ "require": { "php": ">=8.1" },
409+ "autoload": { "psr-4": { "Acme\\Lib\\": "src/" } },
410+ "scripts": { "test": "phpunit" },
411+ "config": { "sort-packages": true },
412+ });
413+ let version = tag_version("v1.0.0").unwrap();
414+ let origin = Origin {
415+ git_url: "https://g1t.sh/acme/lib.git",
416+ dist_url: "https://g1t.sh/-/composer/acme/dist/acme/lib/abc.zip",
417+ commit: "abc",
418+ default_branch: false,
419+ };
420+ let entry = version_entry(&composer, "acme/lib", &version, &origin);
421+ assert_eq!(entry["version"], "v1.0.0");
422+ assert_eq!(entry["version_normalized"], "1.0.0.0");
423+ assert_eq!(entry["license"], json!(["MIT"]));
424+ assert_eq!(entry["type"], "library");
425+ assert_eq!(entry["require"]["php"], ">=8.1");
426+ assert_eq!(entry["autoload"]["psr-4"]["Acme\\Lib\\"], "src/");
427+ assert_eq!(entry["source"], json!({ "type": "git", "url": "https://g1t.sh/acme/lib.git", "reference": "abc" }));
428+ assert_eq!(entry["dist"]["type"], "zip");
429+ assert_eq!(entry["dist"]["reference"], "abc");
430+ assert!(entry.get("scripts").is_none(), "only what installs");
431+ assert!(entry.get("config").is_none());
432+ assert!(entry.get("default-branch").is_none());
433+ let p2 = p2("acme/lib", &[entry]);
434+ assert_eq!(p2["packages"]["acme/lib"][0]["version"], "v1.0.0");
435+ let root = root("acme", &["acme/lib".to_owned()]);
436+ assert_eq!(root["metadata-url"], "/-/composer/acme/p2/%package%.json");
437+ assert_eq!(root["available-packages"], json!(["acme/lib"]));
438+ }
439+
440+ #[test]
441+ fn export_ignore_leaves_files_out_as_git_archive_does() {
442+ let patterns = export_ignores("# dev only\n/tests export-ignore\n.github/ export-ignore\n*.md export-ignore\n/phpunit.xml.dist export-ignore\nsrc/* text\n");
443+ assert_eq!(patterns, ["/tests", ".github/", "*.md", "/phpunit.xml.dist"]);
444+ assert!(ignored(&patterns, "tests/LibTest.php"));
445+ assert!(ignored(&patterns, ".github/workflows/ci.yml"));
446+ assert!(ignored(&patterns, "README.md"));
447+ assert!(ignored(&patterns, "docs/guide.md"));
448+ assert!(ignored(&patterns, "phpunit.xml.dist"));
449+ assert!(!ignored(&patterns, "src/Lib.php"));
450+ assert!(!ignored(&patterns, "src/tests/Helper.php"), "/tests is anchored at the root");
451+ assert!(!ignored(&patterns, "composer.json"));
452+ }
453+
454+ #[test]
455+ fn a_zip_holds_its_files_and_their_checksums() {
456+ assert_eq!(crc32(b"123456789"), 0xCBF4_3926);
457+ let files = vec![
458+ ("composer.json".to_owned(), br#"{"name":"acme/lib"}"#.to_vec()),
459+ ("src/Lib.php".to_owned(), "<?php\n".repeat(100).into_bytes()),
460+ ];
461+ let zip = zip(&files);
462+ assert_eq!(&zip[..4], &0x0403_4b50u32.to_le_bytes());
463+ // The end record counts both files.
464+ let end = &zip[zip.len() - 22..];
465+ assert_eq!(&end[..4], &0x0605_4b50u32.to_le_bytes());
466+ assert_eq!(u16::from_le_bytes([end[10], end[11]]), 2);
467+ // The repetitive file was deflated, the short one stored.
468+ assert!(zip.len() < 600, "{}", zip.len());
469+ assert_eq!(super::zip(&files), zip, "the same files make the same zip");
470+ }
471+}
+586−0
1+//! The Composer registry: `g1t.sh/-/composer/<workspace>/`, one per
2+//! workspace, built from its repositories (composer.rs says how).
3+//!
4+//! ```sh
5+//! composer config repositories.acme composer https://g1t.sh/-/composer/acme/
6+//! composer config --global --auth http-basic.g1t.sh <you> <g1t token>
7+//! composer require acme/lib
8+//! ```
9+//!
10+//! Nothing is uploaded. A repository's versions are read again when it is
11+//! pushed to (`git.push`), restored, renamed or moved, and once for every
12+//! repository by the backfill; a request only reads what that left. A zip
13+//! of a commit is made the first time it is asked for, and kept by its
14+//! digest like every file here.
15+
16+use std::collections::{HashMap, HashSet};
17+
18+use base64::Engine;
19+use base64::engine::general_purpose::STANDARD;
20+use g1t_contracts::User;
21+use g1t_contracts::events::PackageEvent;
22+use g1t_contracts::new_id;
23+use g1t_contracts::repos::{
24+ AllIdsArgs, FileList, IdPage, ListFilesArgs, MAX_LISTED_FILES, MAX_READ_BLOBS, RawBlob, RawBlobsArgs, RawFile, RawFileArgs, RefsArgs,
25+ RepoRefs,
26+};
27+use g1t_kit::now_ms;
28+use serde_json::{Value, json};
29+use worker::{Context, Headers, Method, Request, Response, Result, Url};
30+
31+use crate::access::{self, Action};
32+use crate::composer::{self, Origin, Version};
33+use crate::db::{NewVersion, PackageRow};
34+use crate::digest::Digest;
35+use crate::oci::{Credentials, origin};
36+use crate::store::BlobStore;
37+use crate::{Caller, Packages, TargetOf};
38+
39+const COMPOSER: &str = "composer";
40+/// The biggest `composer.json` or README read.
41+const MAX_FILE_BYTES: u32 = 1024 * 1024;
42+/// The most branches and tags a repository's package lists.
43+const MAX_BRANCHES: usize = 50;
44+const MAX_TAGS: usize = 300;
45+/// The most a zip may hold before it is made, in all and per file.
46+const MAX_ARCHIVE_BYTES: u64 = 64 * 1024 * 1024;
47+const MAX_ARCHIVED_FILE: u32 = 32 * 1024 * 1024;
48+/// Repositories the backfill reads each hour.
49+const BACKFILL_PAGE: u32 = 25;
50+const README_NAMES: [&str; 4] = ["README.md", "readme.md", "README.markdown", "README"];
51+
52+fn error(status: u16, message: impl Into<String>) -> Result<Response> {
53+ let mut response = Response::from_json(&json!({ "status": "error", "message": message.into() }))?.with_status(status);
54+ if status == 401 {
55+ response.headers_mut().set("www-authenticate", "Basic realm=\"g1t\"")?;
56+ }
57+ Ok(response)
58+}
59+
60+/// One of the registry's endpoints, under `/-/composer/<workspace>/`.
61+#[derive(Clone, Debug, PartialEq, Eq)]
62+pub enum ComposerRoute {
63+ Root { workspace: String },
64+ /// `p2/<vendor>/<name>.json`, or `~dev.json` for the branches.
65+ Metadata { workspace: String, name: String, dev: bool },
66+ Dist { workspace: String, name: String, commit: String },
67+ Downloads { workspace: String },
68+}
69+
70+pub fn route(path: &str) -> Option<ComposerRoute> {
71+ let rest = path.strip_prefix("/-/composer/")?;
72+ let (workspace, rest) = rest.split_once('/')?;
73+ let workspace = workspace.to_ascii_lowercase();
74+ if rest == "packages.json" || rest.is_empty() {
75+ return Some(ComposerRoute::Root { workspace });
76+ }
77+ if rest == "downloads" {
78+ return Some(ComposerRoute::Downloads { workspace });
79+ }
80+ if let Some(file) = rest.strip_prefix("p2/") {
81+ let (name, dev) = match file.strip_suffix("~dev.json") {
82+ Some(name) => (name, true),
83+ None => (file.strip_suffix(".json")?, false),
84+ };
85+ return composer::valid_name(name).then(|| ComposerRoute::Metadata { workspace, name: name.to_owned(), dev });
86+ }
87+ let file = rest.strip_prefix("dist/")?;
88+ let (name, zip) = file.rsplit_once('/')?;
89+ let commit = zip.strip_suffix(".zip")?;
90+ (composer::valid_name(name) && commit.len() == 40 && commit.bytes().all(|b| b.is_ascii_hexdigit())).then(|| ComposerRoute::Dist {
91+ workspace,
92+ name: name.to_owned(),
93+ commit: commit.to_ascii_lowercase(),
94+ })
95+}
96+
97+/// What a version keeps of its ref, to make its entry from on each read.
98+fn stored_metadata(composer_json: &Value, version: &Version, git_ref: &str, default_branch: bool) -> Value {
99+ json!({
100+ "composer": composer_json,
101+ "version_normalized": version.normalized,
102+ "ref": git_ref,
103+ "default_branch": default_branch,
104+ })
105+}
106+
107+/// The versions a repository's refs make: `(version, commit, ref, default)`.
108+fn wanted_versions(refs: &[g1t_contracts::repos::GitRefEntry], default_branch: &str) -> Vec<(Version, String, String, bool)> {
109+ let mut branches = Vec::new();
110+ let mut tags = Vec::new();
111+ for entry in refs {
112+ if let Some(branch) = entry.name.strip_prefix("refs/heads/") {
113+ branches.push((composer::branch_version(branch), entry.commit.clone(), entry.name.clone(), branch == default_branch));
114+ } else if let Some(tag) = entry.name.strip_prefix("refs/tags/")
115+ && let Some(version) = composer::tag_version(tag)
116+ {
117+ tags.push((version, entry.commit.clone(), entry.name.clone(), false));
118+ }
119+ }
120+ // The default branch first, then the newest tags.
121+ branches.sort_by_key(|(_, _, _, default)| !*default);
122+ branches.truncate(MAX_BRANCHES);
123+ tags.sort_by(|a, b| b.0.normalized.cmp(&a.0.normalized));
124+ tags.truncate(MAX_TAGS);
125+ let mut seen = HashSet::new();
126+ branches.into_iter().chain(tags).filter(|(v, ..)| seen.insert(v.version.clone())).collect()
127+}
128+
129+impl Packages {
130+ pub async fn composer(&self, request: Request, ctx: &Context) -> Result<Response> {
131+ let url = request.url()?;
132+ let Some(route) = route(url.path()) else {
133+ return error(404, "There is nothing at this address.");
134+ };
135+ match self.composer_route(request, &url, route, ctx).await {
136+ Ok(response) => Ok(response),
137+ Err(problem) => {
138+ worker::console_error!("packages: composer {}: {problem}", url.path());
139+ error(500, "Something went wrong on our side. Try again in a moment.")
140+ }
141+ }
142+ }
143+
144+ async fn composer_route(&self, mut request: Request, url: &Url, route: ComposerRoute, ctx: &Context) -> Result<Response> {
145+ let credentials = self.credentials(&request).await?;
146+ if matches!(request.method(), Method::Get | Method::Head)
147+ && let Some(refused) = self.limited(&request, &credentials, "http-basic credentials").await?
148+ {
149+ return Ok(refused);
150+ }
151+ let viewer = match credentials {
152+ Credentials::Viewer(viewer) => viewer,
153+ Credentials::None => None,
154+ Credentials::Token(_) | Credentials::Bad => {
155+ return error(401, "The username or token is not right. Use a g1t access token: composer config --auth http-basic.g1t.sh <you> <token>");
156+ }
157+ };
158+ match route {
159+ ComposerRoute::Root { workspace } => self.composer_root(&workspace, viewer.as_ref()).await,
160+ ComposerRoute::Metadata { workspace, name, dev } => self.composer_metadata(url, &workspace, &name, dev, viewer.as_ref()).await,
161+ ComposerRoute::Dist { workspace, name, commit } => self.composer_dist(&workspace, &name, &commit, viewer.as_ref(), ctx).await,
162+ ComposerRoute::Downloads { workspace } => {
163+ let body: Value = request.json().await.unwrap_or_default();
164+ let names: HashSet<&str> = body["downloads"]
165+ .as_array()
166+ .map(|list| list.iter().filter_map(|d| d["name"].as_str()).collect())
167+ .unwrap_or_default();
168+ for name in names.into_iter().take(50) {
169+ if let Some(package) = self.composer_package(&workspace, name).await? {
170+ self.count_download(&package.id, ctx);
171+ }
172+ }
173+ Ok(Response::empty()?.with_status(204))
174+ }
175+ }
176+ }
177+
178+ async fn composer_package(&self, workspace: &str, name: &str) -> Result<Option<PackageRow>> {
179+ Ok(self.db.package(workspace, COMPOSER, name).await?.filter(|p| !p.hidden()))
180+ }
181+
182+ /// The answer when `viewer` may not pull `package`, if they may not.
183+ fn composer_check(&self, viewer: Option<&User>, package: &PackageRow) -> Option<Result<Response>> {
184+ let target = TargetOf::package(package);
185+ if access::decide(viewer, &target.view(), Action::Pull).allowed {
186+ return None;
187+ }
188+ Some(if viewer.is_none() {
189+ error(401, "Sign in to install this package: composer config --auth http-basic.g1t.sh <you> <g1t token>")
190+ } else {
191+ error(404, "Not found: no such package, or you cannot see it.")
192+ })
193+ }
194+
195+ async fn composer_root(&self, workspace: &str, viewer: Option<&User>) -> Result<Response> {
196+ let rows = self.db.list(workspace, Some(COMPOSER), None, None, 1000).await?;
197+ let available: Vec<String> = rows
198+ .iter()
199+ .filter(|row| access::decide(viewer, &TargetOf::package(&row.package).view(), Action::Pull).allowed)
200+ .map(|row| row.package.name.clone())
201+ .collect();
202+ Response::from_json(&composer::root(workspace, &available))
203+ }
204+
205+ async fn composer_metadata(&self, url: &Url, workspace: &str, name: &str, dev: bool, viewer: Option<&User>) -> Result<Response> {
206+ let Some(package) = self.composer_package(workspace, name).await? else {
207+ return error(404, format!("There is no package {name} in {workspace}."));
208+ };
209+ if let Some(refusal) = self.composer_check(viewer, &package) {
210+ return refusal;
211+ }
212+ let base = origin(url);
213+ let repo = package.repo_name.clone().unwrap_or_default();
214+ let git_url = format!("{base}/{}/{repo}.git", package.workspace);
215+ let mut entries = Vec::new();
216+ for row in self.db.versions(&package.id, 1000).await? {
217+ let meta = row.meta();
218+ let version = Version {
219+ version: row.version.clone(),
220+ normalized: meta["version_normalized"].as_str().unwrap_or(&row.version).to_owned(),
221+ };
222+ if version.is_dev() != dev {
223+ continue;
224+ }
225+ let dist_url = format!("{base}/-/composer/{}/dist/{name}/{}.zip", package.workspace, row.digest);
226+ let origin = Origin {
227+ git_url: &git_url,
228+ dist_url: &dist_url,
229+ commit: &row.digest,
230+ default_branch: meta["default_branch"].as_bool().unwrap_or(false),
231+ };
232+ entries.push(composer::version_entry(&meta["composer"], name, &version, &origin));
233+ }
234+ let mut response = Response::from_json(&composer::p2(name, &entries))?;
235+ response.headers_mut().set("last-modified", &package.updated_at)?;
236+ Ok(response)
237+ }
238+
239+ async fn composer_dist(&self, workspace: &str, name: &str, commit: &str, viewer: Option<&User>, ctx: &Context) -> Result<Response> {
240+ let Some(package) = self.composer_package(workspace, name).await? else {
241+ return error(404, format!("There is no package {name} in {workspace}."));
242+ };
243+ if let Some(refusal) = self.composer_check(viewer, &package) {
244+ return refusal;
245+ }
246+ // Only the commits of its versions: a zip is never made of any
247+ // other commit of the repository.
248+ if self.db.version_by_digest(&package.id, commit).await?.is_none() {
249+ return error(404, format!("{commit} is not a version of {name}."));
250+ }
251+ let blob = match self.db.dist_for_commit(&package.id, commit).await? {
252+ Some(blob) => blob,
253+ None => match self.build_dist(&package, commit).await? {
254+ Ok(blob) => blob,
255+ Err(refusal) => return refusal,
256+ },
257+ };
258+ let Some(got) = self.store.get(&blob.object_key, None).await? else {
259+ return error(404, "The archive is missing. Try again.");
260+ };
261+ self.count_download(&package.id, ctx);
262+ let headers = Headers::new();
263+ headers.set("content-type", "application/zip")?;
264+ headers.set("content-length", &blob.size.to_string())?;
265+ headers.set("cache-control", "max-age=31536000")?;
266+ Ok(Response::from_body(got.body)?.with_headers(headers))
267+ }
268+
269+ /// Makes the zip of a commit: its files but those `.gitattributes`
270+ /// marks `export-ignore`, as `git archive` would leave them out.
271+ async fn build_dist(&self, package: &PackageRow, commit: &str) -> Result<std::result::Result<crate::db::BlobRow, Result<Response>>> {
272+ let Some(repo_id) = package.repo_id.clone() else {
273+ return Ok(Err(error(404, "This package has no repository.")));
274+ };
275+ let listed: FileList = g1t_kit::call(
276+ &self.repos,
277+ "list_files",
278+ &ListFilesArgs { repo_id: repo_id.clone(), git_ref: Some(commit.to_owned()), skip_dirs: Vec::new(), limit: MAX_LISTED_FILES },
279+ )
280+ .await?;
281+ if listed.truncated {
282+ return Ok(Err(error(507, format!("The commit has more than {MAX_LISTED_FILES} files, too many for an archive. Install from source: composer install --prefer-source"))));
283+ }
284+ let attributes = self.repo_file(&repo_id, commit, ".gitattributes").await?;
285+ let ignores = attributes.map(|text| composer::export_ignores(&String::from_utf8_lossy(&text))).unwrap_or_default();
286+ let files: Vec<(String, String)> = listed
287+ .files
288+ .into_iter()
289+ .filter_map(|file| Some((file.path, file.hash?)))
290+ .filter(|(path, _)| !composer::ignored(&ignores, path))
291+ .collect();
292+ let mut bytes_of: HashMap<String, Vec<u8>> = HashMap::new();
293+ let unique: Vec<String> = files.iter().map(|(_, hash)| hash.clone()).collect::<HashSet<_>>().into_iter().collect();
294+ let mut total = 0u64;
295+ for chunk in unique.chunks(MAX_READ_BLOBS) {
296+ let read: Vec<RawBlob> = g1t_kit::call(
297+ &self.repos,
298+ "raw_blobs",
299+ &RawBlobsArgs { repo_id: repo_id.clone(), hashes: chunk.to_vec(), max_bytes: MAX_ARCHIVED_FILE },
300+ )
301+ .await?;
302+ for blob in read {
303+ total += blob.size;
304+ if total > MAX_ARCHIVE_BYTES || (blob.data.is_none() && blob.size > 0) {
305+ return Ok(Err(error(
306+ 507,
307+ format!("The commit is too large for an archive (over {} MB). Install from source: composer install --prefer-source", MAX_ARCHIVE_BYTES / 1_048_576),
308+ )));
309+ }
310+ let data = blob.data.as_deref().map(|d| STANDARD.decode(d).unwrap_or_default()).unwrap_or_default();
311+ bytes_of.insert(blob.hash, data);
312+ }
313+ }
314+ let entries: Vec<(String, Vec<u8>)> = files
315+ .into_iter()
316+ .map(|(path, hash)| {
317+ let data = bytes_of.get(&hash).cloned().unwrap_or_default();
318+ (path, data)
319+ })
320+ .collect();
321+ let zip = composer::zip(&entries);
322+ let digest = Digest::of(&zip);
323+ let size = zip.len() as u64;
324+ let now = now_ms();
325+ if self.db.blob(&digest).await?.is_none() {
326+ self.store.put(&digest.object_key(), zip).await?;
327+ }
328+ self.db.keep_blob(&package.id, &digest, size, Some("application/zip"), &digest.object_key(), now).await?;
329+ self.db.add_dist(&package.id, commit, &digest, size).await?;
330+ self.db.measure(&package.workspace).await?;
331+ Ok(Ok(crate::db::BlobRow { digest: digest.to_string(), size, media_type: Some("application/zip".into()), object_key: digest.object_key() }))
332+ }
333+
334+ /// A file of a repository at a commit, if it is there and not large.
335+ async fn repo_file(&self, repo_id: &str, git_ref: &str, path: &str) -> Result<Option<Vec<u8>>> {
336+ let file: Option<RawFile> = g1t_kit::call(
337+ &self.repos,
338+ "raw_file",
339+ &RawFileArgs { repo_id: repo_id.to_owned(), git_ref: git_ref.to_owned(), path: path.to_owned(), max_bytes: MAX_FILE_BYTES },
340+ )
341+ .await?;
342+ Ok(file.and_then(|file| STANDARD.decode(file.data).ok()))
343+ }
344+
345+ /// Reads a repository's Composer package again from its refs: makes it
346+ /// when its default branch gained a `composer.json`, records new and
347+ /// moved versions, lets go of deleted ones, and deletes the package
348+ /// when the repository stopped being one. Says whether it is one.
349+ pub(crate) async fn sync_composer(&self, repo_id: &str) -> Result<bool> {
350+ let found: Option<RepoRefs> = g1t_kit::call(&self.repos, "refs", &RefsArgs { repo_id: repo_id.to_owned() }).await?;
351+ let existing = self.db.package_for_repo(repo_id, COMPOSER).await?;
352+ let Some(RepoRefs { repo, refs }) = found else {
353+ if let Some(package) = existing {
354+ self.drop_composer(&package).await?;
355+ }
356+ return Ok(false);
357+ };
358+ let workspace = repo.namespace.to_lowercase();
359+ if self.db.workspace_hidden(&workspace).await? {
360+ return Ok(false);
361+ }
362+ let default = refs.iter().find(|r| r.name == format!("refs/heads/{}", repo.default_branch)).map(|r| r.commit.clone());
363+ let manifest = match &default {
364+ Some(commit) => self.composer_json(repo_id, commit).await?,
365+ None => None,
366+ };
367+ let Some((name, root_manifest)) = manifest else {
368+ if let Some(package) = existing {
369+ self.drop_composer(&package).await?;
370+ }
371+ return Ok(false);
372+ };
373+ // A repository moved to another workspace takes its package along.
374+ let existing = match existing {
375+ Some(package) if package.workspace != workspace => {
376+ self.drop_composer(&package).await?;
377+ None
378+ }
379+ other => other,
380+ };
381+ let now = now_ms();
382+ let package = match existing {
383+ Some(package) if package.name == name => package,
384+ Some(package) => {
385+ if self.db.package(&workspace, COMPOSER, &name).await?.is_some() {
386+ worker::console_error!("packages: {workspace}/{} names {name}, which another repository has", repo.name);
387+ package
388+ } else {
389+ self.db.rename_package(&package.id, &name, now).await?;
390+ PackageRow { name: name.clone(), ..package }
391+ }
392+ }
393+ None => {
394+ if let Some(other) = self.db.package(&workspace, COMPOSER, &name).await?
395+ && other.repo_id.as_deref() != Some(repo_id)
396+ {
397+ worker::console_error!("packages: {workspace}/{} names {name}, which another repository has", repo.name);
398+ return Ok(false);
399+ }
400+ self.db
401+ .create_package(&new_id("pkg", now), &workspace, COMPOSER, &name, Some((&repo.id, &repo.name, repo.is_private)), "g1t", now)
402+ .await?
403+ }
404+ };
405+
406+ let caller = Caller { actor: None };
407+ let wanted = wanted_versions(&refs, &repo.default_branch);
408+ let stored = self.db.versions(&package.id, 1000).await?;
409+ let mut manifests: HashMap<String, Option<Value>> = HashMap::new();
410+ if let Some(commit) = &default {
411+ manifests.insert(commit.clone(), Some(root_manifest.clone()));
412+ }
413+ let mut changed = false;
414+ for (version, commit, git_ref, is_default) in &wanted {
415+ let current = stored.iter().find(|row| row.version == version.version);
416+ if let Some(row) = current
417+ && row.digest == *commit
418+ && row.meta()["default_branch"].as_bool().unwrap_or(false) == *is_default
419+ {
420+ continue;
421+ }
422+ if !manifests.contains_key(commit) {
423+ let read = self.composer_json(repo_id, commit).await?.map(|(_, json)| json);
424+ manifests.insert(commit.clone(), read);
425+ }
426+ // A ref without a composer.json of its own is not a version.
427+ let Some(Some(json)) = manifests.get(commit) else { continue };
428+ self.db
429+ .replace_version(
430+ &NewVersion {
431+ id: new_id("ver", now),
432+ package_id: package.id.clone(),
433+ version: version.version.clone(),
434+ digest: commit.clone(),
435+ size: 0,
436+ metadata: stored_metadata(json, version, git_ref, *is_default).to_string(),
437+ subject: None,
438+ published_by: None,
439+ files: Vec::new(),
440+ },
441+ now,
442+ )
443+ .await?;
444+ changed = true;
445+ if current.is_none() {
446+ let event = PackageEvent { version: Some(version.version.clone()), digest: Some(commit.clone()), ..self.event_of(&package) };
447+ self.announce("package.published", &package, event, &caller).await;
448+ }
449+ }
450+ let kept: HashSet<&str> = wanted.iter().map(|(v, ..)| v.version.as_str()).collect();
451+ for row in stored.iter().filter(|row| !kept.contains(row.version.as_str())) {
452+ self.db.delete_version(&row.id).await?;
453+ let event = PackageEvent { version: Some(row.version.clone()), digest: Some(row.digest.clone()), ..self.event_of(&package) };
454+ self.announce("package.version_deleted", &package, event, &caller).await;
455+ changed = true;
456+ }
457+ if let Some(commit) = &default {
458+ self.composer_readme(&package, repo_id, commit, &root_manifest, now).await?;
459+ }
460+ if changed {
461+ self.db.measure(&workspace).await?;
462+ }
463+ Ok(true)
464+ }
465+
466+ /// The package's README and description, from the default branch.
467+ async fn composer_readme(&self, package: &PackageRow, repo_id: &str, commit: &str, manifest: &Value, now: u64) -> Result<()> {
468+ let mut readme = None;
469+ for name in README_NAMES {
470+ if let Some(bytes) = self.repo_file(repo_id, commit, name).await? {
471+ readme = Some(bytes);
472+ break;
473+ }
474+ }
475+ let digest = match readme.filter(|b| !b.is_empty()) {
476+ Some(bytes) => {
477+ let digest = Digest::of(&bytes);
478+ if self.db.blob(&digest).await?.is_none() {
479+ self.store.put(&digest.object_key(), bytes.clone()).await?;
480+ }
481+ self.db.keep_blob(&package.id, &digest, bytes.len() as u64, Some("text/markdown"), &digest.object_key(), now).await?;
482+ Some(digest.to_string())
483+ }
484+ None => None,
485+ };
486+ self.db.set_readme(&package.id, digest.as_deref(), manifest["description"].as_str(), now).await
487+ }
488+
489+ /// A commit's `composer.json`, when it has one naming a valid package.
490+ async fn composer_json(&self, repo_id: &str, commit: &str) -> Result<Option<(String, Value)>> {
491+ let Some(bytes) = self.repo_file(repo_id, commit, "composer.json").await? else {
492+ return Ok(None);
493+ };
494+ let Ok(json) = serde_json::from_slice::<Value>(&bytes) else {
495+ return Ok(None);
496+ };
497+ let Some(name) = json["name"].as_str().map(str::to_lowercase).filter(|n| composer::valid_name(n)) else {
498+ return Ok(None);
499+ };
500+ Ok(Some((name, json)))
501+ }
502+
503+ async fn drop_composer(&self, package: &PackageRow) -> Result<()> {
504+ self.db.delete_package(&package.id).await?;
505+ self.db.measure(&package.workspace).await?;
506+ self.announce("package.deleted", package, self.event_of(package), &Caller { actor: None }).await;
507+ Ok(())
508+ }
509+
510+ /// Deletes the Composer package built from a deleted repository.
511+ pub(crate) async fn composer_repo_gone(&self, repo_id: &str) -> Result<()> {
512+ if let Some(package) = self.db.package_for_repo(repo_id, COMPOSER).await? {
513+ self.drop_composer(&package).await?;
514+ }
515+ Ok(())
516+ }
517+
518+ /// Reads a page of repositories the backfill has not yet, until it has
519+ /// read them all once. Says how many were packages.
520+ pub(crate) async fn composer_backfill(&self) -> Result<u32> {
521+ let (after, finished) = self.db.backfill().await?;
522+ if finished {
523+ return Ok(0);
524+ }
525+ let page: IdPage = g1t_kit::call(&self.repos, "all_ids", &AllIdsArgs { after: after.clone(), limit: BACKFILL_PAGE }).await?;
526+ let mut found = 0;
527+ for id in &page.ids {
528+ match self.sync_composer(id).await {
529+ Ok(true) => found += 1,
530+ Ok(false) => {}
531+ Err(error) => worker::console_error!("packages: composer backfill of {id}: {error}"),
532+ }
533+ }
534+ let last = page.ids.last().cloned().or(after);
535+ self.db.set_backfill(last.as_deref(), page.next.is_none(), now_ms()).await?;
536+ Ok(found)
537+ }
538+}
539+
540+#[cfg(test)]
541+mod tests {
542+ use super::*;
543+ use g1t_contracts::repos::GitRefEntry;
544+
545+ #[test]
546+ fn every_endpoint_is_routed() {
547+ let commit = "a".repeat(40);
548+ assert_eq!(route("/-/composer/acme/packages.json"), Some(ComposerRoute::Root { workspace: "acme".into() }));
549+ assert_eq!(route("/-/composer/acme/"), Some(ComposerRoute::Root { workspace: "acme".into() }));
550+ assert_eq!(
551+ route("/-/composer/acme/p2/acme/lib.json"),
552+ Some(ComposerRoute::Metadata { workspace: "acme".into(), name: "acme/lib".into(), dev: false })
553+ );
554+ assert_eq!(
555+ route("/-/composer/acme/p2/acme/lib~dev.json"),
556+ Some(ComposerRoute::Metadata { workspace: "acme".into(), name: "acme/lib".into(), dev: true })
557+ );
558+ assert_eq!(
559+ route(&format!("/-/composer/acme/dist/acme/lib/{commit}.zip")),
560+ Some(ComposerRoute::Dist { workspace: "acme".into(), name: "acme/lib".into(), commit: commit.clone() })
561+ );
562+ assert_eq!(route("/-/composer/acme/downloads"), Some(ComposerRoute::Downloads { workspace: "acme".into() }));
563+ assert_eq!(route("/-/composer/acme/p2/Acme/lib.json"), None);
564+ assert_eq!(route("/-/composer/acme/dist/acme/lib/short.zip"), None);
565+ assert_eq!(route("/-/composer/acme"), None);
566+ }
567+
568+ #[test]
569+ fn a_repositorys_refs_make_its_versions() {
570+ let entry = |name: &str, commit: &str| GitRefEntry { name: name.into(), commit: commit.into() };
571+ let refs = [
572+ entry("refs/heads/feature", "f"),
573+ entry("refs/heads/main", "m"),
574+ entry("refs/tags/v1.0.0", "a"),
575+ entry("refs/tags/v1.1.0", "b"),
576+ entry("refs/tags/nightly", "n"),
577+ ];
578+ let wanted = wanted_versions(&refs, "main");
579+ let names: Vec<(&str, &str, bool)> = wanted.iter().map(|(v, c, _, d)| (v.version.as_str(), c.as_str(), *d)).collect();
580+ assert_eq!(
581+ names,
582+ [("dev-main", "m", true), ("dev-feature", "f", false), ("v1.1.0", "b", false), ("v1.0.0", "a", false)],
583+ "the default branch first, newest tags next, tags that are not versions left out"
584+ );
585+ }
586+}
+111−0
584584 .results()
585585 }
586586
587+ /// The package of an ecosystem built from a repository (Composer's).
588+ pub async fn package_for_repo(&self, repo_id: &str, ecosystem: &str) -> Result<Option<PackageRow>> {
589+ self.prepare(
590+ &format!("SELECT {PACKAGE_COLUMNS} FROM packages WHERE repo_id = ? AND ecosystem = ? LIMIT 1"),
591+ &[text(repo_id), text(ecosystem)],
592+ )?
593+ .first(None)
594+ .await
595+ }
596+
597+ pub async fn rename_package(&self, package_id: &str, name: &str, now_ms: u64) -> Result<()> {
598+ self.prepare(
599+ "UPDATE packages SET name = ?, updated_at = ? WHERE id = ?",
600+ &[text(name), text(&rfc3339(now_ms)), text(package_id)],
601+ )?
602+ .run()
603+ .await?;
604+ Ok(())
605+ }
606+
607+ /// Records a version, in place of one of the same version string
608+ /// (a tag or branch that moved): its files and tags go with the old one.
609+ pub async fn replace_version(&self, version: &NewVersion, now_ms: u64) -> Result<()> {
610+ let now = rfc3339(now_ms);
611+ let old = [text(&version.package_id), text(&version.version)];
612+ let old_ids = "SELECT id FROM versions WHERE package_id = ? AND version = ?";
613+ let mut batch = vec![
614+ self.prepare(&format!("DELETE FROM tags WHERE version_id IN ({old_ids})"), &old)?,
615+ self.prepare(&format!("DELETE FROM version_files WHERE version_id IN ({old_ids})"), &old)?,
616+ self.prepare("DELETE FROM versions WHERE package_id = ? AND version = ?", &old)?,
617+ self.prepare(
618+ "INSERT INTO versions (id, package_id, version, digest, size, metadata, subject, published_by, published_at)
619+ VALUES (?, ?, ?, ?, ?, ?, NULL, ?, ?)",
620+ &[
621+ text(&version.id),
622+ text(&version.package_id),
623+ text(&version.version),
624+ text(&version.digest),
625+ num(version.size),
626+ text(&version.metadata),
627+ opt(version.published_by.as_deref()),
628+ text(&now),
629+ ],
630+ )?,
631+ ];
632+ for file in &version.files {
633+ batch.push(self.prepare(
634+ "INSERT OR IGNORE INTO version_files (version_id, name, digest, size, media_type) VALUES (?, ?, ?, ?, ?)",
635+ &[text(&version.id), text(&file.name), text(&file.digest), num(file.size), opt(file.media_type.as_deref())],
636+ )?);
637+ }
638+ batch.push(self.prepare("UPDATE packages SET updated_at = ? WHERE id = ?", &[text(&now), text(&version.package_id)])?);
639+ self.db.batch(batch).await?;
640+ Ok(())
641+ }
642+
643+ /// The zip already made of a commit of the package, if one was.
644+ pub async fn dist_for_commit(&self, package_id: &str, commit: &str) -> Result<Option<BlobRow>> {
645+ self.prepare(
646+ "SELECT b.digest, b.size, b.media_type, b.object_key FROM versions v
647+ JOIN version_files vf ON vf.version_id = v.id AND vf.name = 'dist'
648+ JOIN blobs b ON b.digest = vf.digest
649+ WHERE v.package_id = ? AND v.digest = ? LIMIT 1",
650+ &[text(package_id), text(commit)],
651+ )?
652+ .first(None)
653+ .await
654+ }
655+
656+ /// Records the zip of a commit as a file of every version at it.
657+ pub async fn add_dist(&self, package_id: &str, commit: &str, digest: &Digest, size: u64) -> Result<()> {
658+ let at = [text(package_id), text(commit)];
659+ self.db
660+ .batch(vec![
661+ self.prepare(
662+ "INSERT OR IGNORE INTO version_files (version_id, name, digest, size, media_type)
663+ SELECT id, 'dist', ?, ?, 'application/zip' FROM versions WHERE package_id = ? AND digest = ?",
664+ &[text(digest.as_str()), num(size), text(package_id), text(commit)],
665+ )?,
666+ self.prepare("UPDATE versions SET size = ? WHERE package_id = ? AND digest = ?", &[num(size), at[0].clone(), at[1].clone()])?,
667+ ])
668+ .await?;
669+ Ok(())
670+ }
671+
672+ /// Where the Composer backfill is: the last repository done, and
673+ /// whether it went through them all.
674+ pub async fn backfill(&self) -> Result<(Option<String>, bool)> {
675+ #[derive(Deserialize)]
676+ struct Row {
677+ after: Option<String>,
678+ finished_at: Option<String>,
679+ }
680+ let row: Option<Row> = self
681+ .prepare("SELECT after, finished_at FROM composer_backfill WHERE key = 'repos'", &[])?
682+ .first(None)
683+ .await?;
684+ Ok(row.map_or((None, false), |row| (row.after, row.finished_at.is_some())))
685+ }
686+
687+ pub async fn set_backfill(&self, after: Option<&str>, finished: bool, now_ms: u64) -> Result<()> {
688+ self.prepare(
689+ "INSERT INTO composer_backfill (key, after, finished_at) VALUES ('repos', ?, ?)
690+ ON CONFLICT (key) DO UPDATE SET after = excluded.after, finished_at = excluded.finished_at",
691+ &[opt(after), if finished { text(&rfc3339(now_ms)) } else { JsValue::NULL }],
692+ )?
693+ .run()
694+ .await?;
695+ Ok(())
696+ }
697+
587698 /// Points `tag` at a version, made or moved.
588699 pub async fn set_tag(&self, package_id: &str, tag: &str, version_id: &str, now_ms: u64) -> Result<()> {
589700 self.prepare(
+43−0
88 //! `BlobStore` port (store/), metadata in D1 (db.rs).
99
1010 mod access;
11+mod composer;
12+mod composer_http;
1113 mod db;
1214 mod digest;
1315 mod limits;
345347 name: p.name.clone(),
346348 address: match p.ecosystem.as_str() {
347349 "npm" => format!("{}/-/npm/@{}/{}", self.host, p.workspace, p.name),
350+ "composer" => format!("{}/-/composer/{}/{}", self.host, p.workspace, p.name),
348351 _ => format!("{}/{}/{}", self.host, p.workspace, p.name),
349352 },
350353 visibility: Visibility::parse(&p.visibility),
565568 /// Follows what happens elsewhere: workspaces renamed and deleted,
566569 /// repositories that change visibility, are renamed, move or go.
567570 async fn on_event(&self, env: &Env, event: &Event) -> Result<()> {
571+ // Composer packages are read from repositories: again when one is
572+ // pushed to (its default branch may have gained a composer.json),
573+ // restored, renamed or moved; gone when it is deleted. A failure is
574+ // logged, not retried with the batch: the next push reads it again.
575+ let repo_id = event.repo_id.clone().or_else(|| event.data["repoId"].as_str().map(str::to_owned));
576+ if let Some(repo_id) = repo_id.as_deref() {
577+ let synced = match event.kind.as_str() {
578+ "git.push" => {
579+ let known = self.db.package_for_repo(repo_id, "composer").await?.is_some();
580+ if known || event.data["defaultBranch"].as_bool() == Some(true) {
581+ Some(self.sync_composer(repo_id).await.map(|_| ()))
582+ } else {
583+ None
584+ }
585+ }
586+ "repo.deleted" => Some(self.composer_repo_gone(repo_id).await),
587+ "repo.restored" | "repo.renamed" | "repo.transferred" => Some(self.sync_composer(repo_id).await.map(|_| ())),
588+ _ => None,
589+ };
590+ if let Some(Err(error)) = synced {
591+ worker::console_error!("packages: composer {} for {repo_id}: {error}", event.kind);
592+ }
593+ if event.kind == "git.push" || event.kind == "repo.deleted" || event.kind == "repo.restored" {
594+ return Ok(());
595+ }
596+ }
568597 let db = &self.db.db;
569598 let protected = g1t_contracts::identity::protected_names(Some(&store::var(env, "PROTECTED_WORKSPACES")));
570599 match workspace_mark(event, &protected, &g1t_contracts::time::rfc3339(now_ms())) {
637666 if request.path().starts_with("/-/npm/") || request.path() == "/-/npm" {
638667 return packages.npm(request, &ctx).await;
639668 }
669+ if request.path().starts_with("/-/composer/") {
670+ return packages.composer(request, &ctx).await;
671+ }
640672 return packages.registry(request, &ctx).await;
641673 };
642674 let body: serde_json::Value = request.json().await?;
648680 "set_package" => reply(&packages.set_package(args(body)?).await?),
649681 // For billing: what a workspace's packages hold.
650682 "storage_all" => reply(&packages.db.storage_all().await?),
683+ // Read a repository's Composer package again now, as a push would.
684+ "sync_composer" => {
685+ let a: SyncComposerArgs = args(body)?;
686+ reply(&packages.sync_composer(&a.repo_id).await?)
687+ }
651688 "storage" => {
652689 let a: StorageArgs = args(body)?;
653690 let (public_bytes, private_bytes) = packages.db.storage(&a.workspace.to_lowercase()).await?;
673710 Ok((uploads, blobs)) => worker::console_log!("packages: let go of {uploads} uploads and {blobs} blobs"),
674711 Err(error) => worker::console_error!("packages: the sweep failed: {error}"),
675712 }
713+ // Repositories that had a composer.json before the registry did.
714+ match packages.composer_backfill().await {
715+ Ok(0) => {}
716+ Ok(found) => worker::console_log!("packages: found {found} Composer packages"),
717+ Err(error) => worker::console_error!("packages: the Composer backfill failed: {error}"),
718+ }
676719 }
677720
678721 #[event(queue)]
+1−0
1919 futures-util = { version = "0.3", default-features = false, features = ["alloc"] }
2020 similar = "2"
2121 miniz_oxide = "0.9"
22+base64 = "0.22"
2223
2324 # wasm-opt at -O1: about the same gzipped size as -O in a tenth of the
2425 # time (docs/DEPLOYING.md, "Build speed").
+51−0
382382 listing::list(&git, a.base.as_deref(), &a.head, &a.skip_dirs, a.limit).await
383383 }
384384
385+ /// Branches and tags with their commits, for g1t's own services.
386+ async fn refs_of(&self, a: RefsArgs) -> Result<Option<RepoRefs>> {
387+ let Some(repo) = self.registry.by_id(&a.repo_id).await?.filter(|repo| repo.fork_of.is_none()) else {
388+ return Ok(None);
389+ };
390+ let git = self.store.open(&store_key(&repo)).await?;
391+ let access = git.access(Scope::Read).await?;
392+ let refs = refs::heads_and_tags(refs::all(&access).await?)
393+ .into_iter()
394+ .map(|(name, commit)| GitRefEntry { name, commit })
395+ .collect();
396+ Ok(Some(RepoRefs { repo, refs }))
397+ }
398+
399+ async fn raw_file(&self, a: RawFileArgs) -> Result<Option<RawFile>> {
400+ use base64::Engine;
401+ let Some(git) = self.stored(&a.repo_id).await? else {
402+ return Ok(None);
403+ };
404+ Ok(git
405+ .read_file(&a.git_ref, &a.path)
406+ .await?
407+ .filter(|bytes| bytes.len() <= a.max_bytes as usize)
408+ .map(|bytes| RawFile { size: bytes.len() as u64, data: base64::engine::general_purpose::STANDARD.encode(bytes) }))
409+ }
410+
411+ async fn raw_blobs(&self, a: RawBlobsArgs) -> Result<Vec<RawBlob>> {
412+ use base64::Engine;
413+ let Some(git) = self.stored(&a.repo_id).await? else {
414+ return Ok(Vec::new());
415+ };
416+ let hashes: Vec<&String> = a.hashes.iter().take(MAX_READ_BLOBS).collect();
417+ let mut out = Vec::with_capacity(hashes.len());
418+ // A few at a time, as listing::read does: each is a round trip.
419+ for group in hashes.chunks(8) {
420+ let read = futures_util::future::try_join_all(group.iter().map(|hash| git.read_blob(hash))).await?;
421+ for (hash, bytes) in group.iter().zip(read) {
422+ let size = bytes.as_ref().map_or(0, |bytes| bytes.len() as u64);
423+ let data = bytes
424+ .filter(|bytes| bytes.len() <= a.max_bytes as usize)
425+ .map(|bytes| base64::engine::general_purpose::STANDARD.encode(bytes));
426+ out.push(RawBlob { hash: (*hash).clone(), size, data });
427+ }
428+ }
429+ Ok(out)
430+ }
431+
385432 async fn read_blobs(&self, a: ReadBlobsArgs) -> Result<Vec<BlobText>> {
386433 let Some(git) = self.stored(&a.repo_id).await? else {
387434 return Ok(Vec::new());
19001947 "list_files" => reply(&repos.list_files(args(body)?).await?),
19011948 "changed_files" => reply(&repos.changed_files(args(body)?).await?),
19021949 "read_blobs" => reply(&repos.read_blobs(args(body)?).await?),
1950+ // Services only: what the Composer registry builds packages from.
1951+ "refs" => reply(&repos.refs_of(args(body)?).await?),
1952+ "raw_file" => reply(&repos.raw_file(args(body)?).await?),
1953+ "raw_blobs" => reply(&repos.raw_blobs(args(body)?).await?),
19031954 "visibility" => {
19041955 let a: g1t_contracts::repos::VisibilityArgs = args(body)?;
19051956 reply(&repos.registry.visibility(&a.paths).await?)
+38−0
3535 .collect()
3636 }
3737
38+/// Branches and tags from every ref, an annotated tag's commit taken from
39+/// its peeled line (`refs/tags/v1^{}`) when the advertisement has one.
40+pub fn heads_and_tags(all: Vec<(String, String)>) -> Vec<(String, String)> {
41+ let mut out: Vec<(String, String)> = Vec::new();
42+ for (name, hash) in all {
43+ if let Some(tag) = name.strip_suffix("^{}") {
44+ match out.iter_mut().find(|(seen, _)| seen == tag) {
45+ Some(entry) => entry.1 = hash,
46+ None => out.push((tag.to_owned(), hash)),
47+ }
48+ continue;
49+ }
50+ if name.starts_with(HEADS) || name.starts_with("refs/tags/") {
51+ out.push((name, hash));
52+ }
53+ }
54+ out
55+}
56+
3857 pub async fn branches(access: &GitAccess) -> Result<Vec<Branch>> {
3958 Ok(parse_advertisement(&advertisement(access).await?))
4059 }
100119 }
101120
102121 #[test]
122+ fn annotated_tags_are_peeled_and_other_refs_left_out() {
123+ let all = vec![
124+ ("refs/heads/main".to_owned(), "a".to_owned()),
125+ ("refs/pull/1/head".to_owned(), "b".to_owned()),
126+ ("refs/tags/v1".to_owned(), "tagobject".to_owned()),
127+ ("refs/tags/v1^{}".to_owned(), "c".to_owned()),
128+ ("refs/tags/v2".to_owned(), "d".to_owned()),
129+ ];
130+ assert_eq!(
131+ super::heads_and_tags(all),
132+ vec![
133+ ("refs/heads/main".to_owned(), "a".to_owned()),
134+ ("refs/tags/v1".to_owned(), "c".to_owned()),
135+ ("refs/tags/v2".to_owned(), "d".to_owned()),
136+ ]
137+ );
138+ }
139+
140+ #[test]
103141 fn an_empty_repository_has_no_branches() {
104142 let advertisement = [pkt("# service=git-upload-pack\n"), b"00000000".to_vec()].concat();
105143 assert!(parse_advertisement(&advertisement).is_empty());