g1t/docs/PACKAGES.md

295 lines18,430 bytesCodeBlame

Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.

Packages: the design every phase builds to1# Packages: design
2
3Packages are registries a workspace publishes to and installs from, beside its code: container
4images, npm, Composer, Cargo and Go first, then Maven, NuGet and RubyGems. They live where the
5code does, take the same people and tokens, and are built and published by g1t Actions and
6agents without a second account anywhere.
7
8This is the design every phase builds to. Each ecosystem's own guide (apps/docs) says how to use
9it; this says how it works.
10
Merge branch 'worktree-agent-ac1de8a731938ed81'11## Status
12
13What is built, as of 2026-10-07. Each protocol section below marks the same. "Checked" says what was run
14on that day against `wrangler dev` (`services/packages/dev/`) with the tool itself, beyond unit tests.
15
16| Registry | Built | Not built |
17| --- | --- | --- |
18| Containers | Pull, push (chunked, multipart, monolithic, cross-repository mount), deletes, referrers, token exchange, anonymous pull limits; `g1t push` for layers over the request limit. | |
19| npm | Scoped publish and install, abbreviated packuments, dist-tags, deprecate, unpublish (72 hours, or Admin), `whoami`. | Proxying unscoped packages and other scopes from the public registry. |
20| Composer | Packages from the workspace's repositories: tags and branches as versions, dist zips made and kept by commit, backfill. | A Packagist mirror. |
21| Cargo | Sparse index, `config.json` with `auth-required`, publish, yank and unyank, search; owners are not kept (`cargo owner` answers why). Checked with `cargo`, including a workflow-like publish and build with only environment variables. | |
22| Go | `go get` from git: `?go-get=1` answers with `go-import`. | The module proxy at `/-/go/`. |
23| Maven | Standard layout, releases and SNAPSHOTs, checksums (MD5, SHA-1, SHA-256, SHA-512), generated `maven-metadata.xml` per artifact, SNAPSHOT and plugin group (`mvn <prefix>:<goal>`), Gradle Module Metadata. Checked with `mvn deploy`, `mvn <prefix>:<goal>`, and Gradle `publish` and resolution (releases, SNAPSHOTs, `--refresh-dependencies`). | |
24| NuGet | v3 feed: push, flat container, registration, search, unlist and relist, symbol packages (`.snupkg`) and a symbol server, per-version download counts. Checked with `dotnet nuget push`, `dotnet restore` and `dotnet-symbol`. | |
25| RubyGems | `gem push`, `gem yank`, the compact index (Bundler), the full index (`specs.4.8.gz`, `latest_`, `prerelease_`, `quick/Marshal.4.8`). Checked with `gem push`, `gem install --source`, `gem search` and `gem specification --remote`. | |
26| PyPI | | Everything. |
27
28Across registries: events, webhooks, the audit log, the billing meter and free limits, the
29Packages pages and a project's packages are built. Workflows triggered by `registry_package`,
30packages in site-wide search, and packages in the activity feed are not.
31
Packages: the design every phase builds to32## Principles
33
34- **One service.** `services/packages` (Rust Worker) owns every registry: its own D1 database
35 (`g1t-packages`), its own file store, its own contract in `crates/contracts` /
36 `packages/contracts`. Other services talk to it over RPC and hear from it through events.
37- **Cloudflare in production, anything in a self-hosted install.** Files go through a
38 `BlobStore` port. In production its adapter is R2; self-hosted it is S3-compatible storage
Docs: the self-hosted object store is RustFS39 (RustFS in the compose file) or a local directory. Upload URLs (`presign`) are part of the port:
Packages: the design every phase builds to40 R2 and S3 sign them, the disk adapter answers with a path back through the service.
41 Metadata is D1, which self-hosting already runs (workerd's D1 over SQLite).
42- **Content-addressed.** Every file is stored once by its SHA-256 (`blobs/sha256/<hex>`). A
43 version is a list of files by digest. Pushing a layer two images share stores it once.
44- **The registry speaks each tool's own protocol**, unchanged. `docker`, `npm`, `composer`,
Merge branch 'worktree-agent-a6a121745e81f639f'45 `cargo`, `go`, `mvn` and Gradle, `dotnet` and `gem` and Bundler work with only a login and an
46 address.
Packages: the design every phase builds to47- **Same access as the code.** A package can be linked to a repository and then has its
48 visibility and roles. Unlinked, it is the workspace's: members by the base permission.
49- **Same front door.** Everything is on `g1t.sh`. `apps/web/workers/app.ts` hands registry paths
50 to the `PACKAGES` binding the way it hands git paths to repos.
51
52## Model
53
54| Table | What it holds |
55| --- | --- |
Merge branch 'worktree-agent-a6a121745e81f639f'56| `packages` | `id`, `workspace`, `ecosystem` (`container`, `npm`, `composer`, `cargo`, `go`, `maven`, `nuget`, `rubygems`), `name` (normalized per ecosystem), `repo_id` (linked repository, or null), `visibility` (`public`, `private`; linked packages follow their repository), `description`, `readme_digest`, `created_by`, `created_at`, `updated_at`, `downloads` |
Packages: the design every phase builds to57| `versions` | `id`, `package_id`, `version` (tag, semver or digest), `digest` (the manifest's or the archive's), `size` (sum of its files), `metadata` (JSON the ecosystem needs: npm's packument entry, a crate's index line, composer.json), `published_by`, `published_at`, `yanked`, `deprecated` |
58| `version_files` | `version_id`, `name`, `digest`, `size`, `media_type` |
59| `blobs` | `digest`, `size`, `media_type`, `created_at` |
60| `workspace_blobs` | `workspace`, `digest`, `public` (any public package uses it): what a workspace stores, counted once each, for billing |
61| `uploads` | `id`, `workspace`, `package`, `multipart_id`, `parts` (JSON), `offset`, `hash_state`, `expires_at`: uploads in progress |
62| `tags` | `package_id`, `tag`, `version_id`: container tags and npm dist-tags |
Merge branch 'worktree-agent-a6a121745e81f639f'63| `checksums` | `digest`, `md5`, `sha1`, `sha512`: a file's other checksums, worked out on upload (Maven asks for them) |
Packages: the design every phase builds to64
65Deleting a version removes its rows; a daily sweep deletes blobs no version references, after a
66day's grace (a push in flight may reference a blob before its manifest lands).
67
68## Access
69
70- **Who:** people (session or personal token), workspace tokens, `G1T_TOKEN` in workflows,
71 agents' run tokens. Scopes `packages:read` and `packages:write` (and `packages:delete`), per the
72 MCP and token scope design.
73- **What:** linked package: the repository's roles (Read pulls, Write publishes, Admin deletes and
74 changes settings). Unlinked: workspace members by base permission; owners delete.
75- **Public packages** pull anonymously. Anonymous pulls are rate limited per address.
76- **Workflows** may publish the packages of their own repository, and new ones linked to it, with
77 `G1T_TOKEN`, as code they push does.
78- Every publish and delete is an audit entry and an event.
79
80## Protocols
81
82All under `g1t.sh`. A name always starts with the workspace.
83
84### Containers (OCI Distribution 1.1)
85
Merge branch 'worktree-agent-ac1de8a731938ed81'86Built.
87
Packages: the design every phase builds to88- Image names: `g1t.sh/<workspace>/<name>[:tag]`, `<name>` may contain `/`. Usually the
89 repository's name, and then linked to it.
90- `GET /v2/` answers 401 with `WWW-Authenticate: Bearer realm="https://g1t.sh/v2/token",
91 service="g1t.sh"`. `GET /v2/token` takes Basic auth (any username, a g1t token as the
92 password) or none (anonymous, public pulls) and returns a short-lived signed token naming the
93 scopes granted. `docker login g1t.sh -u <you> --password-stdin` stores it.
94- Pull: `HEAD/GET /v2/<name>/manifests/<ref>`, `HEAD/GET /v2/<name>/blobs/<digest>` (a
95 redirect to a signed R2 URL for large blobs, so downloads never pass through the Worker),
96 `GET /v2/<name>/tags/list`, `GET /v2/<name>/referrers/<digest>`.
97- Push: `POST /v2/<name>/blobs/uploads/` (`?mount=&from=` cross-repository mount within a
98 workspace; `?digest=` monolithic), `PATCH` chunks, `PUT ?digest=` to finish,
99 `PUT /v2/<name>/manifests/<ref>` (image manifests, indexes, OCI artifacts, `subject` for
100 referrers). `DELETE` manifests and blobs.
101- **Upload size.** On Cloudflare a request body is limited by the zone's plan (Free: 100 MB).
102 Uploads are written to R2 as multipart parts, so a layer may be any size if it arrives in
103 chunks under the limit. `docker push` sends a layer in one request, so on Cloudflare a layer
104 over the limit is refused with a message naming the limit and the way around it: `g1t push`
g1t push: images with layers of any size, uploaded in chunks under the edge's limit105 (the CLI, `crates/g1t`), which reads the image with `docker save` (OCI layout or classic,
106 gzipping uncompressed layers and writing a matching manifest) and sends every blob as OCI
107 chunks of 90 MiB (at most 95 MiB), resuming from the upload's `Range` after a `429`/`5xx`.
108 Self-hosted, there is no such limit.
Packages: the design every phase builds to109
110### npm
111
Merge branch 'worktree-agent-ac1de8a731938ed81'112Built, except the proxy of unscoped packages.
113
Packages: the design every phase builds to114- Registry `https://g1t.sh/-/npm/`; scope = workspace: `@<workspace>/<name>`.
115 `.npmrc`: `@acme:registry=https://g1t.sh/-/npm/` and `//g1t.sh/-/npm/:_authToken=<token>`.
116- `GET /@scope/name` (packument, abbreviated with `Accept: application/vnd.npm.install-v1+json`),
117 `GET` tarballs, `PUT /@scope/name` (publish: JSON with the tarball attached), dist-tags,
118 deprecate, unpublish (within 72 hours or with Admin), `GET /-/whoami`.
119- Unscoped and other scopes: optionally proxied from the public registry and kept, so one
Merge branch 'worktree-agent-ac1de8a731938ed81'120 `.npmrc` line serves everything (later phase; not built).
Packages: the design every phase builds to121
122### Composer
123
Merge branch 'worktree-agent-ac1de8a731938ed81'124Built, except the Packagist mirror.
125
Packages: the design every phase builds to126- Per workspace: `https://g1t.sh/-/composer/<workspace>/` with `packages.json` naming
127 `metadata-url` `/p2/%package%.json` and `available-packages`.
128- Versions come from **the workspace's repositories themselves**: a repository with a
129 `composer.json` at its root is a package (its `name` from that file); each tag is a version and
130 each branch a `dev-` version. Dist archives are zips of the commit, made on first request and
131 kept by commit. Pushing a tag publishes; nothing to upload.
132- Auth: `composer config --auth http-basic.g1t.sh <you> <token>` (`auth.json`).
133- A mirror of the public Packagist (metadata and dists kept, so installs survive its outages) is
Merge branch 'worktree-agent-ac1de8a731938ed81'134 a later phase (not built).
Packages: the design every phase builds to135
136### Cargo
137
Merge branch 'worktree-agent-ac1de8a731938ed81'138Built.
139
140- Sparse registry per workspace: `sparse+https://g1t.sh/-/cargo/<workspace>/index/`. `config.json`
Packages: the design every phase builds to141 (`dl`, `api`, `auth-required` for private), index files at the standard prefix paths, crate
Merge branch 'worktree-agent-ac1de8a731938ed81'142 downloads, `PUT /api/v1/crates/new` (publish), yank and unyank, search. Owners are not kept:
143 who publishes is decided by the repository's or workspace's roles, and `cargo owner` answers
144 with an error saying so.
145- Auth: a g1t token, given to the registry's credential provider (`cargo:token`), named in
146 `.cargo/config.toml` as `[registries.<workspace>]`: `cargo login --registry <workspace>`, or
147 in workflows `CARGO_REGISTRIES_<WORKSPACE>_TOKEN` (with `CARGO_REGISTRIES_<WORKSPACE>_INDEX`
148 and `CARGO_REGISTRIES_<WORKSPACE>_CREDENTIAL_PROVIDER=cargo:token` when no config names the
149 registry). Without a provider Cargo refuses a registry with private crates.
Packages: the design every phase builds to150
151### Go
152
Merge branch 'worktree-agent-ac1de8a731938ed81'153Built from git; the module proxy is not built.
154
Packages: the design every phase builds to155- `go get g1t.sh/<workspace>/<repo>` works from git: repository pages answer `?go-get=1` with the
156 `go-import` meta tag. Private modules need `GOPRIVATE=g1t.sh/<workspace>` and a token in
157 `.netrc` (as for git).
158- A module proxy at `https://g1t.sh/-/go/` (`@v/list`, `.info`, `.mod`, `.zip`) built from tags,
Merge branch 'worktree-agent-ac1de8a731938ed81'159 for faster and repeatable private installs (later phase; not built).
Packages: the design every phase builds to160
Merge branch 'worktree-agent-a6a121745e81f639f'161### Maven
162
Merge branch 'worktree-agent-ac1de8a731938ed81'163Built, for Maven and Gradle.
164
Merge branch 'worktree-agent-a6a121745e81f639f'165- Per workspace: `https://g1t.sh/-/maven/<workspace>/`, the standard layout
166 (`com/acme/web/1.0.0/web-1.0.0.jar`). A package is an artifact, named
167 `groupId:artifactId`; a version holds every file uploaded into its
168 directory, by file name (`version_files`), added one `PUT` at a time.
169- `mvn deploy` and Gradle's `publish`: Basic auth (any username, a token as
170 the password) or `Bearer`. A release's files are written once (`409` for
171 other content, the same content again is accepted); a SNAPSHOT's builds
172 arrive as timestamped files beside each other.
173- Checksums: each file's MD5, SHA-1 and SHA-512 are worked out on upload
174 and kept by digest (`checksums`); `.md5`, `.sha1`, `.sha256`, `.sha512`
175 are answered from them, and uploaded ones are checked, not kept.
176- `maven-metadata.xml` is made on every read: per artifact (versions in
Merge branch 'worktree-agent-ac1de8a731938ed81'177 Maven's order, `latest`, `release`), per SNAPSHOT version (the newest
178 build of each classifier and extension), and per group (`<plugins>`: each
179 `maven-plugin` artifact's prefix, artifactId and name, which is how
180 `mvn <prefix>:<goal>` finds a plugin in its `<pluginGroups>`). The prefix
181 is the `goalPrefix` of the jar's `META-INF/maven/plugin.xml`, read when
182 the jar arrives, else Maven's from the artifactId. A path that is both an
183 artifact's and a group's answers both. Uploaded ones are accepted and let
184 go, a plugin group's included.
185- Gradle: its `.module` files are kept and served like any other file, and
186 its `HEAD` requests and SHA-256 and SHA-512 checksum uploads are
187 answered as Maven's are.
Merge branch 'worktree-agent-a6a121745e81f639f'188- The POM is the version's record: its coordinates must
189 match its path, its description becomes the package's for the highest
190 release, and on a new artifact its `<scm><url>` may link the repository.
191- The artifact's own `maven-metadata.xml`, uploaded last by Maven and Gradle,
192 publishes (event, audit entry) each version or SNAPSHOT build whose POM
193 arrived since, marked in its metadata so it is published once.
194
195### NuGet
196
Merge branch 'worktree-agent-ac1de8a731938ed81'197Built.
198
Merge branch 'worktree-agent-a6a121745e81f639f'199- Per workspace: `https://g1t.sh/-/nuget/<workspace>/v3/index.json` naming
200 `PackageBaseAddress/3.0.0` (flat container), `RegistrationsBaseUrl`
Merge branch 'worktree-agent-ac1de8a731938ed81'201 (one inlined page), `SearchQueryService`, `PackagePublish/2.0.0` and
202 `SymbolPackagePublish/4.9.0`.
Merge branch 'worktree-agent-a6a121745e81f639f'203- `dotnet nuget push`: a `PUT` of a multipart body with the `.nupkg`,
204 `X-NuGet-ApiKey` a g1t token. The `.nuspec` (read from the zip) gives the
205 id, version (normalized as NuGet does), description, dependency groups and
206 README; the `.nupkg` and `.nuspec` are the version's files.
207- Ids are one whatever their case; a version is pushed once (`409`).
208- `DELETE api/v2/package/<id>/<version>` unlists (the `yanked` column), as
209 nuget.org does; `POST` lists again. Unlisted versions stay in the flat
210 container and registration (`listed: false`), not in search.
211- Restores use Basic auth from `nuget.config`, after a `401`.
Merge branch 'worktree-agent-ac1de8a731938ed81'212- Downloads: each `.nupkg` download counts for its version (`versions.downloads`) and its
213 package; the registration's catalog entries and search's versions name them.
214- Symbols: `dotnet nuget push` sends the `.snupkg` beside a `.nupkg` to
215 `api/v2/symbolpackage` after it. It must be for a version already pushed, say
216 `SymbolsPackage` as its package type, and hold portable PDBs. The `.snupkg` is the version's
217 file `snupkg` (also in the flat container), and each PDB its file `pdb:<file>:<key>`, the key
218 being the PDB id's GUID (as `Guid.ToString("N")`) and `ffffffff`. The symbol server,
219 `symbols/<file>.pdb/<key>/<file>.pdb` (the Simple Symbol Query Protocol), finds it by that
220 name across the workspace's packages the viewer may read (`version_files_name` index). A
221 version's symbols are pushed once (`409`). The package page marks versions with symbols.
Merge branch 'worktree-agent-a6a121745e81f639f'222
223### RubyGems
224
Merge branch 'worktree-agent-ac1de8a731938ed81'225Built.
226
Merge branch 'worktree-agent-a6a121745e81f639f'227- Per workspace: `https://g1t.sh/-/rubygems/<workspace>/`. `gem push`
228 (`POST /api/v1/gems`, the token as the whole `Authorization` header),
229 `gem yank` (`DELETE /api/v1/gems/yank`), downloads at
230 `/gems/<name>-<version>[-<platform>].gem`.
231- The compact index Bundler reads: `versions`, `info/<gem>`, `names`, made
232 on every read, each with the quoted MD5 of its body as its `ETag` (Bundler
233 checks it, and `versions` names each info file's MD5). Yanked versions
234 leave the index; their files stay.
235- The gem's `metadata.gz` (YAML, in the `.gem` tar) gives the name,
236 version, platform and runtime dependencies. A version is keyed as the
237 index writes it (`1.0.0`, `1.0.0-x86_64-linux`) and pushed once.
Merge branch 'worktree-agent-ac1de8a731938ed81'238- The full index `gem install --source` and `gem search` read:
239 `specs.4.8.gz` (released versions), `latest_specs.4.8.gz` (the highest of
240 each gem and platform) and `prerelease_specs.4.8.gz`, each a gzipped
241 Ruby Marshal 4.8 array of `[name, Gem::Version, platform]`, and
242 `quick/Marshal.4.8/<name>-<version>[-<platform>].gemspec.rz`, the
243 deflated Marshal of a `Gem::Specification` as its `_dump` writes it. All
244 are made on each read from what each version keeps (`src/marshal.rs` is
245 the writer), and read by RubyGems' `SafeMarshal`.
246- Bundler authenticates with Basic auth from `bundle config`; `gem` with
247 credentials in the source's address.
Merge branch 'worktree-agent-a6a121745e81f639f'248
Packages: the design every phase builds to249### Later
250
Merge branch 'worktree-agent-ac1de8a731938ed81'251PyPI (not built).
Packages: the design every phase builds to252
253## Billing
254
255Storage is what costs: R2 is about $0.015 per GB-month, with no charge for downloads (no egress
256fees) and fractions of a cent per thousand requests. Defaults, all `services/billing` variables:
257
258| | Free workspace | g1t plan |
259| --- | --- | --- |
260| Public packages | Free up to `PUBLIC_PACKAGES_FREE_BYTES` (10 GB) per workspace; pushes past it are refused | The same 10 GB free, then storage at cost plus the margin |
261| Private packages | `PRIVATE_PACKAGES_FREE_BYTES` (500 MB) free; pushes past it are refused | Storage at cost plus the margin from the first byte past 500 MB |
262| Downloads | Free; anonymous pulls rate limited | Free |
263
264- A workspace's package storage is measured daily from `workspace_blobs` (a blob counts once per
265 workspace, and as public when any public package uses it), on the same meter run as private
266 repository storage, and charged as `package_storage` GB-months.
267- Pushes check the free limits before accepting a blob, with a message naming the limit.
268
269## Events
270
271`package.published`, `package.version_deleted`, `package.deleted`, `package.visibility_changed`.
272Webhooks deliver them (`package` and `registry_package` shapes); workflows can run on
273`registry_package`; the activity feed and search index them; the audit log records them.
274
275## UI
276
277- Workspace **Packages** (`/<workspace>/-/packages`): every package, by ecosystem, with search.
278- A project's packages on its overview, and a Packages tab when it has any.
279- Package page: install and log-in commands for its tool, versions and tags, README, size,
280 downloads, linked repository, who published; settings (visibility, link, delete) for Admins.
281- Site-wide search finds public packages.
282
283## Build order
284
2851. **Core and containers.** The service, file store port (R2, S3, disk), access, token exchange,
286 OCI pull and push (chunked and multipart, mounts, deletes, referrers), the sweep, events,
287 billing meter and limits, Packages pages, docs. Then the runner's image moves here from Docker
288 Hub, and `g1t push` for large layers.
2892. **npm.**
2903. **Composer and Go** (both from the repositories themselves).
2914. **Cargo.**
Merge branch 'worktree-agent-a6a121745e81f639f'2925. **Maven, NuGet, RubyGems.**
2936. **Mirrors** (Packagist, npm).
Merge branch 'worktree-agent-ac1de8a731938ed81'294
295Phases 1 to 5 are built (see [Status](#status)); 6 is not.

This file's history is long; its oldest lines are credited to the oldest commit read.