Packages: the design every phase builds to
docs/PACKAGES.md: one packages service; files by SHA-256 through a BlobStore port (R2 in production, S3-compatible or disk self-hosted); access as the code's; container, npm, Composer, Cargo and Go protocols on g1t.sh; the 100 MB request limit on Cloudflare and the way around it; billing (public free to 10 GB, private 500 MB free then at cost plus the margin); events; pages; build order.
1 file+163−00/1 viewed
| 1 | + | # Packages: design | |
| 2 | + | ||
| 3 | + | Packages are registries a workspace publishes to and installs from, beside its code: container | |
| 4 | + | images, npm, Composer, Cargo and Go first, then Maven, NuGet and RubyGems. They live where the | |
| 5 | + | code does, take the same people and tokens, and are built and published by g1t Actions and | |
| 6 | + | agents without a second account anywhere. | |
| 7 | + | ||
| 8 | + | This is the design every phase builds to. Each ecosystem's own guide (apps/docs) says how to use | |
| 9 | + | it; this says how it works. | |
| 10 | + | ||
| 11 | + | ## Principles | |
| 12 | + | ||
| 13 | + | - **One service.** `services/packages` (Rust Worker) owns every registry: its own D1 database | |
| 14 | + | (`g1t-packages`), its own file store, its own contract in `crates/contracts` / | |
| 15 | + | `packages/contracts`. Other services talk to it over RPC and hear from it through events. | |
| 16 | + | - **Cloudflare in production, anything in a self-hosted install.** Files go through a | |
| 17 | + | `BlobStore` port. In production its adapter is R2; self-hosted it is S3-compatible storage | |
| 18 | + | (MinIO in the compose file) or a local directory. Upload URLs (`presign`) are part of the port: | |
| 19 | + | R2 and S3 sign them, the disk adapter answers with a path back through the service. | |
| 20 | + | Metadata is D1, which self-hosting already runs (workerd's D1 over SQLite). | |
| 21 | + | - **Content-addressed.** Every file is stored once by its SHA-256 (`blobs/sha256/<hex>`). A | |
| 22 | + | version is a list of files by digest. Pushing a layer two images share stores it once. | |
| 23 | + | - **The registry speaks each tool's own protocol**, unchanged. `docker`, `npm`, `composer`, | |
| 24 | + | `cargo` and `go` work with only a login and an address. | |
| 25 | + | - **Same access as the code.** A package can be linked to a repository and then has its | |
| 26 | + | visibility and roles. Unlinked, it is the workspace's: members by the base permission. | |
| 27 | + | - **Same front door.** Everything is on `g1t.sh`. `apps/web/workers/app.ts` hands registry paths | |
| 28 | + | to the `PACKAGES` binding the way it hands git paths to repos. | |
| 29 | + | ||
| 30 | + | ## Model | |
| 31 | + | ||
| 32 | + | | Table | What it holds | | |
| 33 | + | | --- | --- | | |
| 34 | + | | `packages` | `id`, `workspace`, `ecosystem` (`container`, `npm`, `composer`, `cargo`, `go`, ...), `name` (normalized per ecosystem), `repo_id` (linked repository, or null), `visibility` (`public`, `private`; linked packages follow their repository), `description`, `readme_digest`, `created_by`, `created_at`, `updated_at`, `downloads` | | |
| 35 | + | | `versions` | `id`, `package_id`, `version` (tag, semver or digest), `digest` (the manifest's or the archive's), `size` (sum of its files), `metadata` (JSON the ecosystem needs: npm's packument entry, a crate's index line, composer.json), `published_by`, `published_at`, `yanked`, `deprecated` | | |
| 36 | + | | `version_files` | `version_id`, `name`, `digest`, `size`, `media_type` | | |
| 37 | + | | `blobs` | `digest`, `size`, `media_type`, `created_at` | | |
| 38 | + | | `workspace_blobs` | `workspace`, `digest`, `public` (any public package uses it): what a workspace stores, counted once each, for billing | | |
| 39 | + | | `uploads` | `id`, `workspace`, `package`, `multipart_id`, `parts` (JSON), `offset`, `hash_state`, `expires_at`: uploads in progress | | |
| 40 | + | | `tags` | `package_id`, `tag`, `version_id`: container tags and npm dist-tags | | |
| 41 | + | ||
| 42 | + | Deleting a version removes its rows; a daily sweep deletes blobs no version references, after a | |
| 43 | + | day's grace (a push in flight may reference a blob before its manifest lands). | |
| 44 | + | ||
| 45 | + | ## Access | |
| 46 | + | ||
| 47 | + | - **Who:** people (session or personal token), workspace tokens, `G1T_TOKEN` in workflows, | |
| 48 | + | agents' run tokens. Scopes `packages:read` and `packages:write` (and `packages:delete`), per the | |
| 49 | + | MCP and token scope design. | |
| 50 | + | - **What:** linked package: the repository's roles (Read pulls, Write publishes, Admin deletes and | |
| 51 | + | changes settings). Unlinked: workspace members by base permission; owners delete. | |
| 52 | + | - **Public packages** pull anonymously. Anonymous pulls are rate limited per address. | |
| 53 | + | - **Workflows** may publish the packages of their own repository, and new ones linked to it, with | |
| 54 | + | `G1T_TOKEN`, as code they push does. | |
| 55 | + | - Every publish and delete is an audit entry and an event. | |
| 56 | + | ||
| 57 | + | ## Protocols | |
| 58 | + | ||
| 59 | + | All under `g1t.sh`. A name always starts with the workspace. | |
| 60 | + | ||
| 61 | + | ### Containers (OCI Distribution 1.1) | |
| 62 | + | ||
| 63 | + | - Image names: `g1t.sh/<workspace>/<name>[:tag]`, `<name>` may contain `/`. Usually the | |
| 64 | + | repository's name, and then linked to it. | |
| 65 | + | - `GET /v2/` answers 401 with `WWW-Authenticate: Bearer realm="https://g1t.sh/v2/token", | |
| 66 | + | service="g1t.sh"`. `GET /v2/token` takes Basic auth (any username, a g1t token as the | |
| 67 | + | password) or none (anonymous, public pulls) and returns a short-lived signed token naming the | |
| 68 | + | scopes granted. `docker login g1t.sh -u <you> --password-stdin` stores it. | |
| 69 | + | - Pull: `HEAD/GET /v2/<name>/manifests/<ref>`, `HEAD/GET /v2/<name>/blobs/<digest>` (a | |
| 70 | + | redirect to a signed R2 URL for large blobs, so downloads never pass through the Worker), | |
| 71 | + | `GET /v2/<name>/tags/list`, `GET /v2/<name>/referrers/<digest>`. | |
| 72 | + | - Push: `POST /v2/<name>/blobs/uploads/` (`?mount=&from=` cross-repository mount within a | |
| 73 | + | workspace; `?digest=` monolithic), `PATCH` chunks, `PUT ?digest=` to finish, | |
| 74 | + | `PUT /v2/<name>/manifests/<ref>` (image manifests, indexes, OCI artifacts, `subject` for | |
| 75 | + | referrers). `DELETE` manifests and blobs. | |
| 76 | + | - **Upload size.** On Cloudflare a request body is limited by the zone's plan (Free: 100 MB). | |
| 77 | + | Uploads are written to R2 as multipart parts, so a layer may be any size if it arrives in | |
| 78 | + | chunks under the limit. `docker push` sends a layer in one request, so on Cloudflare a layer | |
| 79 | + | over the limit is refused with a message naming the limit and the way around it: `g1t push` | |
| 80 | + | (the CLI), which asks for signed part URLs and uploads straight to R2 at any size, and is what | |
| 81 | + | g1t Actions and the runner use. Self-hosted, there is no such limit. | |
| 82 | + | ||
| 83 | + | ### npm | |
| 84 | + | ||
| 85 | + | - Registry `https://g1t.sh/-/npm/`; scope = workspace: `@<workspace>/<name>`. | |
| 86 | + | `.npmrc`: `@acme:registry=https://g1t.sh/-/npm/` and `//g1t.sh/-/npm/:_authToken=<token>`. | |
| 87 | + | - `GET /@scope/name` (packument, abbreviated with `Accept: application/vnd.npm.install-v1+json`), | |
| 88 | + | `GET` tarballs, `PUT /@scope/name` (publish: JSON with the tarball attached), dist-tags, | |
| 89 | + | deprecate, unpublish (within 72 hours or with Admin), `GET /-/whoami`. | |
| 90 | + | - Unscoped and other scopes: optionally proxied from the public registry and kept, so one | |
| 91 | + | `.npmrc` line serves everything (later phase). | |
| 92 | + | ||
| 93 | + | ### Composer | |
| 94 | + | ||
| 95 | + | - Per workspace: `https://g1t.sh/-/composer/<workspace>/` with `packages.json` naming | |
| 96 | + | `metadata-url` `/p2/%package%.json` and `available-packages`. | |
| 97 | + | - Versions come from **the workspace's repositories themselves**: a repository with a | |
| 98 | + | `composer.json` at its root is a package (its `name` from that file); each tag is a version and | |
| 99 | + | each branch a `dev-` version. Dist archives are zips of the commit, made on first request and | |
| 100 | + | kept by commit. Pushing a tag publishes; nothing to upload. | |
| 101 | + | - Auth: `composer config --auth http-basic.g1t.sh <you> <token>` (`auth.json`). | |
| 102 | + | - A mirror of the public Packagist (metadata and dists kept, so installs survive its outages) is | |
| 103 | + | a later phase. | |
| 104 | + | ||
| 105 | + | ### Cargo | |
| 106 | + | ||
| 107 | + | - Sparse registry per workspace: `sparse+https://g1t.sh/-/cargo/<workspace>/`. `config.json` | |
| 108 | + | (`dl`, `api`, `auth-required` for private), index files at the standard prefix paths, crate | |
| 109 | + | downloads, `PUT /api/v1/crates/new` (publish), yank and unyank, owners. | |
| 110 | + | - Auth: a g1t token through `cargo login --registry g1t`. | |
| 111 | + | ||
| 112 | + | ### Go | |
| 113 | + | ||
| 114 | + | - `go get g1t.sh/<workspace>/<repo>` works from git: repository pages answer `?go-get=1` with the | |
| 115 | + | `go-import` meta tag. Private modules need `GOPRIVATE=g1t.sh/<workspace>` and a token in | |
| 116 | + | `.netrc` (as for git). | |
| 117 | + | - A module proxy at `https://g1t.sh/-/go/` (`@v/list`, `.info`, `.mod`, `.zip`) built from tags, | |
| 118 | + | for faster and repeatable private installs (later phase). | |
| 119 | + | ||
| 120 | + | ### Later | |
| 121 | + | ||
| 122 | + | Maven (`/-/maven/<workspace>/`), NuGet (v3), RubyGems; PyPI. | |
| 123 | + | ||
| 124 | + | ## Billing | |
| 125 | + | ||
| 126 | + | Storage is what costs: R2 is about $0.015 per GB-month, with no charge for downloads (no egress | |
| 127 | + | fees) and fractions of a cent per thousand requests. Defaults, all `services/billing` variables: | |
| 128 | + | ||
| 129 | + | | | Free workspace | g1t plan | | |
| 130 | + | | --- | --- | --- | | |
| 131 | + | | Public packages | Free up to `PUBLIC_PACKAGES_FREE_BYTES` (10 GB) per workspace; pushes past it are refused | The same 10 GB free, then storage at cost plus the margin | | |
| 132 | + | | Private packages | `PRIVATE_PACKAGES_FREE_BYTES` (500 MB) free; pushes past it are refused | Storage at cost plus the margin from the first byte past 500 MB | | |
| 133 | + | | Downloads | Free; anonymous pulls rate limited | Free | | |
| 134 | + | ||
| 135 | + | - A workspace's package storage is measured daily from `workspace_blobs` (a blob counts once per | |
| 136 | + | workspace, and as public when any public package uses it), on the same meter run as private | |
| 137 | + | repository storage, and charged as `package_storage` GB-months. | |
| 138 | + | - Pushes check the free limits before accepting a blob, with a message naming the limit. | |
| 139 | + | ||
| 140 | + | ## Events | |
| 141 | + | ||
| 142 | + | `package.published`, `package.version_deleted`, `package.deleted`, `package.visibility_changed`. | |
| 143 | + | Webhooks deliver them (`package` and `registry_package` shapes); workflows can run on | |
| 144 | + | `registry_package`; the activity feed and search index them; the audit log records them. | |
| 145 | + | ||
| 146 | + | ## UI | |
| 147 | + | ||
| 148 | + | - Workspace **Packages** (`/<workspace>/-/packages`): every package, by ecosystem, with search. | |
| 149 | + | - A project's packages on its overview, and a Packages tab when it has any. | |
| 150 | + | - Package page: install and log-in commands for its tool, versions and tags, README, size, | |
| 151 | + | downloads, linked repository, who published; settings (visibility, link, delete) for Admins. | |
| 152 | + | - Site-wide search finds public packages. | |
| 153 | + | ||
| 154 | + | ## Build order | |
| 155 | + | ||
| 156 | + | 1. **Core and containers.** The service, file store port (R2, S3, disk), access, token exchange, | |
| 157 | + | OCI pull and push (chunked and multipart, mounts, deletes, referrers), the sweep, events, | |
| 158 | + | billing meter and limits, Packages pages, docs. Then the runner's image moves here from Docker | |
| 159 | + | Hub, and `g1t push` for large layers. | |
| 160 | + | 2. **npm.** | |
| 161 | + | 3. **Composer and Go** (both from the repositories themselves). | |
| 162 | + | 4. **Cargo.** | |
| 163 | + | 5. **Mirrors** (Packagist, npm), **Maven, NuGet, RubyGems.** |