| 1 | # Packages: design |
| 2 | |
| 3 | Packages are registries a workspace publishes to and installs from, beside its code: container |
| 4 | images, npm, Composer, Cargo and Go first, then Maven, NuGet and RubyGems. They live where the |
| 5 | code does, take the same people and tokens, and are built and published by g1t Actions and |
| 6 | agents without a second account anywhere. |
| 7 | |
| 8 | This is the design every phase builds to. Each ecosystem's own guide (apps/docs) says how to use |
| 9 | it; this says how it works. |
| 10 | |
| 11 | ## Principles |
| 12 | |
| 13 | - **One service.** `services/packages` (Rust Worker) owns every registry: its own D1 database |
| 14 | (`g1t-packages`), its own file store, its own contract in `crates/contracts` / |
| 15 | `packages/contracts`. Other services talk to it over RPC and hear from it through events. |
| 16 | - **Cloudflare in production, anything in a self-hosted install.** Files go through a |
| 17 | `BlobStore` port. In production its adapter is R2; self-hosted it is S3-compatible storage |
| 18 | (MinIO in the compose file) or a local directory. Upload URLs (`presign`) are part of the port: |
| 19 | R2 and S3 sign them, the disk adapter answers with a path back through the service. |
| 20 | Metadata is D1, which self-hosting already runs (workerd's D1 over SQLite). |
| 21 | - **Content-addressed.** Every file is stored once by its SHA-256 (`blobs/sha256/<hex>`). A |
| 22 | version is a list of files by digest. Pushing a layer two images share stores it once. |
| 23 | - **The registry speaks each tool's own protocol**, unchanged. `docker`, `npm`, `composer`, |
| 24 | `cargo` and `go` work with only a login and an address. |
| 25 | - **Same access as the code.** A package can be linked to a repository and then has its |
| 26 | visibility and roles. Unlinked, it is the workspace's: members by the base permission. |
| 27 | - **Same front door.** Everything is on `g1t.sh`. `apps/web/workers/app.ts` hands registry paths |
| 28 | to the `PACKAGES` binding the way it hands git paths to repos. |
| 29 | |
| 30 | ## Model |
| 31 | |
| 32 | | Table | What it holds | |
| 33 | | --- | --- | |
| 34 | | `packages` | `id`, `workspace`, `ecosystem` (`container`, `npm`, `composer`, `cargo`, `go`, ...), `name` (normalized per ecosystem), `repo_id` (linked repository, or null), `visibility` (`public`, `private`; linked packages follow their repository), `description`, `readme_digest`, `created_by`, `created_at`, `updated_at`, `downloads` | |
| 35 | | `versions` | `id`, `package_id`, `version` (tag, semver or digest), `digest` (the manifest's or the archive's), `size` (sum of its files), `metadata` (JSON the ecosystem needs: npm's packument entry, a crate's index line, composer.json), `published_by`, `published_at`, `yanked`, `deprecated` | |
| 36 | | `version_files` | `version_id`, `name`, `digest`, `size`, `media_type` | |
| 37 | | `blobs` | `digest`, `size`, `media_type`, `created_at` | |
| 38 | | `workspace_blobs` | `workspace`, `digest`, `public` (any public package uses it): what a workspace stores, counted once each, for billing | |
| 39 | | `uploads` | `id`, `workspace`, `package`, `multipart_id`, `parts` (JSON), `offset`, `hash_state`, `expires_at`: uploads in progress | |
| 40 | | `tags` | `package_id`, `tag`, `version_id`: container tags and npm dist-tags | |
| 41 | |
| 42 | Deleting a version removes its rows; a daily sweep deletes blobs no version references, after a |
| 43 | day's grace (a push in flight may reference a blob before its manifest lands). |
| 44 | |
| 45 | ## Access |
| 46 | |
| 47 | - **Who:** people (session or personal token), workspace tokens, `G1T_TOKEN` in workflows, |
| 48 | agents' run tokens. Scopes `packages:read` and `packages:write` (and `packages:delete`), per the |
| 49 | MCP and token scope design. |
| 50 | - **What:** linked package: the repository's roles (Read pulls, Write publishes, Admin deletes and |
| 51 | changes settings). Unlinked: workspace members by base permission; owners delete. |
| 52 | - **Public packages** pull anonymously. Anonymous pulls are rate limited per address. |
| 53 | - **Workflows** may publish the packages of their own repository, and new ones linked to it, with |
| 54 | `G1T_TOKEN`, as code they push does. |
| 55 | - Every publish and delete is an audit entry and an event. |
| 56 | |
| 57 | ## Protocols |
| 58 | |
| 59 | All under `g1t.sh`. A name always starts with the workspace. |
| 60 | |
| 61 | ### Containers (OCI Distribution 1.1) |
| 62 | |
| 63 | - Image names: `g1t.sh/<workspace>/<name>[:tag]`, `<name>` may contain `/`. Usually the |
| 64 | repository's name, and then linked to it. |
| 65 | - `GET /v2/` answers 401 with `WWW-Authenticate: Bearer realm="https://g1t.sh/v2/token", |
| 66 | service="g1t.sh"`. `GET /v2/token` takes Basic auth (any username, a g1t token as the |
| 67 | password) or none (anonymous, public pulls) and returns a short-lived signed token naming the |
| 68 | scopes granted. `docker login g1t.sh -u <you> --password-stdin` stores it. |
| 69 | - Pull: `HEAD/GET /v2/<name>/manifests/<ref>`, `HEAD/GET /v2/<name>/blobs/<digest>` (a |
| 70 | redirect to a signed R2 URL for large blobs, so downloads never pass through the Worker), |
| 71 | `GET /v2/<name>/tags/list`, `GET /v2/<name>/referrers/<digest>`. |
| 72 | - Push: `POST /v2/<name>/blobs/uploads/` (`?mount=&from=` cross-repository mount within a |
| 73 | workspace; `?digest=` monolithic), `PATCH` chunks, `PUT ?digest=` to finish, |
| 74 | `PUT /v2/<name>/manifests/<ref>` (image manifests, indexes, OCI artifacts, `subject` for |
| 75 | referrers). `DELETE` manifests and blobs. |
| 76 | - **Upload size.** On Cloudflare a request body is limited by the zone's plan (Free: 100 MB). |
| 77 | Uploads are written to R2 as multipart parts, so a layer may be any size if it arrives in |
| 78 | chunks under the limit. `docker push` sends a layer in one request, so on Cloudflare a layer |
| 79 | over the limit is refused with a message naming the limit and the way around it: `g1t push` |
| 80 | (the CLI, `crates/g1t`), which reads the image with `docker save` (OCI layout or classic, |
| 81 | gzipping uncompressed layers and writing a matching manifest) and sends every blob as OCI |
| 82 | chunks of 90 MiB (at most 95 MiB), resuming from the upload's `Range` after a `429`/`5xx`. |
| 83 | Self-hosted, there is no such limit. |
| 84 | |
| 85 | ### npm |
| 86 | |
| 87 | - Registry `https://g1t.sh/-/npm/`; scope = workspace: `@<workspace>/<name>`. |
| 88 | `.npmrc`: `@acme:registry=https://g1t.sh/-/npm/` and `//g1t.sh/-/npm/:_authToken=<token>`. |
| 89 | - `GET /@scope/name` (packument, abbreviated with `Accept: application/vnd.npm.install-v1+json`), |
| 90 | `GET` tarballs, `PUT /@scope/name` (publish: JSON with the tarball attached), dist-tags, |
| 91 | deprecate, unpublish (within 72 hours or with Admin), `GET /-/whoami`. |
| 92 | - Unscoped and other scopes: optionally proxied from the public registry and kept, so one |
| 93 | `.npmrc` line serves everything (later phase). |
| 94 | |
| 95 | ### Composer |
| 96 | |
| 97 | - Per workspace: `https://g1t.sh/-/composer/<workspace>/` with `packages.json` naming |
| 98 | `metadata-url` `/p2/%package%.json` and `available-packages`. |
| 99 | - Versions come from **the workspace's repositories themselves**: a repository with a |
| 100 | `composer.json` at its root is a package (its `name` from that file); each tag is a version and |
| 101 | each branch a `dev-` version. Dist archives are zips of the commit, made on first request and |
| 102 | kept by commit. Pushing a tag publishes; nothing to upload. |
| 103 | - Auth: `composer config --auth http-basic.g1t.sh <you> <token>` (`auth.json`). |
| 104 | - A mirror of the public Packagist (metadata and dists kept, so installs survive its outages) is |
| 105 | a later phase. |
| 106 | |
| 107 | ### Cargo |
| 108 | |
| 109 | - Sparse registry per workspace: `sparse+https://g1t.sh/-/cargo/<workspace>/`. `config.json` |
| 110 | (`dl`, `api`, `auth-required` for private), index files at the standard prefix paths, crate |
| 111 | downloads, `PUT /api/v1/crates/new` (publish), yank and unyank, owners. |
| 112 | - Auth: a g1t token through `cargo login --registry g1t`. |
| 113 | |
| 114 | ### Go |
| 115 | |
| 116 | - `go get g1t.sh/<workspace>/<repo>` works from git: repository pages answer `?go-get=1` with the |
| 117 | `go-import` meta tag. Private modules need `GOPRIVATE=g1t.sh/<workspace>` and a token in |
| 118 | `.netrc` (as for git). |
| 119 | - A module proxy at `https://g1t.sh/-/go/` (`@v/list`, `.info`, `.mod`, `.zip`) built from tags, |
| 120 | for faster and repeatable private installs (later phase). |
| 121 | |
| 122 | ### Later |
| 123 | |
| 124 | Maven (`/-/maven/<workspace>/`), NuGet (v3), RubyGems; PyPI. |
| 125 | |
| 126 | ## Billing |
| 127 | |
| 128 | Storage is what costs: R2 is about $0.015 per GB-month, with no charge for downloads (no egress |
| 129 | fees) and fractions of a cent per thousand requests. Defaults, all `services/billing` variables: |
| 130 | |
| 131 | | | Free workspace | g1t plan | |
| 132 | | --- | --- | --- | |
| 133 | | Public packages | Free up to `PUBLIC_PACKAGES_FREE_BYTES` (10 GB) per workspace; pushes past it are refused | The same 10 GB free, then storage at cost plus the margin | |
| 134 | | Private packages | `PRIVATE_PACKAGES_FREE_BYTES` (500 MB) free; pushes past it are refused | Storage at cost plus the margin from the first byte past 500 MB | |
| 135 | | Downloads | Free; anonymous pulls rate limited | Free | |
| 136 | |
| 137 | - A workspace's package storage is measured daily from `workspace_blobs` (a blob counts once per |
| 138 | workspace, and as public when any public package uses it), on the same meter run as private |
| 139 | repository storage, and charged as `package_storage` GB-months. |
| 140 | - Pushes check the free limits before accepting a blob, with a message naming the limit. |
| 141 | |
| 142 | ## Events |
| 143 | |
| 144 | `package.published`, `package.version_deleted`, `package.deleted`, `package.visibility_changed`. |
| 145 | Webhooks deliver them (`package` and `registry_package` shapes); workflows can run on |
| 146 | `registry_package`; the activity feed and search index them; the audit log records them. |
| 147 | |
| 148 | ## UI |
| 149 | |
| 150 | - Workspace **Packages** (`/<workspace>/-/packages`): every package, by ecosystem, with search. |
| 151 | - A project's packages on its overview, and a Packages tab when it has any. |
| 152 | - Package page: install and log-in commands for its tool, versions and tags, README, size, |
| 153 | downloads, linked repository, who published; settings (visibility, link, delete) for Admins. |
| 154 | - Site-wide search finds public packages. |
| 155 | |
| 156 | ## Build order |
| 157 | |
| 158 | 1. **Core and containers.** The service, file store port (R2, S3, disk), access, token exchange, |
| 159 | OCI pull and push (chunked and multipart, mounts, deletes, referrers), the sweep, events, |
| 160 | billing meter and limits, Packages pages, docs. Then the runner's image moves here from Docker |
| 161 | Hub, and `g1t push` for large layers. |
| 162 | 2. **npm.** |
| 163 | 3. **Composer and Go** (both from the repositories themselves). |
| 164 | 4. **Cargo.** |
| 165 | 5. **Mirrors** (Packagist, npm), **Maven, NuGet, RubyGems.** |