Sign in

flagon-io/g1t

public

Commit

NuGet: symbol packages (.snupkg) pushed to SymbolPackagePublish, a symbol server serving their portable PDBs by key, Symbols on the package page; each .nupkg download counts for its version

syntaqxcommitted Parentdbb0224Browse files
10 files+466−310/10 viewed
+46−4
8282 [who can see and publish a package](/guides/packages/#who-can-see-and-publish-a-package).
8383
8484 The package's page on g1t.sh shows the README the package names
85−(`PackageReadmeFile`) and the description of its highest stable version.
85+(`PackageReadmeFile`) and the description of its highest stable version,
86+and each version's downloads: every `.nupkg` restored counts for its
87+version, and the registration (`downloads` in each catalog entry) and
88+search (each version's `downloads`, and the package's `totalDownloads`)
89+say them too. Counts are approximate.
90+
91+## Symbols
92+
93+Push a symbol package beside the package, and debuggers can step into
94+its code: the feed has a symbol server that serves each PDB by the key
95+the debugger asks for. Build a `.snupkg` with the package:
96+
97+```xml
98+<PropertyGroup>
99+ <IncludeSymbols>true</IncludeSymbols>
100+ <SymbolPackageFormat>snupkg</SymbolPackageFormat>
101+</PropertyGroup>
102+```
103+
104+`dotnet pack` then writes `Acme.Http.0.3.1.snupkg` beside the `.nupkg`,
105+and `dotnet nuget push` of the `.nupkg` pushes it after the package, to
106+the feed's `SymbolPackagePublish` resource, with the same API key. A
107+symbol package is for a version already pushed; its PDBs must be portable
108+PDBs (`DebugType` `portable`, the default). A version's symbols are pushed
109+once. Its page marks the versions that have them, and the `.snupkg` is in
110+the flat container beside the `.nupkg`.
111+
112+The symbol server is at:
86113
114+```text
115+https://g1t.sh/-/nuget/<workspace>/symbols/
116+```
117+
118+Add that address as a symbol server in your debugger (in Visual Studio,
119+**Tools > Options > Debugging > Symbols**). It answers the Simple Symbol
120+Query Protocol, `symbols/<file>.pdb/<key>/<file>.pdb`, for the
121+packages the credentials' owner may see; debuggers that send no
122+credentials to a symbol server load the symbols of public packages.
123+`dotnet-symbol` sends a token with `--authenticated-server-path`:
124+
125+```sh
126+dotnet-symbol --authenticated-server-path <token> https://g1t.sh/-/nuget/acme/symbols/ -o symbols bin/Debug/net8.0/Acme.Http.dll
127+```
128+
87129 ## Restore
88130
89131 ```sh
177219 | --- | --- |
178220 | `401` | No credentials or API key, or a wrong or expired token. Check the source's username and password, or the `--api-key`. |
179221 | `403` | Signed in, but your role or your token's scopes do not allow it, or the workspace is out of free package storage. The response says which. |
180−| `404` | No such package or version, or a private one you cannot see. |
181−| `409` | That version is already pushed. Bump `Version`. |
182−| `400` | The push was refused: not a `.nupkg`, no `.nuspec` in it, or an id or version NuGet would not take. The response says which. |
222+| `404` | No such package or version, or a private one you cannot see. For a symbol package: its version is not pushed yet. |
223+| `409` | That version is already pushed, or already has symbols. Bump `Version`. |
224+| `400` | The push was refused: not a `.nupkg`, no `.nuspec` in it, an id or version NuGet would not take, or a symbol package that is not one or holds a PDB that is not portable. The response says which. |
183225 | `413` | The `.nupkg` is over 100 MB. |
+4−4
1717 | npm | `https://g1t.sh/-/npm/`, for the scope `@<workspace>` | [npm](/guides/npm/) |
1818 | Cargo | `sparse+https://g1t.sh/-/cargo/<workspace>/index/`, a registry per workspace | [Cargo](/guides/cargo/) |
1919 | Maven | `https://g1t.sh/-/maven/<workspace>/`, a repository per workspace, for Maven and Gradle | [Maven](/guides/maven/) |
20−| NuGet | `https://g1t.sh/-/nuget/<workspace>/v3/index.json`, a feed per workspace | [NuGet](/guides/nuget/) |
21−| RubyGems | `https://g1t.sh/-/rubygems/<workspace>/`, a registry per workspace, for `gem push` and Bundler | [RubyGems](/guides/rubygems/) |
20+| NuGet | `https://g1t.sh/-/nuget/<workspace>/v3/index.json`, a feed per workspace, with a symbol server | [NuGet](/guides/nuget/) |
21+| RubyGems | `https://g1t.sh/-/rubygems/<workspace>/`, a registry per workspace, for `gem push`, `gem install` and Bundler | [RubyGems](/guides/rubygems/) |
2222 | Composer | `https://g1t.sh/-/composer/<workspace>/`, from the workspace's repositories | [Composer](/guides/composer/) |
2323 | Go | `g1t.sh/<workspace>/<repo>`, straight from git | [Go modules](/guides/go/) |
2424
101101
102102 Publishing a version, deleting a version and deleting a package are
103103 [audit log](/guides/audit-log/) entries (so are deprecating an npm version,
104−yanking or unyanking a crate version, unlisting or listing a NuGet version
105−and yanking a gem version), and the events
104+yanking or unyanking a crate version, unlisting or listing a NuGet version,
105+pushing a NuGet version's symbols and yanking a gem version), and the events
106106 `package.published`, `package.version_deleted`, `package.deleted` and
107107 `package.visibility_changed`, which [webhooks](/guides/webhooks/) can be
108108 sent: a linked package's go to its repository's webhooks and its
+10−0
230230 Deprecated
231231 </Badge>
232232 )}
233+ {version.symbols && (
234+ <Badge tone="neutral" title="A symbol package (.snupkg) was pushed: debuggers load its PDBs from the feed's symbol server.">
235+ Symbols
236+ </Badge>
237+ )}
233238 </div>
234239 {version.deprecated && <p className="text-xs text-muted">{version.deprecated}</p>}
235240 <p className="flex flex-wrap gap-x-3 text-xs text-faint tabular-nums">
236241 <span>{formatBytes(version.size)}</span>
242+ {version.downloads != null && (
243+ <span>
244+ {version.downloads.toLocaleString("en-US")} {version.downloads === 1 ? "download" : "downloads"}
245+ </span>
246+ )}
237247 {version.platforms.length > 0 && <span>{version.platforms.join(", ")}</span>}
238248 {attached.length > 0 && (
239249 <span title={attached.map((a) => a.artifact_type ?? a.media_type ?? "artifact").join(", ")}>
+6−0
140140 /// npm: why the version should no longer be used, when it is deprecated.
141141 #[serde(default)]
142142 pub deprecated: Option<String>,
143+ /// NuGet: whether a symbol package (`.snupkg`) was pushed for it.
144+ #[serde(default)]
145+ pub symbols: bool,
146+ /// NuGet: its own downloads, where they are counted by version.
147+ #[serde(default)]
148+ pub downloads: Option<u64>,
143149 }
144150
145151 #[derive(Clone, Debug, Serialize, Deserialize)]
+4−0
6262 published_at: string;
6363 /** npm: why the version should no longer be used, when it is deprecated. */
6464 deprecated?: string | null;
65+ /** NuGet: whether a symbol package (`.snupkg`) was pushed for it. */
66+ symbols?: boolean;
67+ /** NuGet: its own downloads, where they are counted by version. */
68+ downloads?: number | null;
6569 };
6670
6771 export type PackageTag = { tag: string; digest: string; updated_at: string };
+7−0
1+-- Each version's own downloads, beside its package's: NuGet's registration
2+-- and search name them. Counted the way the package's are, approximately.
3+ALTER TABLE versions ADD COLUMN downloads INTEGER NOT NULL DEFAULT 0;
4+
5+-- The NuGet symbol server finds a PDB by the name a version keeps it under
6+-- (`pdb:<file>:<key>`), across a workspace's packages.
7+CREATE INDEX version_files_name ON version_files (name);
+41−3
188188 /// Cargo: 1 when the version is yanked.
189189 #[serde(default)]
190190 pub yanked: u32,
191+ /// Its own downloads, counted for NuGet's.
192+ #[serde(default)]
193+ pub downloads: u64,
191194 }
192195
193196 impl VersionRow {
249252 pub media_type: Option<String>,
250253 }
251254
255+/// A file found by its name, and the package that keeps it.
256+#[derive(Clone, Debug, Deserialize)]
257+pub struct NamedFile {
258+ pub package_id: String,
259+ pub digest: String,
260+ pub size: u64,
261+}
262+
252263 /// A file's other checksums, in hex, beside its SHA-256 digest.
253264 #[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
254265 pub struct Checksums {
293304 "id, workspace, ecosystem, name, repo_id, repo_name, visibility, description, created_by, created_at, updated_at, downloads, workspace_deleted_at";
294305 /// Workspaces that are deleted, waiting to be purged or restored.
295306 const DELETED_WORKSPACES: &str = "SELECT workspace FROM packages WHERE workspace_deleted_at IS NOT NULL";
296−const VERSION_COLUMNS: &str = "id, package_id, version, digest, size, metadata, subject, published_by, published_at, deprecated, yanked";
307+const VERSION_COLUMNS: &str = "id, package_id, version, digest, size, metadata, subject, published_by, published_at, deprecated, yanked, downloads";
297308
298309 pub struct Db {
299310 pub db: D1Database,
457468 Ok(())
458469 }
459470
460− pub async fn add_downloads(&self, counts: &[(String, u64)]) -> Result<()> {
471+ /// Adds downloads to packages, and to the versions named with them.
472+ pub async fn add_downloads(&self, counts: &[((String, Option<String>), u64)]) -> Result<()> {
461473 if counts.is_empty() {
462474 return Ok(());
463475 }
464476 let mut batch = Vec::with_capacity(counts.len());
465− for (id, count) in counts {
477+ for ((id, version), count) in counts {
466478 batch.push(self.prepare("UPDATE packages SET downloads = downloads + ? WHERE id = ?", &[num(*count), text(id)])?);
479+ if let Some(version) = version {
480+ batch.push(self.prepare("UPDATE versions SET downloads = downloads + ? WHERE id = ?", &[num(*count), text(version)])?);
481+ }
467482 }
468483 self.db.batch(batch).await?;
469484 Ok(())
11921207 .results()
11931208 }
11941209
1210+ pub async fn package_by_id(&self, package_id: &str) -> Result<Option<PackageRow>> {
1211+ self.prepare(&format!("SELECT {PACKAGE_COLUMNS} FROM packages WHERE id = ?"), &[text(package_id)])?
1212+ .first(None)
1213+ .await
1214+ }
1215+
1216+ /// The files a workspace's packages of an ecosystem keep under `name`,
1217+ /// newest first: how the NuGet symbol server finds a PDB.
1218+ pub async fn files_named(&self, workspace: &str, ecosystem: &str, name: &str, limit: u32) -> Result<Vec<NamedFile>> {
1219+ self.prepare(
1220+ &format!(
1221+ "SELECT v.package_id, f.digest, f.size FROM version_files f
1222+ JOIN versions v ON v.id = f.version_id JOIN packages p ON p.id = v.package_id
1223+ WHERE f.name = ? AND p.workspace = ? AND p.ecosystem = ? AND p.workspace_deleted_at IS NULL
1224+ ORDER BY v.published_at DESC LIMIT {limit}"
1225+ ),
1226+ &[text(name), text(workspace), text(ecosystem)],
1227+ )?
1228+ .all()
1229+ .await?
1230+ .results()
1231+ }
1232+
11951233 /// A workspace's Maven artifacts of one groupId (`com.acme:*`), by
11961234 /// name: those named from `com.acme:` up to `com.acme;`, the
11971235 /// character after `:`.
+17−6
6363 const SWEEP_BATCH: u32 = 200;
6464
6565 thread_local! {
66− /// Pulls counted since the last write, by package: written at most
67− /// every few seconds, so a busy image costs one write, not one a pull.
68− /// What an isolate holds when it goes away is lost: the count is
69− /// approximate.
70− static DOWNLOADS: RefCell<(HashMap<String, u64>, u64)> = RefCell::new((HashMap::new(), 0));
66+ /// Pulls counted since the last write, by package (and by version,
67+ /// where it is counted too): written at most every few seconds, so a
68+ /// busy image costs one write, not one a pull. What an isolate holds
69+ /// when it goes away is lost: the count is approximate.
70+ static DOWNLOADS: RefCell<(HashMap<(String, Option<String>), u64>, u64)> = RefCell::new((HashMap::new(), 0));
7171 }
7272 const DOWNLOADS_FLUSH_MS: u64 = 10_000;
7373
286286 }
287287
288288 fn count_download(&self, package_id: &str, ctx: &Context) {
289+ self.count_downloads(package_id, None, ctx);
290+ }
291+
292+ /// A download of one version, counted for it and its package.
293+ fn count_version_download(&self, package_id: &str, version_id: &str, ctx: &Context) {
294+ self.count_downloads(package_id, Some(version_id), ctx);
295+ }
296+
297+ fn count_downloads(&self, package_id: &str, version_id: Option<&str>, ctx: &Context) {
289298 let due = DOWNLOADS.with(|counts| {
290299 let mut counts = counts.borrow_mut();
291− *counts.0.entry(package_id.to_owned()).or_default() += 1;
300+ *counts.0.entry((package_id.to_owned(), version_id.map(str::to_owned))).or_default() += 1;
292301 let now = now_ms();
293302 if now.saturating_sub(counts.1) < DOWNLOADS_FLUSH_MS {
294303 return None;
457466 } else {
458467 version.deprecated
459468 },
469+ symbols: meta["symbols"] == true,
470+ downloads: (row.package.ecosystem == "nuget").then_some(version.downloads),
460471 }
461472 })
462473 .collect();
+205−4
11 //! What the NuGet feed needs that does not touch the network: package ids
22 //! and NuGet's normalized versions, the feed's paths, the `.nuspec` read
3−//! from a `.nupkg` (a zip), the multipart body `dotnet nuget push` sends,
4−//! and the service index, registration and search documents of the v3
5−//! protocol.
3+//! from a `.nupkg` (a zip), the portable PDBs read from a `.snupkg` and
4+//! the keys the symbol server finds them by, the multipart body `dotnet
5+//! nuget push` sends, and the service index, registration and search
6+//! documents of the v3 protocol.
67 //!
78 //! A version keeps what the documents need from its `.nuspec` as its
89 //! metadata, made once when it is pushed. Unlisting (`dotnet nuget
121122 pub enum Content {
122123 Nupkg,
123124 Nuspec,
125+ /// The symbol package, when one was pushed.
126+ Snupkg,
124127 }
125128
129+impl Content {
130+ /// The name the version keeps the file by.
131+ pub fn file(self) -> &'static str {
132+ match self {
133+ Content::Nupkg => "nupkg",
134+ Content::Nuspec => "nuspec",
135+ Content::Snupkg => "snupkg",
136+ }
137+ }
138+}
139+
126140 /// One of the feed's endpoints, under `/-/nuget/<workspace>/`.
127141 #[derive(Clone, Debug, PartialEq, Eq)]
128142 pub enum NugetRoute {
142156 Push,
143157 /// `api/v2/package/<id>/<version>`: `DELETE` unlists, `POST` lists again.
144158 Listing { id: String, version: String },
159+ /// `api/v2/symbolpackage`: `dotnet nuget push` of a `.snupkg`.
160+ SymbolPush,
161+ /// `symbols/<file>.pdb/<key>/<file>.pdb`: a PDB from the symbol server,
162+ /// by the key a debugger asks with; both lowercased.
163+ Symbol { file: String, key: String },
145164 }
146165
147166 /// The workspace and endpoint a path is. Ids are checked; versions are
163182 let lower = format!("{}.{}", name.to_ascii_lowercase(), version.to_ascii_lowercase());
164183 let file = if file.eq_ignore_ascii_case(&format!("{lower}.nupkg")) {
165184 Content::Nupkg
185+ } else if file.eq_ignore_ascii_case(&format!("{lower}.snupkg")) {
186+ Content::Snupkg
166187 } else if file.eq_ignore_ascii_case(&format!("{name}.nuspec")) {
167188 Content::Nuspec
168189 } else {
174195 ["v3", "registration", name, leaf] => NugetRoute::Leaf { id: id(name)?, version: leaf.strip_suffix(".json")?.to_owned() },
175196 ["api", "v2", "package"] => NugetRoute::Push,
176197 ["api", "v2", "package", name, version] => NugetRoute::Listing { id: id(name)?, version: (*version).to_owned() },
198+ ["api", "v2", "symbolpackage"] => NugetRoute::SymbolPush,
199+ ["symbols", file, key, again] if file.eq_ignore_ascii_case(again) && valid_pdb_name(file) && valid_key(key) => {
200+ NugetRoute::Symbol { file: file.to_ascii_lowercase(), key: key.to_ascii_lowercase() }
201+ }
177202 _ => return None,
178203 };
179204 Some((workspace, route))
180205 }
181206
207+/// A PDB's file name, as a symbol server path holds it: no folders.
208+fn valid_pdb_name(file: &str) -> bool {
209+ file.len() <= 255
210+ && file.to_ascii_lowercase().ends_with(".pdb")
211+ && file.len() > 4
212+ && file.bytes().all(|b| b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_' | b'+'))
213+}
214+
215+/// A symbol server key: hex, as `<guid><age>` is.
216+fn valid_key(key: &str) -> bool {
217+ (1..=64).contains(&key.len()) && key.bytes().all(|b| b.is_ascii_hexdigit())
218+}
219+
220+/// The 20-byte id of a portable PDB (`#Pdb` stream's first bytes: a GUID
221+/// and a stamp), which the assembly built with it names too. `None` for a
222+/// file that is not a portable PDB (a Windows PDB, say).
223+pub fn pdb_id(bytes: &[u8]) -> Option<[u8; 20]> {
224+ let u16_at = |at: usize| Some(u16::from_le_bytes(bytes.get(at..at + 2)?.try_into().ok()?));
225+ let u32_at = |at: usize| Some(u32::from_le_bytes(bytes.get(at..at + 4)?.try_into().ok()?));
226+ // ECMA-335 II.24.2.1: the metadata root, its version string, then
227+ // each stream's offset, size and name, padded to four bytes.
228+ if u32_at(0)? != 0x424A_5342 {
229+ return None;
230+ }
231+ let length = u32_at(12)? as usize;
232+ let mut at = 16usize.checked_add(length)?;
233+ let streams = u16_at(at + 2)?;
234+ at += 4;
235+ for _ in 0..streams {
236+ let (offset, size) = (u32_at(at)? as usize, u32_at(at + 4)? as usize);
237+ let name_start = at + 8;
238+ let name_len = bytes.get(name_start..)?.iter().take(32).position(|b| *b == 0)?;
239+ let name = &bytes[name_start..name_start + name_len];
240+ at = name_start + (name_len + 1).div_ceil(4) * 4;
241+ if name == b"#Pdb" && size >= 20 {
242+ return bytes.get(offset..offset + 20)?.try_into().ok();
243+ }
244+ }
245+ None
246+}
247+
248+/// The key a symbol server finds a portable PDB by: its GUID as .NET
249+/// writes it (`Guid.ToString("N")`: the first three fields byte-swapped)
250+/// and `ffffffff` for its age, lowercased.
251+pub fn symbol_key(id: &[u8; 20]) -> String {
252+ let mut guid = Vec::with_capacity(16);
253+ guid.extend(id[..4].iter().rev());
254+ guid.extend(id[4..6].iter().rev());
255+ guid.extend(id[6..8].iter().rev());
256+ guid.extend(&id[8..16]);
257+ format!("{}ffffffff", hex::encode(guid))
258+}
259+
260+/// The name a version keeps a PDB by: `pdb:<file>:<key>`, lowercased, as
261+/// the symbol server looks it up.
262+pub fn symbol_file(file: &str, key: &str) -> String {
263+ format!("pdb:{}:{}", file.to_ascii_lowercase(), key.to_ascii_lowercase())
264+}
265+
266+/// One PDB from a symbol package: its file name and key, and its bytes.
267+#[derive(Debug)]
268+pub struct Pdb {
269+ pub file: String,
270+ pub key: String,
271+ pub bytes: Vec<u8>,
272+}
273+
274+/// What a `.snupkg` holds: its `.nuspec`, which names the package and
275+/// version it is for, and its portable PDBs.
276+#[derive(Debug)]
277+pub struct Symbols {
278+ pub nuspec: Nuspec,
279+ pub pdbs: Vec<Pdb>,
280+}
281+
282+/// The largest PDB read from a symbol package.
283+const MAX_PDB_BYTES: usize = 64 * 1024 * 1024;
284+
285+/// Reads a `.snupkg`: a zip with a `.nuspec` of the `SymbolsPackage`
286+/// type, and one or more portable PDBs.
287+pub fn read_symbols(snupkg: &[u8]) -> Result<Symbols, String> {
288+ let package = read_package(snupkg)?;
289+ if !package.nuspec.package_types.iter().any(|t| t.eq_ignore_ascii_case("SymbolsPackage")) {
290+ return Err("The .nuspec does not say it is a symbol package (<packageType name=\"SymbolsPackage\" />). Build it with SymbolPackageFormat snupkg.".to_owned());
291+ }
292+ let entries = archive::zip_entries(snupkg)?;
293+ let mut pdbs = Vec::new();
294+ for entry in entries.iter().filter(|e| e.name.to_ascii_lowercase().ends_with(".pdb")) {
295+ let file = entry.name.rsplit(['/', '\\']).next().unwrap_or(&entry.name).to_owned();
296+ let bytes = archive::zip_read(snupkg, entry, MAX_PDB_BYTES)?;
297+ let Some(id) = pdb_id(&bytes) else {
298+ return Err(format!("{} is not a portable PDB. Build with DebugType portable (the default).", entry.name));
299+ };
300+ pdbs.push(Pdb { file, key: symbol_key(&id), bytes });
301+ }
302+ if pdbs.is_empty() {
303+ return Err("The symbol package holds no .pdb files.".to_owned());
304+ }
305+ Ok(Symbols { nuspec: package.nuspec, pdbs })
306+}
307+
182308 /// The `.nupkg` file in a `multipart/form-data` body, as `dotnet nuget
183309 /// push` sends it; a body that is not multipart is taken as the file.
184310 pub fn pushed_file<'a>(content_type: Option<&str>, body: &'a [u8]) -> Result<&'a [u8], String> {
228354 pub readme: Option<String>,
229355 pub require_license_acceptance: bool,
230356 pub groups: Vec<Group>,
357+ /// `<packageTypes>`: `SymbolsPackage` for a `.snupkg`.
358+ pub package_types: Vec<String>,
231359 }
232360
233361 /// A dependency's `version` as a range: `1.0` (at least 1.0) is
277405 readme: metadata.child_text("readme"),
278406 require_license_acceptance: metadata.child_text("requireLicenseAcceptance").is_some_and(|v| v.eq_ignore_ascii_case("true")),
279407 groups,
408+ package_types: metadata
409+ .child("packageTypes")
410+ .map(|types| types.children_named("packageType").filter_map(|t| t.attribute("name")).map(str::to_owned).collect())
411+ .unwrap_or_default(),
280412 })
281413 }
282414
353485 resource(query.clone(), "SearchQueryService/3.0.0-beta"),
354486 resource(query, "SearchQueryService/3.5.0"),
355487 resource(format!("{base}/api/v2/package"), "PackagePublish/2.0.0"),
488+ resource(format!("{base}/api/v2/symbolpackage"), "SymbolPackagePublish/4.9.0"),
356489 ],
357490 })
358491 }
429562 "listed": listed.listed,
430563 "published": listed.published,
431564 "packageContent": at.content,
565+ "downloads": listed.downloads,
432566 },
433567 "packageContent": at.content,
434568 "registration": at.registration,
536670 assert_eq!(route("/-/nuget/acme/api/v2/package/"), at(NugetRoute::Push));
537671 assert_eq!(route("/-/nuget/acme/api/v2/package/Acme.Web/1.0.0"), at(NugetRoute::Listing { id: "Acme.Web".into(), version: "1.0.0".into() }));
538672 assert_eq!(route("/-/nuget/acme/v3/flatcontainer/a..b/index.json"), None);
673+ assert_eq!(
674+ route("/-/nuget/acme/v3/flatcontainer/acme.web/1.0.0/acme.web.1.0.0.snupkg"),
675+ at(NugetRoute::Content { id: "acme.web".into(), version: "1.0.0".into(), file: Content::Snupkg })
676+ );
677+ assert_eq!(route("/-/nuget/acme/api/v2/symbolpackage"), at(NugetRoute::SymbolPush));
678+ assert_eq!(
679+ route("/-/nuget/acme/symbols/Acme.Web.pdb/0A1B2C3D4E5F60718293A4B5C6D7E8F9ffffffff/acme.web.pdb"),
680+ at(NugetRoute::Symbol { file: "acme.web.pdb".into(), key: "0a1b2c3d4e5f60718293a4b5c6d7e8f9ffffffff".into() })
681+ );
682+ assert_eq!(route("/-/nuget/acme/symbols/a.pdb/xyz/a.pdb"), None, "not hex");
683+ assert_eq!(route("/-/nuget/acme/symbols/a.pdb/00/b.pdb"), None, "two names");
684+ assert_eq!(route("/-/nuget/acme/symbols/a.dll/00/a.dll"), None, "only PDBs");
539685 assert_eq!(route("/-/nuget/acme"), None);
540686 assert_eq!(route("/-/nuget/acme/v2"), None);
541687 }
599745 fn the_documents_are_nugets_shape() {
600746 let index = service_index("https://g1t.sh/-/nuget/acme");
601747 let kinds: Vec<&str> = index["resources"].as_array().unwrap().iter().map(|r| r["@type"].as_str().unwrap()).collect();
602− for kind in ["PackageBaseAddress/3.0.0", "RegistrationsBaseUrl", "SearchQueryService", "PackagePublish/2.0.0"] {
748+ for kind in ["PackageBaseAddress/3.0.0", "RegistrationsBaseUrl", "SearchQueryService", "PackagePublish/2.0.0", "SymbolPackagePublish/4.9.0"] {
603749 assert!(kinds.contains(&kind), "{kind}");
604750 }
605751 let spec = read_nuspec(NUSPEC).unwrap();
620766 assert_eq!(entry["dependencyGroups"][0]["targetFramework"], "net8.0");
621767 assert_eq!(entry["dependencyGroups"][0]["dependencies"][1]["range"], "[1.0.0, 2.0.0)");
622768 assert_eq!(page["items"][0]["catalogEntry"]["listed"], false);
769+ assert_eq!(page["items"][0]["catalogEntry"]["downloads"], 3, "each version's own");
623770 let found = search_result(base, "Acme.Web", &versions).unwrap();
624771 assert_eq!(found["version"], "1.2.0");
625772 assert_eq!(found["versions"].as_array().unwrap().len(), 1, "unlisted versions are not searched");
627774 assert_eq!(found["authors"], json!(["Ada", "Bo"]));
628775 assert!(search_result(base, "Acme.Web", &versions[..1]).is_none());
629776 }
777+
778+ /// A portable PDB's start: the metadata root, a version string, and
779+ /// two streams, `#Pdb` holding the id.
780+ fn portable_pdb(id: &[u8; 20]) -> Vec<u8> {
781+ let version = b"PDB v1.0\0\0\0\0";
782+ let mut pdb = Vec::new();
783+ pdb.extend_from_slice(&0x424A_5342u32.to_le_bytes());
784+ pdb.extend_from_slice(&[1, 0, 1, 0, 0, 0, 0, 0]);
785+ pdb.extend_from_slice(&(version.len() as u32).to_le_bytes());
786+ pdb.extend_from_slice(version);
787+ pdb.extend_from_slice(&[0, 0, 2, 0]);
788+ // Each stream: offset, size, and its name padded to four bytes.
789+ pdb.extend_from_slice(&84u32.to_le_bytes());
790+ pdb.extend_from_slice(&16u32.to_le_bytes());
791+ pdb.extend_from_slice(b"#GUID\0\0\0");
792+ pdb.extend_from_slice(&64u32.to_le_bytes());
793+ pdb.extend_from_slice(&20u32.to_le_bytes());
794+ pdb.extend_from_slice(b"#Pdb\0\0\0\0");
795+ assert_eq!(pdb.len(), 64);
796+ pdb.extend_from_slice(id);
797+ pdb.extend_from_slice(&[0; 16]);
798+ pdb
799+ }
800+
801+ #[test]
802+ fn a_portable_pdb_is_found_by_its_guid() {
803+ // The GUID 3d2c1b0a-5f4e-7160-8293-a4b5c6d7e8f9, as .NET lays it
804+ // out in bytes, and a stamp.
805+ let mut id = [0u8; 20];
806+ id[..16].copy_from_slice(&[0x0a, 0x1b, 0x2c, 0x3d, 0x4e, 0x5f, 0x60, 0x71, 0x82, 0x93, 0xa4, 0xb5, 0xc6, 0xd7, 0xe8, 0xf9]);
807+ id[16..].copy_from_slice(&[1, 2, 3, 4]);
808+ let pdb = portable_pdb(&id);
809+ assert_eq!(pdb_id(&pdb), Some(id));
810+ assert_eq!(symbol_key(&id), "3d2c1b0a5f4e71608293a4b5c6d7e8f9ffffffff");
811+ assert_eq!(pdb_id(b"Microsoft C/C++ MSF 7.00\r\n"), None, "a Windows PDB");
812+ assert_eq!(symbol_file("Acme.Web.pdb", "ABC"), "pdb:acme.web.pdb:abc");
813+
814+ let nuspec = r#"<package><metadata><id>Acme.Web</id><version>1.0.0</version><packageTypes><packageType name="SymbolsPackage" /></packageTypes></metadata></package>"#;
815+ let snupkg = crate::composer::zip(&[
816+ ("Acme.Web.nuspec".to_owned(), nuspec.as_bytes().to_vec()),
817+ ("lib/net8.0/Acme.Web.pdb".to_owned(), pdb.clone()),
818+ ]);
819+ let symbols = read_symbols(&snupkg).unwrap();
820+ assert_eq!(symbols.nuspec.id, "Acme.Web");
821+ assert_eq!(symbols.pdbs.len(), 1);
822+ assert_eq!((symbols.pdbs[0].file.as_str(), symbols.pdbs[0].key.as_str()), ("Acme.Web.pdb", symbol_key(&id).as_str()));
823+ let plain = crate::composer::zip(&[("Acme.Web.nuspec".to_owned(), NUSPEC.as_bytes().to_vec()), ("lib/a.pdb".to_owned(), pdb)]);
824+ assert!(read_symbols(&plain).is_err(), "not a symbol package");
825+ let windows = crate::composer::zip(&[
826+ ("Acme.Web.nuspec".to_owned(), nuspec.as_bytes().to_vec()),
827+ ("lib/a.pdb".to_owned(), b"Microsoft C/C++ MSF 7.00\r\n".to_vec()),
828+ ]);
829+ assert!(read_symbols(&windows).unwrap_err().contains("portable"));
830+ }
630831 }
+126−10
77 //! A `.nupkg` is stored once, by its SHA-256, with its `.nuspec` beside it;
88 //! the flat container, registration and search documents are made from the
99 //! versions on each read. `dotnet nuget delete` unlists a version, as
10−//! nuget.org does: it is still downloaded by those who name it.
10+//! nuget.org does: it is still downloaded by those who name it. Each
11+//! `.nupkg` download counts for its version as well as its package.
12+//!
13+//! A symbol package (`.snupkg`, pushed to `api/v2/symbolpackage` after its
14+//! `.nupkg`) is kept beside the version, and each portable PDB in it by
15+//! the key debuggers ask the symbol server (`symbols/`) with, as the
16+//! Simple Symbol Query Protocol names it: `<file>/<guid>ffffffff/<file>`.
1117
1218 use g1t_contracts::User;
1319 use g1t_contracts::audit::AuditActor;
130136 NugetRoute::Push if method == Method::Put => self.nuget_push(&mut request, workspace, viewer).await,
131137 NugetRoute::Listing { id, version } if method == Method::Delete => self.nuget_listing(workspace, &id, &version, false, viewer).await,
132138 NugetRoute::Listing { id, version } if method == Method::Post => self.nuget_listing(workspace, &id, &version, true, viewer).await,
139+ NugetRoute::SymbolPush if method == Method::Put => self.nuget_symbol_push(&mut request, workspace, viewer).await,
140+ NugetRoute::Symbol { file, key } if read => self.nuget_symbol(workspace, &file, &key, viewer, head).await,
133141 _ => error(405, "Not a method this address takes."),
134142 }
135143 }
202210 let Some(row) = versions.iter().find(|v| v.version.to_ascii_lowercase() == wanted) else {
203211 return self.nuget_absent(workspace, viewer).await;
204212 };
205− let name = match file {
206− Content::Nupkg => "nupkg",
207− Content::Nuspec => "nuspec",
208− };
209− let Some(kept) = self.db.file(&row.id, name).await? else {
213+ let Some(kept) = self.db.file(&row.id, file.file()).await? else {
210214 return self.nuget_absent(workspace, viewer).await;
211215 };
212216 let Some(digest) = Digest::parse(&kept.digest) else {
216220 return self.nuget_absent(workspace, viewer).await;
217221 };
218222 let headers = Headers::new();
219− headers.set("content-type", if file == Content::Nupkg { "application/octet-stream" } else { "application/xml" })?;
223+ headers.set("content-type", if file == Content::Nuspec { "application/xml" } else { "application/octet-stream" })?;
220224 headers.set("content-length", &blob.size.to_string())?;
221225 headers.set("cache-control", "max-age=31536000")?;
222226 if head {
226230 return self.nuget_absent(workspace, viewer).await;
227231 };
228232 if file == Content::Nupkg {
229− self.count_download(&package.id, ctx);
233+ self.count_version_download(&package.id, &row.id, ctx);
230234 }
231235 Ok(Response::from_body(got.body)?.with_headers(headers))
232236 }
241245 let listed: Vec<Listed<'_>> = versions
242246 .iter()
243247 .zip(&metadata)
244− .map(|(row, metadata)| Listed { version: &row.version, metadata, published: &row.published_at, listed: !row.is_yanked(), downloads: 0 })
248+ .map(|(row, metadata)| Listed { version: &row.version, metadata, published: &row.published_at, listed: !row.is_yanked(), downloads: row.downloads })
245249 .collect();
246250 match version {
247251 None => json_response(&nuget::registration(base, &package.name, &listed), head),
287291 let mut listed: Vec<Listed<'_>> = rows
288292 .iter()
289293 .zip(&metadata)
290− .map(|(row, metadata)| Listed { version: &row.version, metadata, published: &row.published_at, listed: !row.is_yanked(), downloads: 0 })
294+ .map(|(row, metadata)| Listed { version: &row.version, metadata, published: &row.published_at, listed: !row.is_yanked(), downloads: row.downloads })
291295 .collect();
292296 listed.sort_by(|a, b| nuget::compare(a.version, b.version));
293297 if let Some(mut result) = nuget::search_result(base, &package.name, &listed) {
470474 error(201, format!("{} {version} was pushed.", package.name))
471475 }
472476
477+ /// `dotnet nuget push` of a `.snupkg`, which it sends after the
478+ /// `.nupkg` beside it: the symbols of a version already pushed, kept
479+ /// with it, and each portable PDB in it kept by its symbol server key.
480+ async fn nuget_symbol_push(&self, request: &mut Request, workspace: &str, viewer: Option<&User>) -> Result<Response> {
481+ let declared = request.headers().get("content-length")?.and_then(|n| n.parse::<u64>().ok());
482+ let too_large = || {
483+ let mb = self.max_request / 1_000_000;
484+ error(413, format!("A push may be at most {mb} MB. See {DOCS}#size"))
485+ };
486+ if declared.is_some_and(|n| n > self.max_request) {
487+ return too_large();
488+ }
489+ let content_type = request.headers().get("content-type")?;
490+ let body = request.bytes().await?;
491+ if body.len() as u64 > self.max_request {
492+ return too_large();
493+ }
494+ if viewer.is_none() {
495+ return error(401, format!("Push with a g1t access token as the API key: dotnet nuget push <file> --api-key <token>. Make one at {TOKENS}."));
496+ }
497+ let snupkg = match nuget::pushed_file(content_type.as_deref(), &body) {
498+ Ok(file) => file,
499+ Err(message) => return error(400, message),
500+ };
501+ let symbols = match nuget::read_symbols(snupkg) {
502+ Ok(symbols) => symbols,
503+ Err(message) => return error(400, message),
504+ };
505+ let spec = &symbols.nuspec;
506+ let Some(version) = nuget::normalize(&spec.version) else {
507+ return error(400, format!("{} is not a version NuGet reads.", spec.version));
508+ };
509+ let push_first = || error(404, format!("Push {} {version} before its symbols: dotnet nuget push pushes the .snupkg beside a .nupkg after it.", spec.id));
510+ let Some(package) = self.nuget_package(workspace, &spec.id).await? else {
511+ return push_first();
512+ };
513+ if let Some(refusal) = self.nuget_check(viewer, &package, Action::Push).await? {
514+ return Ok(refusal);
515+ }
516+ let versions = self.db.versions(&package.id, MAX_VERSIONS).await?;
517+ let Some(row) = versions.iter().find(|v| v.version.eq_ignore_ascii_case(&version)) else {
518+ return push_first();
519+ };
520+ let snupkg = snupkg.to_vec();
521+ let digest = Digest::of(&snupkg);
522+ if let Some(kept) = self.db.file(&row.id, Content::Snupkg.file()).await? {
523+ if kept.digest == digest.to_string() {
524+ return error(201, format!("The symbols of {} {} were pushed.", package.name, row.version));
525+ }
526+ return error(409, format!("{} {} already has symbols, and a version's symbols are pushed once. Bump the version.", package.name, row.version));
527+ }
528+ let mut files = vec![(Content::Snupkg.file().to_owned(), digest.clone(), snupkg)];
529+ for pdb in symbols.pdbs {
530+ let name = nuget::symbol_file(&pdb.file, &pdb.key);
531+ if files.iter().all(|(kept, _, _)| *kept != name) {
532+ files.push((name, Digest::of(&pdb.bytes), pdb.bytes));
533+ }
534+ }
535+ let sizes: Vec<(String, u64)> = files.iter().map(|(_, d, bytes)| (d.to_string(), bytes.len() as u64)).collect();
536+ if let Some(refusal) = self.storage_refusal(&package, &sizes).await? {
537+ return error(403, refusal);
538+ }
539+ let now = now_ms();
540+ for (name, digest, bytes) in files {
541+ let size = bytes.len() as u64;
542+ let stored = match self.db.blob(&digest).await? {
543+ Some(blob) => self.store.head(&blob.object_key).await?.is_some(),
544+ None => false,
545+ };
546+ if !stored {
547+ self.store.put(&digest.object_key(), bytes).await?;
548+ }
549+ self.db.keep_blob(&package.id, &digest, size, Some("application/octet-stream"), &digest.object_key(), now).await?;
550+ let file = NewFile { name, digest: digest.to_string(), size, media_type: Some("application/octet-stream".to_owned()) };
551+ self.db.put_file(&package.id, &row.id, &file, now).await?;
552+ }
553+ let mut metadata = row.meta();
554+ if metadata.is_object() {
555+ metadata["symbols"] = json!(true);
556+ self.db.set_version(&row.id, &row.digest, &metadata.to_string()).await?;
557+ }
558+ self.db.measure(&package.workspace).await?;
559+ let caller = Caller { actor: viewer.map(AuditActor::of) };
560+ self.audit(&caller, "package.publish_symbols", &package, Some(&format!("{workspace}/{}@{}", package.name, row.version)), None).await;
561+ error(201, format!("The symbols of {} {} were pushed.", package.name, row.version))
562+ }
563+
564+ /// The symbol server: a PDB by its file name and key, from a package
565+ /// of the workspace the viewer may read.
566+ async fn nuget_symbol(&self, workspace: &str, file: &str, key: &str, viewer: Option<&User>, head: bool) -> Result<Response> {
567+ for found in self.db.files_named(workspace, NUGET, &nuget::symbol_file(file, key), 10).await? {
568+ let Some(package) = self.db.package_by_id(&found.package_id).await?.filter(|p| !p.hidden()) else {
569+ continue;
570+ };
571+ if !access::decide(viewer, &TargetOf::package(&package).view(), Action::Pull).allowed {
572+ continue;
573+ }
574+ let Some(digest) = Digest::parse(&found.digest) else { continue };
575+ let Some(blob) = self.db.package_blob(&package.id, &digest).await? else { continue };
576+ let headers = Headers::new();
577+ headers.set("content-type", "application/octet-stream")?;
578+ headers.set("content-length", &blob.size.to_string())?;
579+ headers.set("cache-control", "max-age=31536000")?;
580+ if head {
581+ return Ok(Response::from_body(ResponseBody::Empty)?.with_headers(headers));
582+ }
583+ let Some(got) = self.store.get(&blob.object_key, None).await? else { continue };
584+ return Ok(Response::from_body(got.body)?.with_headers(headers));
585+ }
586+ self.nuget_absent(workspace, viewer).await
587+ }
588+
473589 /// `dotnet nuget delete` unlists a version; a `POST` lists it again.
474590 async fn nuget_listing(&self, workspace: &str, id: &str, version: &str, listed: bool, viewer: Option<&User>) -> Result<Response> {
475591 let Some(package) = self.nuget_package(workspace, id).await? else {