Skip to content

Commit

The artifacts plan records what Phase 1 decided, self-hosting and the deploy list name the g1t-folios index and the trash cron, and folio events are listed as published but never offered to webhooks.

syntaqxcommitted Parent88821c2Browse files
6 files+84−100/6 viewed
+3−3
663663
664664 // --- Events -------------------------------------------------------------------
665665
666−/// The `folio.*` types the docs service will publish (Phase 1), with no
667−/// `repoId` on the event: a folio may be private, so it never reaches a
668−/// repository's timeline or webhooks. Nothing subscribes to them yet.
666+/// The `folio.*` types the docs service publishes, with no `repoId` on
667+/// the event: a folio may be private, so it never reaches a repository's
668+/// timeline or webhooks. Nothing subscribes to them yet.
669669 pub const FOLIO_EVENTS: [&str; 6] = ["folio.created", "folio.updated", "folio.trashed", "folio.restored", "folio.shared", "folio.stale"];
670670
671671 /// What every `folio.*` event carries (`FolioEventData` in events.ts).
+8−1
232232 use super::*;
233233
234234 /// Types published on the bus that hooks are not offered.
235− const UNOFFERED: [&str; 22] = [
235+ const UNOFFERED: [&str; 28] = [
236236 "pull.mergecheck",
237237 "pull.mergeability",
238238 "deployment.review_requested",
256256 "doc.page.updated",
257257 "doc.page.archived",
258258 "doc.page.stale",
259+ // Artifacts (folios): no repository either, and a folio may be private.
260+ "folio.created",
261+ "folio.updated",
262+ "folio.trashed",
263+ "folio.restored",
264+ "folio.shared",
265+ "folio.stale",
259266 ];
260267
261268 fn published(kind: &str) -> bool {
+5−3
127127 ],
128128 "self_host": "run"
129129 },
130− // Docs mode's service. Its Worker is g1t-docs-service: g1t-docs is
131− // the documentation site (apps/docs).
130+ // Docs mode's service, which also hosts artifacts (folios,
131+ // docs/ARTIFACTS_MODE.md). Its Worker is g1t-docs-service: g1t-docs
132+ // is the documentation site (apps/docs).
132133 "docs-service": {
133134 "path": "services/docs",
134135 "kind": "ts-worker",
140141 "The D1 database, before the first deploy: npx wrangler d1 create g1t-docs, then put its id in services/docs/wrangler.jsonc",
141142 "The R2 bucket for files in pages: npx wrangler r2 bucket create g1t-docs-files",
142143 "The queue the events service sends it merges and pushes on (pages whose cited code changed, projects' docs): npx wrangler queues create g1t-events-docs. The service also sends its own backfill jobs to it (JOBS)",
143− "The Vectorize index agents recall Docs from: npx wrangler vectorize create g1t-docs --dimensions=768 --metric=cosine, with string metadata indexes on workspace_id and space_id (npx wrangler vectorize create-metadata-index g1t-docs --property-name=<name> --type=string)"
144+ "The Vectorize index agents recall Docs from: npx wrangler vectorize create g1t-docs --dimensions=768 --metric=cosine, with string metadata indexes on workspace_id and space_id (npx wrangler vectorize create-metadata-index g1t-docs --property-name=<name> --type=string)",
145+ "The Vectorize index for artifacts (folios), before the first deploy with FOLIO_VECTORS: npx wrangler vectorize create g1t-folios --dimensions=768 --metric=cosine, with string metadata indexes on workspace_id, scope and kind (npx wrangler vectorize create-metadata-index g1t-folios --property-name=<name> --type=string). Without it, artifacts are searched and recalled by words"
144146 ],
145147 "self_host": "run"
146148 },
+58−0
818818 - Contracts files import only types from each other, because Node runs the tests on the files as they are. So `dashboardOpError` takes the query validator (`datasetQueryError`) as an argument.
819819 - **Ids.** `fol_` for folios and `prp_` for proposals. Versions, templates and files keep `ver_`, `tpl_` and `fil_`.
820820 - **Slides themes** are a slug plus an optional `#rrggbb` accent. Phase 4 names the themes.
821+
822+---
823+
824+## 11. Decided in Phase 1
825+
826+Phase 1 shipped the service core and the doc kind in `services/docs` (migration `0004_folios.sql`, `src/folios/**`, `src/kinds/**`, `src/who.ts`). Old Docs keeps running on its own tables. Where the plan was open, the build settled these:
827+
828+- **Where the code lives.**
829+ - `FolioRoom` is a new file, `src/folios/room.ts`, beside an untouched `src/room.ts` (`PageRoom`). Both share `ROOM_MEMBER_HEADER` and `awarenessEntries`.
830+ - The doc kind (`src/kinds/doc/`) uses `blocks.ts`, `markdown.ts`, `edits.ts`, `citations.ts` and `threads.ts` where they are, because `PageRoom` still needs them. They move under `src/kinds/doc/` in Phase 7.
831+ - `src/who.ts` is a copy of the legacy `Docs` class's workspace, people, teams and space helpers, since `index.ts` only gets appended to in this phase. The legacy copy goes in Phase 7.
832+ - `index.ts` sends `/rpc/<method>` to `src/folios/rpc.ts` first. A test checks that table against `FOLIO_RPC_METHODS`. `/live?folio=` and `PUT /files?folio=` are folios'. `GET /files/<key>` looks in `folio_files` first, then `files`.
833+- **Access.**
834+ - The owner of an ancestor that a folio inherits from counts as a `manage` grant there (`folio_access.via` is that ancestor). So whoever owns a doc keeps full access to what others add under it, unless a restriction cuts it off.
835+ - Workspace owners get `manage` only through a space the folio inherits: open and team spaces, as for pages. A restricted folio inside an open space is "Only people invited", for owners too.
836+ - General access is set only on an access root: a top-level folio, or a restricted one. On a folio that inherits, the call is refused and says where to change it. Following the parent again (`inherit` back to true) clears the child's own general access.
837+ - Link access counts a visit to the folio itself or to its access root. Opening it is the `folio` read or the live socket; only the `folio` read records the visit.
838+ - "Private" (the lock) is computed by `isPrivateFolio`: no other owner along the chain, no grant, no general access, no inherited space.
839+- **Moving and copying.**
840+ - A child always shares its parent's space. Only the owner can move a folio to the top of their Private.
841+ - A move is refused when it would put a folio inside itself, under something that isn't a doc, or deeper than 10.
842+ - **Duplicate** needs only `view`. The copy lands beside the original when the person can add there, otherwise in their Private. It is theirs, with no grants and no general access, so a copy is never shared more widely than the original.
843+- **Subtree work** goes through `rebuildSubtree`, which recomputes `space_id`, `acl_root` and `path` from the top down and then rewrites `folio_access`.
844+ - Above `FOLIO_INLINE_REACL` (2,000), a `folios.reacl` job on `JOBS` does the work.
845+ - Then open rooms get `setRole` for each connected person. The local smoke test showed the `{ type: "access", role: null }` notice arriving, but `wrangler dev`'s proxy never passed the 4403 close frame to the client. Check that in production.
846+ - Passages are re-filed under their new scope, which only rewrites vector metadata.
847+- **D1 limits met while building.** D1 refuses LIKE patterns over 50 bytes, so subtrees are found with a `substr(path, 1, n) = path` prefix compare. Lists of ids and keys go in as a single JSON parameter (`json_each`), which stays inside the 100-parameter limit.
848+- **Agents** (`src/folios/agents.ts`, tested against the rules in section 4.3):
849+ - Finding things (`folios_for_agent`, search, `recall_folios_for_agent`, `stale_folios_for_agent`) is narrowed to folios that the asker and everyone in the audience can read.
850+ - `read_folio_for_agent` by id works whenever the asker can read the folio, and returns `audience_can_read: false` when someone in the conversation can't (rule 2).
851+ - Create and edit calls take no audience.
852+ - `share_folio_as_agent` works only in a people audience (a DM or a private channel), only for people already in it, only `view` or `comment`, and it never lowers an existing grant.
853+ - A folio an agent creates belongs to its asker, has `created_by` set to the agent, and gives the agent an `edit` grant. `where: { conversation }` adds `view` grants for the conversation's members. `source` is stored in `folios.source` and is not added to the text (Docs' agents used to prepend a note).
854+- **Index.**
855+ - Folios go to `g1t-folios` (`FOLIO_VECTORS`). When the binding or `AI` is missing, the service logs it once per isolate and matches words over `folio_chunks_fts`.
856+ - Projects' docs stay in `g1t-docs` and `doc_chunks` for now. `recall_folios_for_agent` merges them in, as `FolioPassage.repo_file`. Moving them into `g1t-folios` is left for Phase 7.
857+ - The backfill gained a folio stage, with cursors `p:`, then `o:`, then `f:`.
858+- **Kinds not built yet.**
859+ - Making, reading or exporting slides, designs or dashboards answers `invalid`, saying they "aren't here yet".
860+ - `folio_proposals` lists whatever is in the table, which is empty until later phases. `decide_folio_proposal` answers `invalid`.
861+ - `query_tile`, `query_dataset` and `query_dataset_for_agent` answer `invalid` until Phase 5b.
862+- **Templates.** Built-in ids are now `builtin:doc:<slug>`; the old `builtin:<slug>` still resolves.
863+- **History.** A version's Yjs state over 1.5 MB goes to the file store at `docs/versions/<folio>/<version>` (`state_key`). A new room's first content is seeded with no origin, so creating a folio records only its `created` version and is nobody's edit.
864+- **Events.**
865+ - `folio.created` goes out once per new folio. Grants given at creation don't send `folio.shared`.
866+ - `folio.shared` goes out once per grant change. Revokes send nothing.
867+ - `folio.updated` goes out once per version, except the `created` one.
868+ - Titles are included only when the whole workspace can read the folio.
869+ - `subscribers.rs` lists `folio.*` among the types that are published but not offered to webhooks.
870+- **Staleness.** `src/staleness.ts` also records folios whose citations a change touched, in `folio_changes`. It tells the owner, notifies open rooms and publishes `folio.stale`.
871+- **Trash.** A daily cron (`17 3 * * *`) deletes folios that have been in the trash for over 30 days, 500 per run. A self-hosted install runs it too, via `scheduler.mjs`.
872+- **Sidebar.** The General space always shows. Other open spaces show once joined (`space_joins`), and `join_space` takes open spaces only.
873+
874+Open for Phase 2:
875+- **Old pages (D2).** The plan drops them, so check that before the web switches over.
876+- **"Editors can share."** `canShare` supports this per-space setting, but nothing sets it yet.
877+- **Access requests.** `request_folio_access` sends to the owner and the people with full access, at most 20, with no limit on how often. Decide whether it needs one.
878+- **Self-hosted rooms.** `deploy/self-host/configs.mjs` doesn't copy `durable_objects` into the self-hosted configs. That gap predates this phase and affects `PageRoom` too. Check it before relying on rooms self-hosted.
+9−1
112112 | `services/events` | Rust | Queues (producer and fan-out) | Runs unchanged; the off services' queues are not produced to |
113113 | `services/projects` | TS | Queue consumer | Runs unchanged |
114114 | `services/chat` | TS | Durable Objects (one room per channel, WebSocket hibernation), KV `AVATARS` (custom emoji images, under `emoji/`) | Runs unchanged; workerd runs its Durable Objects, and the site serves emoji images from the same KV |
115−| `services/docs` | TS | Durable Objects (one room per page: the Yjs document, WebSocket hibernation, SQLite storage, alarms), **R2** (`FILES`, files in pages, behind the `FileStore` interface in `src/files.ts`), D1 with FTS5, a queue (`g1t-events-docs`: merges and pushes, for pages whose cited code changed and projects' docs) | Runs unchanged; workerd runs its Durable Objects, and files in pages go to RustFS (`DOCS_FILES=s3`, the `g1t-docs-files` bucket; see "Files in Docs pages") |
115+| `services/docs` | TS | Hosts Docs' pages and artifacts (folios, docs/ARTIFACTS_MODE.md). Durable Objects (one room per page, `PageRoom`, and one per artifact, `FolioRoom`: the Yjs document, WebSocket hibernation, SQLite storage, alarms), **R2** (`FILES`, files in pages and artifacts, behind the `FileStore` interface in `src/files.ts`), D1 with FTS5, a queue (`g1t-events-docs`: merges and pushes, for pages and artifacts whose cited code changed and projects' docs; also its own `docs.index` and `folios.reacl` jobs), a daily cron (artifacts in the trash for 30 days are deleted) | Runs unchanged; workerd runs its Durable Objects, files go to RustFS (`DOCS_FILES=s3`, the `g1t-docs-files` bucket; see "Files in Docs pages"), and `scheduler.mjs` runs its cron |
116116 | `services/notify` | TS | Durable Objects (one feed per person: WebSocket hibernation, SQLite storage); outbound HTTPS to browsers' push services | Runs unchanged; browser push needs a VAPID key pair (`node scripts/ops/vapid-keys.mjs`), else notifications are live in open tabs only |
117117 | `services/agents` | TS | Durable Objects (one desk per agent, alarms) | Runs unchanged; replies reach a model through the `MODELS` binding (the model proxy), which is off, so an agent answers with a short apology |
118118 | `services/search` | Rust | Queues (events and its own jobs); FTS5 | Runs unchanged |
327327 recall and the Docs search page match words instead of meaning. The
328328 backfill and catch-up jobs ride the docs service's events queue
329329 (`JOBS`); without a queue one batch runs at a time as recall asks.
330+- **Artifacts' semantic index** (folios, the same service) uses the same
331+ two interfaces with an index of its own: Vectorize `g1t-folios`
332+ (binding `FOLIO_VECTORS`, filtered by `workspace_id`, `scope` and
333+ `kind`). Self-hosted it is absent like the others, and artifacts keep
334+ every passage in D1 (`folio_chunks`, with full text): search and
335+ agents' recall match words, and the service says so once in its log.
336+ Every passage is checked against the artifact's access before anyone
337+ sees it, so the index is never what keeps something private.
330338
331339 ### Files in Docs pages
332340
+1−2
491491 /**
492492 * Artifacts (folios, services/docs): a folio was made. Like `doc.page.*`,
493493 * published with no `repoId`, never offered to webhooks, and readers check
494− * access with the docs service before showing anything of it. Not
495− * published yet: Phase 1 of docs/ARTIFACTS_MODE.md starts them.
494+ * access with the docs service before showing anything of it.
496495 */
497496 "folio.created": FolioEventData;
498497 /** A folio's content changed: a version (`versionKind`) with everyone whose changes are in it. */