flagon-io/g1t

public

g1t/apps/docs/src/content/docs/about/open-letter-to-cloudflare.md

150 lines7,588 bytesCode Blame

1	---
2	title: An open letter to Cloudflare
3	description: From the team at Flagon, Inc. building g1t, a git platform that runs entirely on Cloudflare. What works, where we hit walls, and what we'd ask for.
4	---
5
6	Dear Cloudflare,
7
8	We're the small team at Flagon, Inc. building g1t, a git platform where
9	people and coding agents work in the same issues, pull requests and merge
10	queue. Every part of it runs on you: about twenty Workers, a D1 database for
11	each service that keeps data, Artifacts for every repository and every pull
12	request's fork, Containers for agents and CI, R2, KV, Queues, and Cloudflare
13	for SaaS for our customers' domains. We have no servers.
14
15	This is a thank-you, and a list of what would help us most next.
16
17	## Why we built on you
18
19	A git platform is usually a fleet: storage nodes, a job system, a database
20	cluster, a CDN in front, and people on call for all of it. We wanted to run
21	one for thousands of teams with a handful of people. You offered a global
22	platform where the unit of work is a request, the unit of storage is a
23	Durable Object, and nothing costs money while nobody is using it.
24
25	Artifacts is the reason g1t exists in this shape. One Durable Object per
26	repository is the right isolation unit: a busy repository doesn't slow its
27	neighbours, and there is nothing to shard by hand. It speaks real git, so
28	stock clients cloned, fetched and pushed through our proxy from the first
29	day. `fork()` is a single call, and per-pull-request isolation for agents
30	fell out of it almost for free. Scoped tokens that expire on their own let
31	us hand a sandbox a credential that dies with it. The read binding powers
32	every page we render, blame, mergeability and search, without a git client
33	anywhere.
34
35	The rest of the platform held up too. Rust compiled to WebAssembly runs most of
36	our services, and TypeScript the rest. D1's read replication is free and good. Containers gave us
37	sandboxes in three sizes. With a cached credential and ref listing, a
38	`git fetch` with nothing new answers in under half a second, and most of
39	our pages answer in under 250 ms. We went from an empty repository to a
40	launch on this stack, and most of it worked the first time.
41
42	## Where we hit walls
43
44	Running a real platform for many teams found the edges. None of these
45	stopped us. Each one costs us code, latency or certainty, and each one will
46	cost the next team building on you the same.
47
48	What a billable operation is. Artifacts pricing names "repo operations,
49	such as create, push, pull, and clone", and the metrics list a different set
50	of event names. Neither says whether binding reads, token mints or ref
51	listings count. We price from cost, so this decides what our customers pay.
52	Depending on the answer, our model for a few thousand workspaces lands
53	anywhere between about $1.8k and $31k a month. Today we count every clone,
54	fetch and push ourselves, and hope it matches.
55
56	What a fork stores. Forks are the natural primitive for a pull request,
57	and agents open pull requests by the thousand. We can't find whether a fork
58	shares objects with its source or copies them. If it copies, an agent-heavy
59	account reaches the 1 TB account limit in days (it can be raised on
60	request, but only by asking), and at that point every push
61	in the account fails, for every customer at once. We keep forks for now,
62	and are measuring it ourselves.
63
64	A write path and a pre-receive hook. The binding reads, but it can't
65	list refs, move them, or write objects. So to land a pull request we speak
66	git's wire protocol to our own storage from inside a Worker, buffering packs
67	in an isolate with 128 MB to share. With no hook before refs move, branch
68	protection and secret scanning only hold for pushes through our proxy, which
69	parses each pack in WebAssembly before forwarding it, up to the size an
70	isolate can hold. We wrote a second
71	implementation of git's pack format to get there.
72
73	Ref-change events and the cost of a credential. Push events need one
74	subscription per repository, which doesn't scale to tens of thousands of
75	repositories and forks. We record ref changes ourselves, and a test scans
76	our own source to make sure every code path that moves a ref says so. Every
77	git credential takes three binding calls and about 0.8 s to mint, so we
78	cache sealed tokens across isolates in KV.
79
80	Placement that follows data. Smart Placement once ran our site in
81	Amsterdam for a visitor in Denver, while every D1 primary we have is in
82	western North America. Every query crossed the Atlantic, and our Explore
83	page took 0.85 s instead of 0.17 s. We turned placement off everywhere and
84	measure each Worker by hand.
85
86	D1 sessions across service bindings. Read replicas need a bookmark to
87	give read-your-writes. Our site reads from seven services, each with its
88	own database, so we built a header protocol to carry bookmarks through service
89	bindings into a cookie and back.
90
91	Containers that build images and keep disks. We found no supported way
92	to run Docker or BuildKit in a Container, so our own CI can't rebuild our
93	sandbox image; that waits for a machine outside. Container disk is
94	ephemeral, so the self-hostable git store we'd like as a warm fallback has
95	to live off Cloudflare.
96
97	Inbound TCP for SSH. Git users expect `git@host:owner/repo`. Workers
98	take no inbound TCP, so g1t is HTTPS only. We've applied for the beta and
99	are waiting.
100
101	## What we built in the meantime
102
103	A per-workspace operation counter that is our best guess at your invoice. A
104	smart HTTP
105	client inside a Worker for landing, catch-up, mirrors and imports. Our own
106	push policy in front of Artifacts. A versioned ref cache with a test that
107	guards it. Two layers of credential caching. A bookmark protocol for D1.
108	A probe that deploys throwaway Workers to measure placement, and a
109	`Server-Timing` header on every response so we see the next regression.
110	Image builds on a laptop.
111
112	All of it works. Most of it is code we'd happily delete. Next is a fork
113	sweep, to switch on once we know what forks cost.
114
115	## What we're asking for
116
117	1. A published definition of a billable Artifacts operation, with
118	per-repository metrics that use the same names as the invoice.
119	2. Documented fork storage, an expiry on `fork()`, a repository's stored
120	bytes in `info()`, and a warning before the account storage limit, with
121	failures per repository rather than account-wide.
122	3. Ref listing, atomic compare-and-swap ref updates and streaming pack
123	writes in the binding.
124	4. A pre-receive hook that a Worker answers.
125	5. Account-level ref-change events to a Queue, a read-after-write guarantee
126	for refs, and git forwarding authenticated by the binding, with no token
127	to mint.
128	6. Placement that accounts for D1 primaries and service bindings, and D1 as
129	a placement target.
130	7. D1 sessions that travel across service bindings.
131	8. Image builds and persistent volumes for Containers.
132	9. Inbound TCP, so git can run over SSH.
133	10. A support path during the beta, snapshot restore and export for
134	repositories, and a date for general availability with an SLA.
135
136	## Let's work on it together
137
138	We chose you on purpose, and we'd choose you again. A small team running a
139	global git platform with no servers is the promise of what you've built,
140	and g1t shows that it mostly holds. We'd like to help close the
141	rest of the gap: traces, test repositories, early builds to try, or a call
142	with the teams involved. Whatever is useful.
143
144	You can reach us through [g1t.sh](https://g1t.sh/support). Our code lives
145	at [g1t.sh/flagon-io/g1t](https://g1t.sh/flagon-io/g1t), on the platform
146	it describes.
147
148	With thanks,
149
150	The team at Flagon, Inc.