The open letter to Cloudflare says what is true now: more than twenty-five Workers, Rust for the core, forks removed a day after their pull request closes, Docker running in Containers with a shared network rather than not at all, the site reading from about twenty services, and sharding across namespaces as a wall and an ask; image builds on a laptop and the fork sweep still to come are gone.
1 file+31−190/1 viewed
| 7 | 7 | ||
| 8 | 8 | We're the small team at Flagon, Inc. building g1t, a git platform where | |
| 9 | 9 | people and coding agents work in the same issues, pull requests and merge | |
| 10 | − | queue. Every part of it runs on you: about twenty Workers, a D1 database for | |
| 10 | + | queue. Every part of it runs on you: more than twenty-five Workers, a D1 database for | |
| 11 | 11 | each service that keeps data, Artifacts for every repository and every pull | |
| 12 | 12 | request's fork, Containers for agents and CI, R2, KV, Queues, and Cloudflare | |
| 13 | 13 | for SaaS for our customers' domains. We have no servers. | |
| ⋯ | |||
| 32 | 32 | every page we render, blame, mergeability and search, without a git client | |
| 33 | 33 | anywhere. | |
| 34 | 34 | ||
| 35 | − | The rest of the platform held up too. Rust compiled to WebAssembly runs most of | |
| 36 | − | our services, and TypeScript the rest. D1's read replication is free and good. Containers gave us | |
| 35 | + | The rest of the platform held up too. Rust compiled to WebAssembly runs our | |
| 36 | + | core (repositories, issues and pull requests, the API, security and | |
| 37 | + | billing), and TypeScript the rest. D1's read replication is free and good. Containers gave us | |
| 37 | 38 | sandboxes in three sizes. With a cached credential and ref listing, a | |
| 38 | 39 | `git fetch` with nothing new answers in under half a second, and most of | |
| 39 | 40 | our pages answer in under 250 ms. We went from an empty repository to a | |
| ⋯ | |||
| 58 | 59 | shares objects with its source or copies them. If it copies, an agent-heavy | |
| 59 | 60 | account reaches the 1 TB account limit in days (it can be raised on | |
| 60 | 61 | request, but only by asking), and at that point every push | |
| 61 | − | in the account fails, for every customer at once. We keep forks for now, | |
| 62 | − | and are measuring it ourselves. | |
| 62 | + | in the account fails, for every customer at once. So we remove a pull | |
| 63 | + | request's fork a day after it closes, keeping its head in the repository it | |
| 64 | + | came from, and measure the rest ourselves. | |
| 63 | 65 | ||
| 64 | 66 | **A write path and a pre-receive hook.** The binding reads, but it can't | |
| 65 | 67 | list refs, move them, or write objects. So to land a pull request we speak | |
| ⋯ | |||
| 84 | 86 | measure each Worker by hand. | |
| 85 | 87 | ||
| 86 | 88 | **D1 sessions across service bindings.** Read replicas need a bookmark to | |
| 87 | − | give read-your-writes. Our site reads from seven services, each with its | |
| 88 | − | own database, so we built a header protocol to carry bookmarks through service | |
| 89 | + | give read-your-writes. Our site reads from about twenty services, most with | |
| 90 | + | their own database, so we built a header protocol to carry bookmarks through service | |
| 89 | 91 | bindings into a cookie and back. | |
| 90 | 92 | ||
| 91 | − | **Containers that build images and keep disks.** We found no supported way | |
| 92 | − | to run Docker or BuildKit in a Container, so our own CI can't rebuild our | |
| 93 | − | sandbox image; that waits for a machine outside. Container disk is | |
| 94 | − | ephemeral, so the self-hostable git store we'd like as a warm fallback has | |
| 95 | − | to live off Cloudflare. | |
| 93 | + | **Container networks and disks.** Docker runs inside a Container, and our | |
| 94 | + | CI jobs use it, but a container can't route a network of its own or change | |
| 95 | + | its packet filter, so every container in a job shares the job's network and | |
| 96 | + | two of them can't listen on the same port. Container disk is ephemeral, so | |
| 97 | + | the self-hostable git store we keep as a fallback has to live off | |
| 98 | + | Cloudflare. | |
| 99 | + | ||
| 100 | + | **Sharding is ours to do.** One Artifacts namespace takes 2,000 requests | |
| 101 | + | every 10 seconds, and a binding addresses one namespace. To grow past that, | |
| 102 | + | and to keep EU workspaces' repositories in the EU, we route each repository | |
| 103 | + | to a namespace and move repositories between them ourselves. | |
| 96 | 104 | ||
| 97 | 105 | **Inbound TCP for SSH.** Git users expect `git@host:owner/repo`. Workers | |
| 98 | 106 | take no inbound TCP, so g1t is HTTPS only. We've applied for the beta and | |
| ⋯ | |||
| 106 | 114 | push policy in front of Artifacts. A versioned ref cache with a test that | |
| 107 | 115 | guards it. Two layers of credential caching. A bookmark protocol for D1. | |
| 108 | 116 | A probe that deploys throwaway Workers to measure placement, and a | |
| 109 | − | `Server-Timing` header on every response so we see the next regression. | |
| 110 | − | Image builds on a laptop. | |
| 117 | + | `Server-Timing` header on every response so we see the next regression. A | |
| 118 | + | sweep that removes each pull request's fork a day after it closes. | |
| 119 | + | Repository sharding across namespaces, with a read-only fallback rebuilt | |
| 120 | + | from nightly backups. | |
| 111 | 121 | ||
| 112 | − | All of it works. Most of it is code we'd happily delete. Next is a fork | |
| 113 | − | sweep, to switch on once we know what forks cost. | |
| 122 | + | All of it works. Most of it is code we'd happily delete. | |
| 114 | 123 | ||
| 115 | 124 | ## What we're asking for | |
| 116 | 125 | ||
| ⋯ | |||
| 128 | 137 | 6. Placement that accounts for D1 primaries and service bindings, and D1 as | |
| 129 | 138 | a placement target. | |
| 130 | 139 | 7. D1 sessions that travel across service bindings. | |
| 131 | − | 8. Image builds and persistent volumes for Containers. | |
| 132 | − | 9. Inbound TCP, so git can run over SSH. | |
| 133 | − | 10. A support path during the beta, snapshot restore and export for | |
| 140 | + | 8. Persistent volumes for Containers, and container networks a job can | |
| 141 | + | route. | |
| 142 | + | 9. One binding that addresses many namespaces, and a namespace rate limit | |
| 143 | + | that can be raised on request. | |
| 144 | + | 10. Inbound TCP, so git can run over SSH. | |
| 145 | + | 11. A support path during the beta, snapshot restore and export for | |
| 134 | 146 | repositories, and a date for general availability with an SLA. | |
| 135 | 147 | ||
| 136 | 148 | ## Let's work on it together | |
Comments
Sign in to comment.