g1t/docs/BILLING_OPERATIONS.md

398 lines23,952 bytesCodeBlame
1# Billing operations: costs, margin and prices
2
3How g1t checks what it charges against what Cloudflare charges it, keeps
4prices at cost plus 20%, and tells staff when the margin slips. Internal.
5Code: `services/billing/src/costs.rs` (reading the bill), `margin.rs`
6(reconciliation, drift, alerts), `pricing.rs` (versions, proposals,
7notice), `keeper.rs` (sandbox and Workers for Platforms measurements),
8`budget.rs` (what g1t pays for itself, and its caps; see
9[Spend caps](#spend-caps)).
10Page: sudo **Costs & margin** (`/costs`).
11
12Several Cloudflare products g1t runs on are new. Artifacts bills
13"operations" from 2026-10-14 without defining them (see
14[ARTIFACTS.md](ARTIFACTS.md), M1). So nothing here hard-codes a product
15list or a unit: every line Cloudflare bills is kept, and how a line maps
16to what g1t sells is data you change from sudo, without a deploy.
17
18## Data sources
19
20| Source | What | Where it lands |
21| --- | --- | --- |
22| Billable usage, `GET /accounts/{account}/billable-usage?from=&to=` | One row per service per day in FOCUS columns: `ServiceFamilyName`, `ServiceName`, `ChargePeriodStart`, `PricingQuantity`, `ContractedCost` / `BilledCost` / `ListCost`. Every product g1t uses appears once it is used: Workers, Workers for Platforms, D1, KV, R2, Queues, Containers, Durable Objects, Artifacts, Browser Rendering, Workers AI, Vectorize, Cloudflare for SaaS, Email. Inside an included amount the cost is 0. | `cost_lines`, source `billable_usage` |
23| GraphQL `artifactsEventsAdaptiveGroups` | Artifacts' own count by `date`, `eventType` and `repositoryName`. Operations are `create`, `fork`, `push`, `pull`, `delete`; errors (`rateLimited`, `serverError`, …) are kept but not counted. | `cost_lines`, source `artifacts_events`; per workspace (from the store key `<workspace>--<repo>`) in `own_counts` as `cloudflare_git` |
24| The ledger | Every charge: its cost at the price book's cost, what it was charged at price, what paid for it. | read, never written |
25| `pending_usage` | Month-end meters (git, storage, scans, embeddings, the cache) as they stand. | snapshotted daily into `pending_days` |
26| `plan_payments` | The plan's $20. | read |
27| repos `git_operations` | Operations customers are charged for, per workspace, counted by repos through its `operation_mapping`. | `own_counts` meter `git_operations` |
28| repos `artifacts_usage` | Every raw meter of the git store (`git.fetch`, `git.receive_pack`, `binding.*`, …) per day and workspace, with repos' `operation_mapping`. | `own_counts` meters `artifacts_<raw meter>`, and `cost_operations` (raw counts × the mapping's `cost_operations`: what g1t expects Cloudflare to bill) |
29
30A meter's slug is Cloudflare's name lower-cased with words joined by `_`
31and the "(First … included)" note dropped: `Workers for Platforms CPU ms
32(First 60M ms are included)` under `Workers` is product `workers`, meter
33`workers_for_platforms_cpu_ms`. Several rows of the same day and meter
34(regions, tiers) are added together before they are stored.
35
36## Credentials
37
38| Secret on g1t-billing | Permissions | Used for |
39| --- | --- | --- |
40| `CLOUDFLARE_BILLING_TOKEN` (optional) | Account: **Billing Read**, Account: **Account Analytics Read**, for the g1t account only | Reading the bill and the Artifacts events |
41| `CLOUDFLARE_USAGE_TOKEN` (exists) | Billing Read, Account Analytics Read, AI Gateway Read | The keeper; also the bill when `CLOUDFLARE_BILLING_TOKEN` is not set |
42
43With neither, the daily run reconciles only what g1t counted itself, and
44the page says the bill cannot be read. Nothing fails. To set the scoped one:
45
461. Cloudflare dashboard → My Profile → API Tokens → Create Token → Custom token.
472. Permissions: Account · Billing · Read; Account · Account Analytics · Read.
483. Account resources: Include · the g1t account. No zone permissions.
494. `cd services/billing && npx wrangler secret put CLOUDFLARE_BILLING_TOKEN`.
505. In sudo, Costs & margin → **Run the analysis now**.
51
52Alerts are emailed through the `EMAIL` binding (Cloudflare Email Sending)
53to `COSTS_ALERT_EMAIL` (`hey@flagon.io`). An empty value sends none.
54
55## Schedule
56
57The daily cron (`17 4 * * *`, `keeper::DAILY`) runs, in order:
58
591. The keeper's measurements (sandbox seconds, app requests and CPU), each
60 a proposal now, not a direct change.
612. `costs_daily`:
62 1. Read the bill and the Artifacts events. The first run reads the last
63 31 days (GraphQL keeps 31); later runs the last 4, since Cloudflare
64 restates recent days, or back to the last day read after a gap.
65 Lines are upserted on `(day, source, product, meter)`, so a re-read
66 replaces, never adds.
67 2. g1t's own counts for the same days (replaced per day).
68 3. Snapshot `pending_usage` into `pending_days`.
69 4. Reconcile the last 31 days (further back after a gap) into
70 `margin_days` and `workspace_costs`
71 (replaced per day).
72 5. Drift over the last 7 days into `cost_drift`.
73 6. Unit costs over the last 30 days, proposed to the price book.
74 7. Apply price versions whose date has come.
75 8. Open, update and close margin alerts; email new ones.
76 9. Email owners on the plan about rises to come.
77
78**Run the analysis now** at the top of sudo's Costs & margin page
79(`admin_run_costs`) runs all of step 2 at once, alerts included, with no
80need to wait for 04:17 UTC. Running it twice is safe: every step replaces
81what it wrote.
82
83## Reconciliation math
84
85Every Cloudflare line goes to one of g1t's products ("buckets") by
86`cost_map`: the row for its product with the longest matching meter
87prefix, `*` last. A line no row claims goes to `unmapped`.
88
89| Bucket | Cloudflare | Paid for by (`revenue_map`) |
90| --- | --- | --- |
91| `sandboxes` | Containers, Durable Objects compute duration | `sandbox`, `self_hosted`, `builds` |
92| `deployments` | Workers for Platforms | `deployments` |
93| `git` | Artifacts operations (and its events, as counts) | `git` |
94| `repo_storage` | Artifacts storage | `storage` |
95| `actions_cache` | R2 | `cache` |
96| `embeddings` | Workers AI, Vectorize | `context` |
97| `security` | (Workers CPU, under `platform`) | `security` |
98| `domains` | Cloudflare for SaaS | `domains` |
99| `models` | not Cloudflare: AI Gateway's settled cost on the ledger | every other task (agent runs) |
100| `platform` | Workers, D1, KV, Queues, Email, Browser Rendering, other Durable Objects | the plan's price |
101
102For each day and bucket:
103
104- **Cloudflare cost** = Σ the bucket's lines' cost, as billed: after the
105 included allowances, so a month inside them costs $0 here as on
106 Cloudflare's Billable usage page. `models` uses the ledger's cost of the
107 tokens instead; that is paid to the model providers and is not on
108 Cloudflare's bill.
109- **Own cost** = Σ the ledger's `cost_micros` for the bucket's keys (the
110 price book's cost when charged), plus month-end deltas. A workspace's own
111 model provider is no cost to g1t.
112- **Value** = what customers were charged at price: `-amount_micros` plus
113 what the plan's included usage, a trial, the open-source pool or g1t paid.
114 g1t's own (comped) workspaces are valued at cost plus the margin.
115- **Cash** = what workspaces paid: `-amount_micros`, and the plan's price.
116- **Given away** = the part of the cost that went on usage g1t paid for
117 itself on purpose, by why:
118 - **comped**: all of a comped workspace's cost, every bucket;
119 - **free use**: a free period's usage, the overruns g1t covered
120 (`ledger.given_micros`), and all of a workspace's cost on a day it had
121 nothing priced (free allowances);
122 - **trial** and **open-source pool**: what `trial_micros` and
123 `oss_micros` paid.
124
125 Otherwise a workspace's day is split by those shares of its value at
126 price, and the same shares of each of its buckets' cost are given, its
127 part of running g1t included. The Team plan's included usage is sold:
128 the plan's price paid for it. Stored on `margin_days` (`given_micros`
129 and `given_<why>_micros`) and `workspace_costs` (`given_micros`).
130- **Month-end meters**: a day's figure is that day's `pending_days`
131 snapshot less the day before's, within a month. Their month-end ledger
132 entries are left out, so nothing is counted twice.
133- **Product margin** = (value − cost) / value.
134- **Sudo's statement** keeps apart:
135 - **Usage sold**: cash for usage against the cost of the usage buckets
136 less what was given. Its margin is the headline; at cost plus 20% it
137 sits near 16.7%.
138 - **Running g1t**: the plan's price against `platform` less its given
139 share.
140 - **Cloudflare subscriptions**: `CLOUDFLARE_FIXED_MONTHLY_MICROS` over
141 the range, an estimate, since they are not on the usage bill.
142 - **Not mapped**: billed, charged for by nothing.
143 - **Given away**: by why. A budget, watched under g1t's own spend, never
144 shown as a loss.
145 - **All in**: money in against all of it, with the figure without what
146 was given beside it. **Who g1t paid** splits the cost into Cloudflare
147 and the model providers.
148
149 The overall alert is (Σ cash − (Σ cost − Σ given)) / Σ cash.
150- **Quantities**: where a mapping names an `own_meter`, Cloudflare's
151 billed quantity of those lines (or, without one, Artifacts' operation
152 events) against g1t's own count.
153
154**Shared costs to workspaces.** A bucket's cost is shared in proportion
155to, first available: Cloudflare's own per-workspace count
156(`cloudflare_<bucket>`, today the Artifacts events by repository), g1t's
157own count, what its usage cost (so free use carries its own cost), what
158each was charged for it. `platform`
159and `unmapped` are shared by each workspace's share of all usage that
160day. Shares are whole micros that add up to the bill exactly (largest
161remainder).
162
163## Drift (last 7 days)
164
165| Kind | When | What to do |
166| --- | --- | --- |
167| Count | g1t's count and Cloudflare's differ by more than the mapping's `drift_percent` (10%) | Find out what Cloudflare counts: compare its events with `own_counts` `artifacts_*` and `cost_operations`. If it counts more (binding reads, `ls-refs`), either change repos' `operation_mapping` so customers are charged for what Cloudflare counts, or leave it and let the per-unit cost rise (below). |
168| Cost | Cloudflare charged more than `drift_percent` away from the price book's cost of the same usage, with at least `min_daily_cost` | A price is stale: check the proposals. |
169| Leak | Cost of at least `min_daily_cost` and nothing charged for it (never for `platform`), or a meter in `unmapped` | Map the meter (below), or decide it is overhead (`platform`). |
170
171## Prices: versions, proposals, notice
172
173- `price_versions` holds every price ever, never edited. `prices` is the
174 version in force. The daily run applies a version once its
175 `effective_at` has come, and adds the public `price_changes` record.
176 Ledger entries made from the price book carry `price_version` (the
177 version ids, comma-separated), so a past statement is always explained
178 by the prices of its day.
179- Proposals come from the keeper (sandbox seconds, app requests and CPU)
180 and the reconciler (mappings with `scale_to_own`: today git operations).
181 For git operations: Cloudflare's rate per its own operation (the median
182 over charged days of cost ÷ quantity) × (Cloudflare's operations ÷ g1t's)
183 × 1,000. If Cloudflare counts three for each one g1t counts, the per-1,000
184 price triples. At least 1,000 of g1t's operations are needed.
185- Decision (`pricing::decide`): under 2% is noise; more than 4× either way
186 is suspect and waits for staff; within `auto_apply_percent` (25%) it is
187 applied on its own when `auto_apply` is on; anything else waits.
188- Notice: a fall applies at once. A rise applies `notice_days` (14) after
189 the decision, and for a monthly meter (git, storage, cache, domains,
190 embeddings, scans) at the start of the month after that, so no month
191 is charged at two prices. Owners of workspaces on the plan are emailed
192 once per rise (`price_notices`), and the pricing page lists it with
193 "takes effect". Rises are never retroactive; margin protection is for
194 new usage once notice has run.
195- Staff approve or reject in sudo. A rejection needs a note.
196
197## Changing a mapping
198
199In sudo, Costs & margin → **Mappings**: Cloudflare's product and meter
200prefix (as **Cloudflare's lines** lists them; `*` for the rest of the
201product), g1t's product, and optionally:
202
203- **Price meter**: the price book meter the line measures.
204- **Own meter**: g1t's count of the same units (`own_counts.meter`).
205- **Scale to g1t's count**: price one of g1t's units at as many of
206 Cloudflare's as it took (proposals as above).
207- **Drift threshold**.
208
209It applies from the next run; **Run the analysis now** applies it at once.
210Every change is in the audit log (`cost_mapping`).
211
212## Which raw meters are operations
213
214There is one mapping, and the repos service owns it: `operation_mapping`
215in g1t-repos' database, one row per raw meter with `cost_operations` (how
216many operations Cloudflare bills for it) and `billable_operations` (how
217many the customer is charged for). Change it with repos'
218`set_operation_mapping` RPC (services only), or
219`npx wrangler d1 execute g1t-repos --remote` until sudo has a form. A
220change applies to counts from then on, never to what was counted.
221Billing keeps no mapping of its own: it reads repos' `git_operations`
222(already mapped) for what customers are charged, and `artifacts_usage`
223(raw counts with the mapping) for `cost_operations`. Migration 0023 drops
224the `billable_units` table 0022 made for this, which was never written.
225
226**Charged at price** (a product's value) is what each day's usage was paid:
227charged to a card or credit, or drawn from the plan's included usage, a
228trial, a pool or a gift. Usage nothing paid for, as in a free period, is
229valued at price (cost plus the margin), since it was given away at its price
230rather than sold for nothing; so are g1t's own workspaces. Runs on a
231workspace's own model provider have no cost to g1t. Every daily run
232reconciles the whole 31-day window from what is already kept, so a change in
233how a day is valued reaches every day sudo shows.
234
235## Alerts runbook
236
237| Alert | Raised when | First steps |
238| --- | --- | --- |
239| Margin under the floor | A product's value against cost under `margin_floor_percent` (10%) for `alert_days` (3) days running, each with at least `min_daily_cost` | Open the product on Costs & margin. Cost up? Check proposals (approve a rise; it waits out the notice). Value down? A mapping or `revenue_map` may have moved. |
240| All of g1t under the floor | The same for money in against the cost of what was sold: every cost less what was given away (comped workspaces, free periods, the trial, the pools), which is a budget watched in budget.rs. While less than $1 a day comes in, it says the dollars, not a percentage | Look at which products moved; check `platform` (it has no revenue of its own and grows with traffic). Before launch, with little paid usage, expect it. |
241| Leak | Drift of kind leak | Map the meter, or decide it is overhead. |
242| Drift | Count drift | See Drift above. Cloudflare's definitions change in beta: ask them in writing ([ARTIFACTS.md](ARTIFACTS.md), §7). |
243| Costs more than it pays | A workspace's shared cost over 30 days above what its usage was priced at (`value_micros`, whoever paid: card, trial, gift or included usage) × `anomaly_factor`, at least `anomaly_floor`; not comped workspaces | Shown on Reach out as "Costs more than it pays": its usage is priced below what it costs. Abuse (Abuse & fraud page) or a gap in pricing. Not emailed. A trial or gift paying for usage does not raise it. |
244
245Alerts close on their own when the condition clears. Open ones are
246emailed again weekly. The red bar on every sudo page shows margin,
247overall and leak alerts.
248
249## Token usage
250
251The model proxy (`services/models`) reads Anthropic's `usage` from every
252`/v1/messages` answer, streamed or whole, on g1t's models and on a
253workspace's own provider alike (OpenAI-shaped providers are translated
254first). Count-tokens requests are not answers and are skipped. After the
255answer, it calls `record_tokens`, which adds input, output, cache reads and
256cache writes to one row per day, workspace, person, session and model in
257`token_usage` (migration `0030_token_usage.sql`). The person is who the run
258was for, from the model session's `requested_by`; never g1t's agent. A
259report that fails is dropped and never affects the answer.
260
261`token_usage` reads a window (42 days by default, 366 at most) for the
262workspace or one person: totals, every day's tokens and the active days,
263with `costMicros` the window's run charges from the ledger, measured as
264`usage` measures them. These counts are for views only: runs are still
265priced from AI Gateway's logs, never from `token_usage`.
266
267## Tables (migration `0022_costs_and_margin.sql`)
268
269`cost_lines`, `cost_map`, `revenue_map`, `own_counts`,
270`pending_days`, `margin_days`, `workspace_costs`, `cost_drift`,
271`margin_alerts`, `price_versions` (seeded with every current price as
272version 1), `price_proposals`, `price_notices`, `cost_settings` (the
273guardrails, seeded), the `actions_cache` price, and `ledger.price_version`.
274Every create is `IF NOT EXISTS` and every seed `INSERT OR IGNORE`; the one
275`ALTER` is applied once by D1's migration tracking. Migration
276`0023_one_operation_mapping.sql` drops `billable_units` (see above).
277
278## Spend caps
279
280Two caps keep what g1t pays for itself bounded while billing takes no
281real money. Both are measured at **cost** (what Cloudflare and the model
282providers charge g1t), never at price. Code: `services/billing/src/budget.rs`.
283Page: sudo **Costs & margin** → **g1t's own spend** (`/costs#spend`).
284
285### What counts as g1t's own spend
286
287Every charge that settles (an agent run's model cost from `finish_run` or
288AI Gateway's settlement, sandbox time from `record_sandbox`, a build from
289`charge_feature`) is split by what paid for it and g1t's part is added to
290`g1t_spend` (day, bucket, billing account):
291
292| Bucket | What |
293| --- | --- |
294| `comped` | All of a comped account's work (flagon-io) |
295| `trial` | The trial credit's share |
296| `oss` | The open-source pool's share |
297| `given` | A free workspace's overrun past its last bit of trial |
298| `unpaid` | Charged, but with no real money behind it: Stripe's test key, or `FREE_WHILE_BUILDING` |
299
300The plan's included usage and on-demand charges count as revenue only
301with live payments; in test mode they are `unpaid`. A workspace's own
302model provider costs g1t nothing and is not counted. Month-end meters
303(git, storage, scans, embeddings, the cache) are not counted here; the
304daily reconciliation covers them. Migration `0024_spend_caps.sql`
305backfills the current month from the ledger.
306
307### Caps
308
309| Cap | Variable (g1t-billing) | Default | At the cap |
310| --- | --- | --- | --- |
311| A comped account's monthly budget | `COMPED_MONTHLY_CEILING_MICROS`, or the account's own **Limit** in its terms | $150 a month | New work on the account (agents, checks, workflows, builds) is refused with "<name>'s monthly budget for g1t's own agents is used up … Staff can raise it in sudo". Runs already going finish; the per-run cap still applies to them. It lifts when staff raise the budget or the month turns (UTC). |
312| The daily breaker | `PLATFORM_DAILY_SPEND_CAP_MICROS` | $75 a day (UTC) | New agent runs on g1t's hosted models that g1t would pay for are refused until 00:00 UTC. Not paused: agents on the workspace's own model provider, checks and builds, and workspaces paying with live payments on the plan (not given by staff) or an enterprise contract. In test mode that exemption covers no one. |
313
314`0` turns either off. `CLOUDFLARE_FIXED_MONTHLY_MICROS` ($30: Workers
315Paid and Workers for Platforms) is shown on the page only.
316
317The checks are cheap: `reserve` reads today's total (one indexed sum) and,
318for a comped account, its month's comped rows. Refusals come back as
319`paused`, which the compute gate honours for every plan, internal and
320enterprise included (`packages/contracts/src/compute.ts`). The runner tells
321billing whether an agent run is on hosted models (`hostedModel` on
322`reserve`); a caller that does not say is treated as hosted.
323
324### Alerts
325
326All to `COSTS_ALERT_EMAIL` (`hey@flagon.io`), through the `EMAIL` binding:
327
328- **Comped budget**: at 50, 75, 90 and 100%, once each per account and month
329 (`budget_alerts`), checked every 15 minutes. A jump past several levels
330 sends only the highest.
331- **Breaker**: at once, from the charge that trips it; if that email fails,
332 the 15-minute cron sends it (`spend_breaker.told_at`).
333
334While the breaker is open or a comped budget is used up, every sudo page
335shows a red **Spend cap** bar.
336
337### Raising and lifting
338
339- **Raise a comped budget**: sudo → the workspace → Billing → **Terms**, set
340 **Limit $** to the new monthly budget (blank goes back to the default),
341 with a note. It applies to the next start; nothing to deploy. The change
342 is in the account's audit log.
343- **Lift the breaker for today**: sudo → Costs & margin → **g1t's own
344 spend** → **Lift for today**, with why (`admin_lift_breaker`; audit action
345 `breaker_lifted`). It resets by itself at 00:00 UTC.
346- **Change a default**: edit the variable in `services/billing/wrangler.jsonc`
347 and deploy g1t-billing.
348
349## Stripe
350
351Billing keeps what it needs from Stripe so reads never wait on it, and
352hears of changes three ways (`webhooks.rs`, `stripe_sync.rs`).
353
354**The webhook.** A destination made in Stripe's dashboard (Developers →
355Webhooks → Add destination) with the endpoint URL
356`https://api.g1t.sh/stripe/webhook`, in the mode of billing's key (at
357launch, make one in live mode and put its secret). Its signing secret
358(`whsec_…`: the destination, Signing secret, Reveal) is the billing
359Worker's secret:
360
361```sh
362cd services/billing && npx wrangler secret put STRIPE_WEBHOOK_SECRET
363```
364
365Without it every event is refused with 400. After rolling the secret in
366Stripe, put the new one; during the roll Stripe signs with both, so there
367is no gap. The event list need not be exact: billing adds any event it
368handles that the destination does not send (daily, or **Fix destination**
369in sudo → Stripe), and enables it again if Stripe disabled it. It never
370changes the secret. sudo → Stripe shows whether the secret is set, the
371destination and its status, missing events, and the latest events.
372Events are claimed once each in `stripe_events`; a handler that fails
373forgets its claim, and Stripe retries.
374
375**What is kept, and how it stays current**
376
377| Kept | Where | Refreshed by |
378| --- | --- | --- |
379| Saved card (brand, last 4, expiry) | `accounts.card_*`, `card_synced_at` | `customer.updated`, `payment_method.*`, `setup_intent.succeeded`; forgotten when g1t sets a default card, so the next read asks once; the daily pass for cards older than 7 days |
380| Plans | `subscriptions` | `customer.subscription.*`, `invoice.*`; a read asks Stripe at most hourly once a period is over; the daily pass for rows older than a day |
381| Payments, refunds, disputes | ledger, `checkouts`, invoices | their events |
382
383**Every cron run (every 15 minutes)** replays missed events: Stripe's event
384list from an hour before `stripe_sync.through`, oldest first, through the
385same once-only claim. `through` moves to 5 minutes before now when all were
386handled, back to the first failure otherwise, and stays when more than
3871,000 events were listed. Claims stuck at `handling` for 10 minutes are
388dropped so the replay retries them. The first run reads 3 days back.
389
390**Daily** (`keeper::DAILY`): the destination at billing's address is
391enabled again if Stripe disabled it and given any missing event, audited as
392`stripe`/`webhook`; then up to 25 stale cards and 25 stale plans are read
393again.
394
395**What still calls Stripe on a request**: starting a payment page, a plan
396or a card check; opening the billing portal; settling a page the person
397came back from; renaming a workspace (the customer's name). Nothing a page
398view reads.