Sign in

flagon-io/g1t

public

g1t/services/billing/src/keeper.rs

783 lines33,946 bytesCodeBlame

Pick any line to see why it is the way it is: the commit, the pull request and issue it came from, and what the agent was thinking.

Prices keep themselves current with what g1t pays1//! Keeps every price current with what g1t actually pays.
2//!
3//! g1t passes its own costs through, so a price is only right while the
4//! cost under it is. Two jobs keep them right, on the billing service's
5//! cron:
6//!
7//! - **Settling runs** (every 15 minutes). A model run is charged when it
8//! finishes at what the sandbox reported. Each of g1t's hosted runs goes
9//! through its AI Gateway, which prices every request at the provider's
10//! current rates and logs it with the run's session. Settling sums those
11//! logs and corrects the charge to the gateway's figure, with a
12//! correction on the statement. A run whose sandbox died before
13//! reporting is charged here instead of never.
14//! - **Checking costs** (daily). What Cloudflare billed g1t's account, from
15//! its usage API, is measured against how much was used: Containers
16//! against the seconds containers ran, Workers for Platforms per request
17//! and per CPU millisecond. When a measured cost moves, the price book
18//! moves with it, since each price is its cost plus a set markup, and
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily19//! the change is recorded where anyone can see it. A measurement goes
20//! through `pricing` as a proposal: a small move is applied on its own (a
21//! rise only after customers have had notice), a large or suspect one
22//! waits for staff in sudo, so one odd day of data cannot reprice
23//! everything.
Prices keep themselves current with what g1t pays24
25use g1t_contracts::billing::{EntryKind, MICROS_PER_DOLLAR, Price, PriceBook, PriceChange};
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily26
Prices keep themselves current with what g1t pays27use g1t_contracts::time::rfc3339;
28use g1t_kit::now_ms;
29use serde::Deserialize;
30use serde_json::{Value, json};
31use worker::{Env, Fetch, Headers, Method, Request, RequestInit, Result};
32
33use crate::{Billing, RunRow, charge_micros};
34
35/// The cron that also checks costs against Cloudflare's bill.
36pub(crate) const DAILY: &str = "17 4 * * *";
37
38/// A run is settled once its logs have had time to land.
39const SETTLE_AFTER_MS: u64 = 5 * 60 * 1000;
40/// A run with no gateway logs after this is left as reported.
41const GIVE_UP_AFTER_MS: u64 = 3 * 60 * 60 * 1000;
42/// A run never finished after this died without reporting.
43const ABANDONED_AFTER_MS: u64 = 3 * 60 * 60 * 1000;
44
45/// Where the keeper reads what g1t pays.
46pub(crate) struct Keeper {
47 /// `CLOUDFLARE_USAGE_TOKEN`: Billing, Account Analytics and AI Gateway,
48 /// read only.
49 token: Option<String>,
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily50 /// What reads the bill for `costs`: `CLOUDFLARE_BILLING_TOKEN`
51 /// (Account: Billing Read and Account Analytics Read), or the usage
52 /// token, which has both.
53 billing_token: Option<String>,
Prices keep themselves current with what g1t pays54 account: String,
55 gateway: String,
56}
57
58impl Keeper {
59 pub(crate) fn from_env(env: &Env) -> Self {
60 let var = |name: &str| env.var(name).map(|v| v.to_string()).unwrap_or_default();
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily61 let secret = |name: &str| env.secret(name).ok().map(|v| v.to_string()).filter(|v| !v.is_empty());
62 let token = secret("CLOUDFLARE_USAGE_TOKEN");
Prices keep themselves current with what g1t pays63 Keeper {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily64 billing_token: secret("CLOUDFLARE_BILLING_TOKEN").or_else(|| token.clone()),
65 token,
Prices keep themselves current with what g1t pays66 account: var("CLOUDFLARE_ACCOUNT_ID"),
67 gateway: var("AI_GATEWAY_ID"),
68 }
69 }
70
71 async fn send(&self, method: Method, url: &str, body: Option<Value>) -> Result<Value> {
72 let Some(token) = &self.token else {
73 return Err(worker::Error::RustError("no CLOUDFLARE_USAGE_TOKEN".into()));
74 };
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily75 send_with(token, method, url, body).await
76 }
77
78 /// Whether Cloudflare's bill can be read.
79 pub(crate) fn can_read_bill(&self) -> bool {
80 self.billing_token.is_some() && !self.account.is_empty()
81 }
82
83 fn billing_token(&self) -> Result<&str> {
84 self.billing_token
85 .as_deref()
86 .ok_or_else(|| worker::Error::RustError("no CLOUDFLARE_BILLING_TOKEN or CLOUDFLARE_USAGE_TOKEN".into()))
87 }
88
89 /// Billable usage from `from` to `to` (dates), as Cloudflare answers it.
90 pub(crate) async fn billable_usage_body(&self, from: &str, to: &str) -> Result<Value> {
91 send_with(self.billing_token()?, Method::Get, &self.api(&format!("/billable-usage?from={from}&to={to}")), None).await
Prices keep themselves current with what g1t pays92 }
93
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily94 /// A GraphQL Analytics query, as Cloudflare answers it, errors and all.
95 pub(crate) async fn graphql(&self, body: Value) -> Result<Value> {
96 send_with(self.billing_token()?, Method::Post, "https://api.cloudflare.com/client/v4/graphql", Some(body)).await
97 }
98
99 pub(crate) fn account(&self) -> &str {
100 &self.account
101 }
102
Prices keep themselves current with what g1t pays103 fn api(&self, path: &str) -> String {
104 format!("https://api.cloudflare.com/client/v4/accounts/{}{path}", self.account)
105 }
106
107 /// What AI Gateway priced a session's requests at, in dollars, and how
108 /// many there were.
109 async fn session_cost(&self, session: &str) -> Result<(f64, u32)> {
110 let mut cost = 0.0;
111 let mut count = 0;
112 for page in 1..=40 {
The keeper reads Cloudflare as it really answers113 // The filter goes as URL-encoded JSON; the bracket form is
114 // ignored, and would sum every log there is. Session ids are
115 // [a-z0-9_], which need no escaping inside it.
116 let filter = format!(
117 "%5B%7B%22key%22%3A%22metadata.value%22%2C%22operator%22%3A%22eq%22%2C%22value%22%3A%5B%22{session}%22%5D%7D%5D"
118 );
Prices keep themselves current with what g1t pays119 let url = self.api(&format!(
The keeper reads Cloudflare as it really answers120 "/ai-gateway/gateways/{}/logs?per_page=50&page={page}&filters={filter}",
Prices keep themselves current with what g1t pays121 self.gateway
122 ));
123 let body = self.send(Method::Get, &url, None).await?;
124 let logs = body["result"].as_array().cloned().unwrap_or_default();
125 for log in &logs {
126 cost += log["cost"].as_f64().unwrap_or(0.0);
127 count += 1;
128 }
129 if logs.len() < 50 {
130 break;
131 }
132 }
133 Ok((cost, count))
134 }
135
The keeper reads Cloudflare as it really answers136 /// The account's billable usage, one row per service per day, as
137 /// Cloudflare reports it.
Prices keep themselves current with what g1t pays138 async fn billable_usage(&self, from: &str, to: &str) -> Result<Vec<UsageRow>> {
139 let body = self
The keeper reads Cloudflare as it really answers140 .send(Method::Get, &self.api(&format!("/billable-usage?from={from}&to={to}")), None)
Prices keep themselves current with what g1t pays141 .await?;
142 let rows = body["result"].as_array().cloned().unwrap_or_default();
143 Ok(rows.iter().filter_map(UsageRow::from_value).collect())
144 }
145
The keeper reads Cloudflare as it really answers146 /// What g1t's containers used from `since` to `until` (dates), as
147 /// Cloudflare bills it: memory in byte-seconds, and CPU seconds.
148 async fn container_usage(&self, since: &str, until: &str) -> Result<ContainerUsage> {
149 let query = "query ($account: String!, $since: Date!, $until: Date!) {
Prices keep themselves current with what g1t pays150 viewer { accounts(filter: { accountTag: $account }) {
The keeper reads Cloudflare as it really answers151 containersUsageAdaptiveGroups(limit: 1000, filter: { date_geq: $since, date_leq: $until }) {
152 sum { cpuTimeSec allocatedMemory }
Prices keep themselves current with what g1t pays153 }
154 } }
155 }";
156 let body = self
157 .send(
158 Method::Post,
159 "https://api.cloudflare.com/client/v4/graphql",
160 Some(json!({ "query": query, "variables": { "account": self.account, "since": since, "until": until } })),
161 )
162 .await?;
The keeper reads Cloudflare as it really answers163 let groups = body["data"]["viewer"]["accounts"][0]["containersUsageAdaptiveGroups"]
Prices keep themselves current with what g1t pays164 .as_array()
165 .cloned()
166 .unwrap_or_default();
The keeper reads Cloudflare as it really answers167 Ok(groups.iter().fold(ContainerUsage::default(), |total, g| ContainerUsage {
168 cpu_seconds: total.cpu_seconds + g["sum"]["cpuTimeSec"].as_f64().unwrap_or(0.0),
169 memory_byte_seconds: total.memory_byte_seconds + g["sum"]["allocatedMemory"].as_f64().unwrap_or(0.0),
170 }))
Prices keep themselves current with what g1t pays171 }
172}
173
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily174/// A request to Cloudflare's API with a bearer token; anything but 200 is
175/// an error with what Cloudflare said.
176async fn send_with(token: &str, method: Method, url: &str, body: Option<Value>) -> Result<Value> {
177 let headers = Headers::new();
178 headers.set("authorization", &format!("Bearer {token}"))?;
179 headers.set("content-type", "application/json")?;
180 let mut init = RequestInit::new();
181 init.with_method(method).with_headers(headers);
182 if let Some(body) = body {
183 init.with_body(Some(body.to_string().into()));
184 }
185 let mut response = Fetch::Request(Request::new_with_init(url, &init)?).send().await?;
186 let status = response.status_code();
187 let value: Value = response.json().await.unwrap_or(Value::Null);
188 if status != 200 {
189 return Err(worker::Error::RustError(format!("Cloudflare answered {status}: {value}")));
190 }
191 Ok(value)
192}
193
The keeper reads Cloudflare as it really answers194#[derive(Debug, Default, Clone, Copy)]
195pub(crate) struct ContainerUsage {
196 cpu_seconds: f64,
197 memory_byte_seconds: f64,
198}
199
200/// g1t's sandboxes: Containers' standard-1, half a vCPU, 4 GiB, 8 GB disk.
201const SANDBOX_GIB: f64 = 4.0;
202const SANDBOX_DISK_GB: f64 = 8.0;
203const GIB: f64 = 1024.0 * 1024.0 * 1024.0;
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look204/// The Durable Object behind each container is billed for as long as the
205/// container runs, at 128 MB.
206const SANDBOX_DO_GB: f64 = 0.125;
The keeper reads Cloudflare as it really answers207
208/// Cloudflare's published Containers rates, in dollars, used for any rate
209/// the bill does not show yet (while usage is inside the included amount).
210const LIST_MEMORY_GIB_SECOND: f64 = 0.000_002_5;
211const LIST_DISK_GB_SECOND: f64 = 0.000_000_07;
212const LIST_VCPU_SECOND: f64 = 0.000_02;
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look213/// Durable Objects duration: $12.50 per million GB-seconds.
214const LIST_DO_GB_SECOND: f64 = 0.000_012_5;
The keeper reads Cloudflare as it really answers215
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look216/// What one second of a sandbox costs whatever it does, in millionths of a
217/// dollar: its memory and disk, and the Durable Object behind it, for the
218/// whole second. CPU is billed only while busy, on top.
219pub(crate) fn sandbox_base_micros(memory: f64, disk: f64, durable_object: f64) -> f64 {
220 (SANDBOX_GIB * memory + SANDBOX_DISK_GB * disk + SANDBOX_DO_GB * durable_object) * MICROS_PER_DOLLAR as f64
221}
222
223/// What one second of a sandbox costs on average, in millionths of a
224/// dollar: its base, and the CPU sandboxes actually use per second of
225/// running. Runs that report their own CPU are priced on it instead (see
226/// `run_cost`).
227pub(crate) fn sandbox_second_micros(usage: ContainerUsage, memory: f64, disk: f64, vcpu: f64, durable_object: f64) -> Option<f64> {
The keeper reads Cloudflare as it really answers228 let instance_seconds = usage.memory_byte_seconds / (SANDBOX_GIB * GIB);
229 if instance_seconds < 3600.0 {
230 return None;
231 }
232 let cpu_share = usage.cpu_seconds / instance_seconds;
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look233 Some(sandbox_base_micros(memory, disk, durable_object) + cpu_share * vcpu * MICROS_PER_DOLLAR as f64)
234}
235
Fast pages, required checks on the branch, self-hosted runners, honest incidents236/// How much more a second of a larger machine's memory and disk (and the
237/// Durable Object behind it) costs than the standard sandbox's, at
238/// Cloudflare's list rates: 1 for the standard machine.
239pub(crate) fn base_scale(memory_gib: f64, disk_gb: f64) -> f64 {
240 let base = |memory: f64, disk: f64| memory * LIST_MEMORY_GIB_SECOND + disk * LIST_DISK_GB_SECOND + SANDBOX_DO_GB * LIST_DO_GB_SECOND;
241 base(memory_gib, disk_gb) / base(SANDBOX_GIB, SANDBOX_DISK_GB)
242}
243
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look244/// What a run that reported its own CPU cost g1t: its base for every
245/// second, and its vCPU-seconds at the vCPU rate.
246pub(crate) fn run_cost(seconds: i64, cpu_seconds: f64, base_per_second: f64, per_vcpu_second: f64) -> f64 {
247 seconds.max(0) as f64 * base_per_second + cpu_seconds.max(0.0) * per_vcpu_second
The keeper reads Cloudflare as it really answers248}
249
250/// A unit's marginal rate from the bill: the median, over the days that
251/// were charged, of cost over quantity. None while nothing was charged.
252pub(crate) fn billed_rate(rows: &[&UsageRow]) -> Option<f64> {
253 let mut rates: Vec<f64> = rows
254 .iter()
255 .filter(|r| r.cost > 0.0 && r.quantity > 0.0)
256 .map(|r| r.cost / r.quantity)
257 .collect();
258 if rates.is_empty() {
259 return None;
260 }
261 rates.sort_by(f64::total_cmp);
262 Some(rates[rates.len() / 2])
263}
264
Prices keep themselves current with what g1t pays265/// One line of Cloudflare's billable usage.
266#[derive(Debug, Clone)]
267pub(crate) struct UsageRow {
268 period_start: String,
269 period_end: String,
270 service: String,
271 unit: String,
272 quantity: f64,
273 cost: f64,
274}
275
276impl UsageRow {
277 /// Read leniently: the API is new, and its field names are FOCUS's.
278 fn from_value(row: &Value) -> Option<Self> {
279 let text = |keys: &[&str]| keys.iter().find_map(|k| row[*k].as_str()).unwrap_or_default().to_owned();
280 let number = |keys: &[&str]| {
281 keys.iter()
282 .find_map(|k| row[*k].as_f64().or_else(|| row[*k].as_str().and_then(|s| s.parse().ok())))
283 .unwrap_or(0.0)
284 };
285 let service = text(&["ServiceName", "service_name", "service"]);
286 if service.is_empty() {
287 return None;
288 }
289 let family = text(&["ServiceFamilyName", "service_family_name"]);
290 Some(UsageRow {
291 period_start: text(&["ChargePeriodStart", "charge_period_start"]),
292 period_end: text(&["ChargePeriodEnd", "charge_period_end"]),
293 service: if family.is_empty() { service } else { format!("{family} / {service}") },
The keeper reads Cloudflare as it really answers294 unit: text(&["PricingUnit", "ConsumedUnit", "consumed_unit"]),
Prices keep themselves current with what g1t pays295 quantity: number(&["PricingQuantity", "ConsumedQuantity", "pricing_quantity"]),
The keeper reads Cloudflare as it really answers296 // What g1t pays; list price if nothing was contracted.
297 cost: Some(number(&["ContractedCost", "BilledCost", "contracted_cost"]))
298 .filter(|cost| *cost > 0.0)
299 .unwrap_or_else(|| number(&["ListCost", "list_cost"])),
Prices keep themselves current with what g1t pays300 })
301 }
302}
303
304#[derive(Deserialize)]
305struct PriceRow {
306 meter: String,
307 title: String,
308 unit: String,
309 cost_micros: f64,
310 markup_percent: u32,
311 source: String,
312 checked_at: Option<String>,
313 updated_at: String,
314}
315
316#[derive(Deserialize)]
317struct ChangeRow {
318 meter: String,
319 old_cost_micros: f64,
320 new_cost_micros: f64,
321 markup_percent: u32,
Prices are what g1t pays plus 20%, from the first second322 old_markup_percent: Option<u32>,
Prices keep themselves current with what g1t pays323 reason: String,
324 created_at: String,
325}
326
327#[derive(Deserialize)]
328struct Unsettled {
329 id: String,
330 workspace: String,
331 repo: String,
332 number: u32,
333 task: String,
334 model: String,
335 token_hash: String,
336 billed_to: Option<String>,
337 session_id: String,
338 created_at: String,
339 finished_at: Option<String>,
340}
341
342#[derive(Deserialize)]
343struct Charged {
344 cost_micros: Option<i64>,
345 description: String,
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put346 amount_micros: i64,
347}
348
349/// What a settled run's correction comes to, from the charges at the
350/// reported and the gateway's cost and what the workspace was charged at
351/// first. A charge up is drawn down like any charge; a charge down is
352/// given back only up to what the workspace paid, since what a credit or
353/// a pool paid was never the workspace's money.
354pub(crate) fn correction(reported_charge: i64, gateway_charge: i64, first_charged: i64) -> i64 {
355 let delta = gateway_charge - reported_charge;
356 if delta >= 0 { delta } else { delta.max(-first_charged.max(0)) }
Prices keep themselves current with what g1t pays357}
358
359fn ms(timestamp: &str) -> u64 {
360 // RFC 3339 in UTC, as g1t writes them.
361 worker::js_sys::Date::parse(timestamp) as u64
362}
363
364impl Billing {
365 pub(crate) async fn prices(&self) -> Result<PriceBook> {
366 let prices = self
367 .db
368 .prepare("SELECT * FROM prices ORDER BY rowid")
369 .all()
370 .await?
371 .results::<PriceRow>()?;
372 let changes = self
373 .db
374 .prepare("SELECT * FROM price_changes ORDER BY created_at DESC LIMIT 20")
375 .all()
376 .await?
377 .results::<ChangeRow>()?;
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily378 let mut plans = Vec::new();
379 for feature in g1t_contracts::billing::Feature::ALL.iter() {
380 plans.push(self.plan(*feature).await?);
381 }
382 // Changes still to come first, so a rise is seen before it is charged.
383 let coming = self.coming_changes().await?;
Prices keep themselves current with what g1t pays384 Ok(PriceBook {
385 prices: prices
386 .into_iter()
387 .map(|row| Price {
388 price_micros: Price::price_for(row.cost_micros, row.markup_percent),
389 meter: row.meter,
390 title: row.title,
391 unit: row.unit,
392 cost_micros: row.cost_micros,
393 markup_percent: row.markup_percent,
394 source: row.source,
395 checked_at: row.checked_at,
396 updated_at: row.updated_at,
397 })
398 .collect(),
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily399 changes: coming
Prices keep themselves current with what g1t pays400 .into_iter()
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily401 .chain(changes.into_iter().map(|row| PriceChange {
Prices keep themselves current with what g1t pays402 meter: row.meter,
403 old_cost_micros: row.old_cost_micros,
404 new_cost_micros: row.new_cost_micros,
405 markup_percent: row.markup_percent,
Prices are what g1t pays plus 20%, from the first second406 old_markup_percent: row.old_markup_percent,
Prices keep themselves current with what g1t pays407 reason: row.reason,
408 created_at: row.created_at,
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily409 effective_at: None,
410 }))
Prices keep themselves current with what g1t pays411 .collect(),
412 model_margin_percent: self.margin_percent,
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily413 plans,
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put414 free: Some(g1t_contracts::billing::FreeTier {
415 trial_workspace_micros: if self.trials_on { self.plans.trial_workspace_micros } else { 0 },
416 trial_monthly_pool_micros: if self.trials_on { self.plans.trial_monthly_pool_micros } else { 0 },
417 oss_pool_micros: self.plans.oss_pool_micros,
418 oss_repo_micros: self.plans.oss_repo_micros,
419 free_private_storage_bytes: self.plans.free_storage_bytes,
420 audit_retention_days: self.plans.audit_days,
421 min_charge_micros: self.plans.min_charge_micros,
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look422 git_operations_included: self.plans.git_included,
423 paid_start_ceiling_micros: self.plans.paid_start_micros,
424 overage_forgive_cost_micros: self.plans.forgive_cost_micros,
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put425 }),
Prices keep themselves current with what g1t pays426 })
427 }
428
The keeper reads Cloudflare as it really answers429 /// Whether the costs have never been checked against Cloudflare's bill.
430 pub(crate) async fn never_checked(&self) -> Result<bool> {
431 Ok(self
432 .db
433 .prepare("SELECT meter FROM prices WHERE checked_at IS NOT NULL LIMIT 1")
434 .first::<Value>(None)
435 .await?
436 .is_none())
437 }
438
Prices keep themselves current with what g1t pays439 /// A meter's cost and price per unit, from the book.
440 pub(crate) async fn price(&self, meter: &str) -> Result<Option<(f64, f64)>> {
441 #[derive(Deserialize)]
442 struct Row {
443 cost_micros: f64,
444 markup_percent: u32,
445 }
446 Ok(self
447 .db
448 .prepare("SELECT cost_micros, markup_percent FROM prices WHERE meter = ?")
449 .bind(&[meter.into()])?
450 .first::<Row>(None)
451 .await?
452 .map(|row| (row.cost_micros, Price::price_for(row.cost_micros, row.markup_percent))))
453 }
454
455 /// Corrects finished runs to what AI Gateway priced them at, and
456 /// charges runs whose sandbox died before reporting.
457 pub(crate) async fn settle_runs(&self, keeper: &Keeper) -> Result<()> {
458 if keeper.token.is_none() || keeper.gateway.is_empty() {
459 return Ok(());
460 }
461 let now = now_ms();
462 let runs = self
463 .db
464 .prepare(
465 "SELECT id, workspace, repo, number, task, model, token_hash, billed_to, session_id, created_at, finished_at
466 FROM runs
467 WHERE session_id IS NOT NULL AND settled_at IS NULL
468 AND ((finished_at IS NOT NULL AND finished_at < ?1) OR created_at < ?2)
469 ORDER BY created_at LIMIT 10",
470 )
471 .bind(&[rfc3339(now - SETTLE_AFTER_MS).into(), rfc3339(now - ABANDONED_AFTER_MS).into()])?
472 .all()
473 .await?
474 .results::<Unsettled>()?;
475 for run in runs {
476 let (cost_usd, requests) = match keeper.session_cost(&run.session_id).await {
477 Ok(found) => found,
478 Err(error) => {
479 worker::console_error!("could not read gateway logs for {}: {error}", run.id);
480 continue;
481 }
482 };
483 let since = ms(run.finished_at.as_deref().unwrap_or(&run.created_at));
484 if requests == 0 && now.saturating_sub(since) < GIVE_UP_AFTER_MS {
485 continue;
486 }
487 self.settle(&run, cost_usd, requests).await?;
488 }
489 Ok(())
490 }
491
492 async fn settle(&self, run: &Unsettled, cost_usd: f64, requests: u32) -> Result<()> {
493 let row = RunRow {
494 workspace: run.workspace.clone(),
495 repo: run.repo.clone(),
496 number: run.number,
497 task: run.task.clone(),
498 model: run.model.clone(),
499 token_hash: run.token_hash.clone(),
500 billed_to: run.billed_to.clone(),
501 };
502 let gateway_micros = charge_micros(cost_usd, 0);
503 let charged = self
504 .db
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put505 .prepare("SELECT cost_micros, description, amount_micros FROM ledger WHERE reference = ?")
Prices keep themselves current with what g1t pays506 .bind(&[run.id.as_str().into()])?
507 .first::<Charged>(None)
508 .await?;
Billing accounts, terms and enterprises; g1t is no longer free509 let terms = self.terms_of(&run.workspace).await?;
Prices keep themselves current with what g1t pays510 let charge_for = |micros: i64| {
511 if self.free {
512 0
513 } else {
Billing accounts, terms and enterprises; g1t is no longer free514 terms.apply(charge_micros(micros as f64 / MICROS_PER_DOLLAR as f64, self.margin_percent))
Prices keep themselves current with what g1t pays515 }
516 };
517 let settled_at = rfc3339(now_ms());
518 // Claim it, so two crons never settle it twice.
519 let claimed = self
520 .db
521 .prepare("UPDATE runs SET settled_at = ?, gateway_cost_micros = ?, finished_at = COALESCE(finished_at, ?) WHERE id = ? AND settled_at IS NULL RETURNING id")
522 .bind(&[
523 settled_at.as_str().into(),
524 (gateway_micros as f64).into(),
525 settled_at.as_str().into(),
526 run.id.as_str().into(),
527 ])?
528 .first::<Value>(None)
529 .await?;
530 if claimed.is_none() || requests == 0 {
531 return Ok(());
532 }
533 let free_note = if self.free { " (free while g1t is being built out)" } else { "" };
534 match charged {
535 // Never reported: charged now, from the gateway's figure.
536 None => {
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put537 let charge = charge_for(gateway_micros);
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look538 let eligible = crate::credits::eligible_for(Some(g1t_contracts::billing::ComputeKind::Agent), None);
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put539 let drawn = self.draw(&run.workspace, charge, &settled_at[..7], &eligible).await?;
Prices keep themselves current with what g1t pays540 let description = format!(
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put541 "Work on {}#{}, settled from AI Gateway after the sandbox stopped without reporting{free_note}{}",
542 run.repo,
543 run.number,
544 drawn.note()
Prices keep themselves current with what g1t pays545 );
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put546 self.enter(&run.workspace, EntryKind::Usage, -(charge - drawn.total()), &description, &run.id, Some(&row), Some(gateway_micros), None, None)
Prices keep themselves current with what g1t pays547 .await?;
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put548 self.record_drawn(&run.id, &drawn).await?;
Prices keep themselves current with what g1t pays549 }
550 Some(charged) => {
551 let reported = charged.cost_micros.unwrap_or(0);
552 let delta = gateway_micros - reported;
553 if delta == 0 {
554 return Ok(());
555 }
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put556 let change = correction(charge_for(reported), charge_for(gateway_micros), -charged.amount_micros);
557 // A charge up is paid for like any other charge.
558 let drawn = if change > 0 {
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look559 let eligible = crate::credits::eligible_for(Some(g1t_contracts::billing::ComputeKind::Agent), None);
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put560 self.draw(&run.workspace, change, &settled_at[..7], &eligible).await?
561 } else {
562 crate::credits::Drawn::default()
563 };
Prices keep themselves current with what g1t pays564 let description = format!(
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put565 "Correction to “{}”: AI Gateway priced its {requests} model requests at {}, not {}{}",
Prices keep themselves current with what g1t pays566 charged.description,
567 crate::features::dollars(gateway_micros),
568 crate::features::dollars(reported),
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put569 drawn.note(),
Prices keep themselves current with what g1t pays570 );
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put571 let reference = format!("{}/settled", run.id);
Prices keep themselves current with what g1t pays572 self.enter(
573 &run.workspace,
574 EntryKind::Usage,
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put575 -(change - drawn.total()),
Prices keep themselves current with what g1t pays576 &description,
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put577 &reference,
Prices keep themselves current with what g1t pays578 Some(&row),
579 Some(delta),
580 None,
581 None,
582 )
583 .await?;
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put584 self.record_drawn(&reference, &drawn).await?;
Prices keep themselves current with what g1t pays585 }
586 }
587 Ok(())
588 }
589
590 /// Checks each cost against what Cloudflare billed this month, and
591 /// moves the ones that changed.
592 pub(crate) async fn reconcile(&self, keeper: &Keeper) -> Result<()> {
593 if keeper.token.is_none() {
594 return Ok(());
595 }
596 let now = rfc3339(now_ms());
597 let today = &now[..10];
The keeper reads Cloudflare as it really answers598 let since = rfc3339(now_ms() - 30 * 24 * 60 * 60 * 1000);
599 let rows = keeper.billable_usage(&since[..10], today).await?;
Prices keep themselves current with what g1t pays600 for row in &rows {
601 self.db
602 .prepare(
603 "INSERT INTO cloudflare_usage (period_start, period_end, service, unit, quantity, cost_usd, fetched_at)
604 VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
605 ON CONFLICT (period_start, service, unit) DO UPDATE SET
606 period_end = ?2, quantity = ?5, cost_usd = ?6, fetched_at = ?7",
607 )
608 .bind(&[
609 row.period_start.as_str().into(),
610 row.period_end.as_str().into(),
611 row.service.as_str().into(),
612 row.unit.as_str().into(),
613 row.quantity.into(),
614 row.cost.into(),
615 now.as_str().into(),
616 ])?
617 .run()
618 .await?;
619 }
620
The keeper reads Cloudflare as it really answers621 let named = |words: &[&str]| -> Vec<&UsageRow> {
Prices keep themselves current with what g1t pays622 rows.iter()
The keeper reads Cloudflare as it really answers623 .filter(|r| {
624 let service = r.service.to_lowercase();
625 words.iter().all(|word| service.contains(word))
626 })
627 .collect()
Prices keep themselves current with what g1t pays628 };
629
The keeper reads Cloudflare as it really answers630 // Containers: each resource at what the bill shows it costs, or
631 // the published rate while the included amount still covers it,
632 // over how much CPU g1t's sandboxes really use per second.
633 let memory = billed_rate(&named(&["container memory"]));
634 let disk = billed_rate(&named(&["container disk"]));
635 let vcpu = billed_rate(&named(&["container vcpu"]));
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look636 let durable_object = billed_rate(&named(&["durable objects", "duration"]));
The keeper reads Cloudflare as it really answers637 let usage = keeper.container_usage(&since[..10], today).await?;
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look638 let rates = (
The keeper reads Cloudflare as it really answers639 memory.unwrap_or(LIST_MEMORY_GIB_SECOND),
640 disk.unwrap_or(LIST_DISK_GB_SECOND),
641 vcpu.unwrap_or(LIST_VCPU_SECOND),
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look642 durable_object.unwrap_or(LIST_DO_GB_SECOND),
643 );
644 // The parts, for runs that report their own CPU.
645 let parts_reason = "Cloudflare's Containers and Durable Objects rates, as billed or published";
646 self.measure("sandbox_base_second", sandbox_base_micros(rates.0, rates.1, rates.3), parts_reason).await?;
647 self.measure("sandbox_cpu_second", rates.2 * MICROS_PER_DOLLAR as f64, parts_reason).await?;
648 if let Some(per_second) = sandbox_second_micros(usage, rates.0, rates.1, rates.2, rates.3) {
The keeper reads Cloudflare as it really answers649 let instance_seconds = usage.memory_byte_seconds / (SANDBOX_GIB * GIB);
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look650 let billed = [("memory", memory), ("disk", disk), ("vCPU", vcpu), ("Durable Object duration", durable_object)]
The keeper reads Cloudflare as it really answers651 .iter()
652 .filter(|(_, rate)| rate.is_some())
653 .map(|(name, _)| *name)
654 .collect::<Vec<_>>();
655 let reason = format!(
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look656 "Sandboxes used {:.2} vCPU per second over {:.0} hours of Cloudflare Containers in the last 30 days, with the Durable Object behind each; {}",
The keeper reads Cloudflare as it really answers657 usage.cpu_seconds / instance_seconds,
658 instance_seconds / 3600.0,
659 if billed.is_empty() {
660 "rates are Cloudflare's published ones".to_owned()
661 } else {
662 format!("{} at what Cloudflare billed", billed.join(", "))
663 },
664 );
665 for meter in ["sandbox_second", "build_second"] {
666 self.measure(meter, per_second, &reason).await?;
Prices keep themselves current with what g1t pays667 }
668 }
The keeper reads Cloudflare as it really answers669 // Apps run as Workers: per million requests and CPU milliseconds,
670 // once the bill shows them charged.
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put671 // Security scans' CPU follows the same Workers CPU rate.
672 let app_meters: [(&str, &[&str], &str); 3] = [
The keeper reads Cloudflare as it really answers673 ("app_requests", &["workers", "requests"], "requests"),
674 ("app_cpu", &["workers cpu"], "CPU ms"),
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put675 ("scan_cpu", &["workers cpu"], "CPU ms"),
The keeper reads Cloudflare as it really answers676 ];
677 for (meter, words, unit) in app_meters {
678 if let Some(rate) = billed_rate(&named(words)) {
679 let reason = format!("Cloudflare billed Workers {unit} at ${:.2} per million", rate * 1e6);
680 self.measure(meter, rate * 1e6 * MICROS_PER_DOLLAR as f64, &reason).await?;
Prices keep themselves current with what g1t pays681 }
682 }
683 self.db
684 .prepare("UPDATE prices SET checked_at = ?")
685 .bind(&[now.as_str().into()])?
686 .run()
687 .await?;
688 Ok(())
689 }
690
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily691 /// Proposes moving a meter's cost to a measurement (see `pricing`):
692 /// applied on its own when small, after notice when a rise; left for
693 /// staff when large or suspect.
Prices keep themselves current with what g1t pays694 async fn measure(&self, meter: &str, measured: f64, reason: &str) -> Result<()> {
Git storage hardened, pages in tens of milliseconds, honest security alerts, and costs reconciled daily695 if let Some(outcome) = self.propose(meter, measured, reason, "keeper").await? {
696 worker::console_log!("{meter}: {outcome}");
Prices keep themselves current with what g1t pays697 }
698 Ok(())
699 }
700}
701
702#[cfg(test)]
703mod tests {
704 use super::*;
705
706 #[test]
Team plan, an open-source pool, monthly trials and honest metering; the sidebar for everyone; a workspace that stays put707 fn a_correction_never_gives_back_what_the_workspace_did_not_pay() {
708 // Up by 2 cents: charged in full (then drawn down like any charge).
709 assert_eq!(correction(100_000, 120_000, 100_000), 20_000);
710 // Down by 2 cents, all of it paid by the workspace: given back.
711 assert_eq!(correction(120_000, 100_000, 120_000), -20_000);
712 // Down, but the open-source pool paid all but a cent: a cent back.
713 assert_eq!(correction(120_000, 100_000, 10_000), -10_000);
714 // Paid entirely by a credit or pool: nothing back.
715 assert_eq!(correction(120_000, 100_000, 0), 0);
Prices keep themselves current with what g1t pays716 }
717
718 #[test]
The keeper reads Cloudflare as it really answers719 fn a_sandbox_second_is_its_memory_and_disk_and_the_cpu_it_uses() {
720 // An hour of sandboxes that kept a fifth of a vCPU busy.
721 let usage = ContainerUsage { cpu_seconds: 720.0, memory_byte_seconds: 3600.0 * 4.0 * GIB };
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look722 let micros =
723 sandbox_second_micros(usage, LIST_MEMORY_GIB_SECOND, LIST_DISK_GB_SECOND, LIST_VCPU_SECOND, LIST_DO_GB_SECOND).unwrap();
724 // 4 x 2.5 + 8 x 0.07 + 0.125 x 12.5 + 0.2 x 20 = 16.1225
725 assert!((micros - 16.1225).abs() < 1e-9, "{micros}");
726 // The Durable Object adds about 11% to the second it left out.
727 assert!((sandbox_base_micros(LIST_MEMORY_GIB_SECOND, LIST_DISK_GB_SECOND, LIST_DO_GB_SECOND) - 12.1225).abs() < 1e-9);
The keeper reads Cloudflare as it really answers728 // Too little use to say anything.
Invite-only launch: sign in with GitHub, repository access and lifecycle, many emails, a new look729 assert!(sandbox_second_micros(ContainerUsage { cpu_seconds: 1.0, memory_byte_seconds: GIB }, 1.0, 1.0, 1.0, 1.0).is_none());
730 }
731
732 #[test]
733 fn a_run_that_reports_its_cpu_is_priced_on_it() {
734 let base = sandbox_base_micros(LIST_MEMORY_GIB_SECOND, LIST_DISK_GB_SECOND, LIST_DO_GB_SECOND);
735 let vcpu = LIST_VCPU_SECOND * MICROS_PER_DOLLAR as f64;
736 // A 10-minute cargo build that kept its half vCPU busy throughout.
737 let heavy = run_cost(600, 300.0, base, vcpu);
738 assert!((heavy - (600.0 * 12.1225 + 300.0 * 20.0)).abs() < 1e-6);
739 // The same ten minutes, mostly idle, costs less.
740 let light = run_cost(600, 30.0, base, vcpu);
741 assert!(light < heavy);
742 // The average would have under-priced the heavy one.
743 let average = 600.0 * (base + 0.195 * vcpu);
744 assert!(average < heavy && average > light);
745 assert_eq!(run_cost(0, -1.0, base, vcpu), 0.0);
Fast pages, required checks on the branch, self-hosted runners, honest incidents746 // A larger machine's base: its memory and disk, not its CPU.
747 assert!((base_scale(SANDBOX_GIB, SANDBOX_DISK_GB) - 1.0).abs() < 1e-12);
748 assert!((base_scale(12.0, 20.0) - 2.72).abs() < 0.01, "{}", base_scale(12.0, 20.0));
749 assert!((base_scale(8.0, 16.0) - 1.87).abs() < 0.01, "{}", base_scale(8.0, 16.0));
The keeper reads Cloudflare as it really answers750 }
751
752 #[test]
753 fn a_billed_rate_is_the_median_of_the_charged_days() {
754 let row = |quantity: f64, cost: f64| UsageRow {
755 period_start: String::new(),
756 period_end: String::new(),
757 service: "Containers / Container Memory".into(),
758 unit: "Count".into(),
759 quantity,
760 cost,
761 };
762 let rows = [row(100.0, 0.0), row(100.0, 0.0002), row(100.0, 0.00025), row(100.0, 0.00025)];
763 assert_eq!(billed_rate(&rows.iter().collect::<Vec<_>>()), Some(0.000_002_5));
764 assert_eq!(billed_rate(&[&row(5.0, 0.0)]), None);
765 }
766
767 #[test]
Prices keep themselves current with what g1t pays768 fn usage_rows_are_read_by_their_focus_names() {
769 let row = UsageRow::from_value(&json!({
770 "ServiceFamilyName": "Containers",
771 "ServiceName": "Memory",
The keeper reads Cloudflare as it really answers772 "PricingUnit": "GiB-seconds",
Prices keep themselves current with what g1t pays773 "PricingQuantity": "1200.5",
774 "ContractedCost": 0.003,
775 "ChargePeriodStart": "2026-10-01",
776 }))
777 .unwrap();
778 assert_eq!(row.service, "Containers / Memory");
779 assert_eq!(row.quantity, 1200.5);
780 assert_eq!(row.cost, 0.003);
781 assert!(UsageRow::from_value(&json!({ "nothing": 1 })).is_none());
782 }
783}