Skip to content
Sign up

flagon-io/g1t

public
00

Where people and agents ship software together. The open-source git platform for the whole job: issues, agents, checks and deploys to the edge.

g1t.sh
gitagentscloudflarerustforgemcp

Commit

A free workspace's projects send up to 1 GB of observability a month, held a day at a time. OTLP ingest was unmetered and needed no plan, so a free project's token could push 8 MB batches without end into its store at g1t's cost. Each project's store now counts what arrives each UTC day (a usage_days table beside the monthly usage) and remembers its workspace, which the site passes when it shows the project's Observability settings or makes a token. The telemetry worker asks billing, kept ten minutes per isolate, whether that workspace is free; if so its project may send a thirtieth of FREE_TELEMETRY_BYTES_PER_MONTH (1 GiB, a wrangler var; 0 is no cap) a day. Past it, OTLP is answered 429 with the reason until midnight UTC, and Cloudflare's tail events of the project's apps keep their requests, so the charts go on, without their logs. Counted bytes are real: an OTLP body once unzipped, as before, and for tail events the rows as they are kept, where a flat 256-byte guess per request was counted. A project whose workspace is not known yet, or a billing that does not answer, is not capped. The telemetry worker gains a BILLING binding; Settings shows what arrived today; the observability guide says so.

syntaqxcommitted Parentcf68490Browse files
11 files+218−90/11 viewed
+15−0
238238 charts reach back further than its requests do. A project's store holds up
239239 to 4 GB; past that, its oldest day goes first.
240240
241+## On the free plan
242+
243+A project in a workspace on the free plan can send **1 GB a month**, held a
244+day at a time: about 34 MB a day (UTC), a thirtieth of the month, so a burst
245+cannot use the month up in an hour. What counts is what arrives: an OTLP
246+request's body once unzipped, and the rows kept from an app's tail events.
247+**Settings** shows what arrived this month and today.
248+
249+Past the day's share, OTLP requests are answered `429 Too Many Requests`
250+with the reason, until 00:00 UTC; OpenTelemetry SDKs retry or drop the
251+batch. Requests from an app on g1t.page still arrive, so its charts and
252+request list keep going, but without their logs. On the g1t plan there is
253+no such cap. OTLP is also limited to 600 requests a minute from one
254+address; see [rate limits](/reference/rate-limits/).
255+
241256 ## Who can do what
242257
243258 | | Who |
+4−2
3939
4040 export async function loader({ params, context, request }: Route.LoaderArgs) {
4141 const { projectId, service, project } = await observed(context, params, request, "manage_integrations");
42− const ref = { project_id: projectId };
42+ // The workspace goes along, so the project's store knows whose free cap applies.
43+ const ref = { project_id: projectId, workspace: (params.owner ?? "").toLowerCase() };
4344 const [settings, tokens, exports, usage] = await Promise.all([
4445 soft(service.settings(ref), "settings"),
4546 soft(service.tokens(ref), "tokens"),
6768 const { projectId, service } = await observed(context, params, request, "manage_integrations");
6869 const form = await request.formData();
6970 const intent = String(form.get("intent") ?? "");
70− const ref = { project_id: projectId };
71+ const ref = { project_id: projectId, workspace: (params.owner ?? "").toLowerCase() };
7172 try {
7273 switch (intent) {
7374 case "retention":
309310 {(
310311 [
311312 ["Received", bytes(usage.ingested_bytes)],
313+ ["Received today", bytes(usage.today_bytes ?? 0)],
312314 ["Requests", count(usage.requests)],
313315 ["Log lines", count(usage.logs)],
314316 ["Spans", count(usage.spans)],
+3−1
319319 spans: number;
320320 metric_points: number;
321321 stored_bytes: number;
322+ /** Bytes received today (UTC): a workspace without the plan sends a thirtieth of its monthly observability a day. */
323+ today_bytes?: number;
322324 };
323325
324326 /** Whom a call is for: the site and the API check access before they ask. */
325−type ProjectRef = { project_id: string };
327+type ProjectRef = { project_id: string; /** The project's workspace, by slug: the store remembers it, for a free workspace's cap. */ workspace?: string };
326328
327329 export interface TelemetryApi {
328330 overview(args: ProjectRef & TelemetryFilter & { step_s?: number | null }): Promise<TelemetryOverview>;
+48−3
2323 * project; and `GET /live?project_id=…`, a project's live tail.
2424 */
2525 import { type RateLimitBinding, clientAddress, isLimited } from "@g1t/contracts/rate-limits";
26+import { dailyCap, freeMonthlyBytes, overCapMessage } from "./quota.ts";
2627 import { TELEMETRY_TOKEN_PREFIX, type ErrorGroup } from "@g1t/contracts/telemetry";
2728
2829 import { emptyBatch, type Batch } from "./batch.ts";
4142 CLOUDFLARE_EXPORT_KEY?: string;
4243 /** OTLP requests per client address (RATE_LIMITS in packages/contracts). Fails open without it. */
4344 TELEMETRY_INGEST_LIMIT?: RateLimitBinding;
45+ /** Whether a project's workspace has the plan, for the free cap (./quota.ts). Without it, no cap. */
46+ BILLING?: { fetch: typeof fetch };
47+ /** A free workspace's project's observability a month, in bytes (./quota.ts); 0 is no cap. */
48+ FREE_TELEMETRY_BYTES_PER_MONTH?: string;
4449 };
4550
51+/** Each project's daily cap (null: none), kept ten minutes in this isolate. */
52+const caps = new Map<string, { at: number; cap: number | null }>();
53+const CAP_TTL_MS = 10 * 60_000;
54+
55+/**
56+ * The day's share a project may send, or null: only for a project whose
57+ * workspace (as the site last told its store) is on the free plan. A
58+ * project whose workspace is not known yet, or a billing that does not
59+ * answer, gets none, so nobody's telemetry stops on g1t's account.
60+ */
61+async function capFor(env: Env, projectId: string): Promise<number | null> {
62+ const cached = caps.get(projectId);
63+ if (cached && Date.now() - cached.at < CAP_TTL_MS) return cached.cap;
64+ let cap: number | null = null;
65+ const daily = dailyCap(freeMonthlyBytes(env.FREE_TELEMETRY_BYTES_PER_MONTH));
66+ if (daily && env.BILLING) {
67+ try {
68+ const workspace = await store(env, projectId).workspaceOf();
69+ if (workspace) {
70+ const { billingClient } = await import("@g1t/contracts");
71+ const free = await billingClient(env.BILLING as never).freeWorkspaces([workspace]);
72+ if (free.map((slug) => slug.toLowerCase()).includes(workspace)) cap = daily;
73+ }
74+ } catch (error) {
75+ console.error("could not tell whether the project's workspace is free", projectId, error);
76+ }
77+ }
78+ caps.set(projectId, { at: Date.now(), cap });
79+ if (caps.size > 5000) caps.delete(caps.keys().next().value!);
80+ return cap;
81+}
82+
4683 const DEPLOYMENTS_DOMAIN = "g1t.page";
4784
4885 /**
114151 byProject.set(project, into);
115152 }
116153 await Promise.all(
117− [...byProject].map(([project, { batch, halves }]) =>
154+ [...byProject].map(async ([project, { batch, halves }]) =>
118155 store(env, project)
119− .ingest(batch, halves)
156+ .ingest(batch, halves, undefined, await capFor(env, project))
120157 .catch((error: unknown) => console.error("could not keep tail events", project, error)),
121158 ),
122159 );
150187 if (error instanceof IngestError) return refused(error.status, error.message);
151188 throw error;
152189 }
153− const done = await store(env, project).ingest(read.batch, [], token);
190+ const cap = await capFor(env, project);
191+ const done = await store(env, project).ingest(read.batch, [], token, cap);
192+ if (!done.accepted && done.reason === "quota" && cap) return refused(429, overCapMessage(cap));
154193 if (!done.accepted) return refused(401, "That token is not one of the project's, or it was revoked.");
155194 return accepted(read.protobuf);
156195 }
202241 const projectId = String(args.project_id ?? "");
203242 if (!projectId) throw new IngestError(400, "No project_id.");
204243 const s = store(env, projectId);
244+ // The site says which workspace the project is in when it shows its
245+ // settings and makes a token: the store keeps it, for the free cap.
246+ if (typeof args.workspace === "string" && (method === "settings" || method === "create_token" || method === "usage")) {
247+ await s.setWorkspace(args.workspace);
248+ caps.delete(projectId);
249+ }
205250 switch (method) {
206251 case "overview":
207252 return s.overview(args);
+26−0
1+import assert from "node:assert/strict";
2+import { test } from "node:test";
3+
4+import { DEFAULT_FREE_TELEMETRY_BYTES_PER_MONTH, dailyCap, freeMonthlyBytes, overCap, overCapMessage, sizeOf } from "./quota.ts";
5+
6+test("a free workspace's project sends a thirtieth of 1 GB a day", () => {
7+ assert.equal(freeMonthlyBytes(undefined), DEFAULT_FREE_TELEMETRY_BYTES_PER_MONTH);
8+ assert.equal(freeMonthlyBytes(" "), DEFAULT_FREE_TELEMETRY_BYTES_PER_MONTH);
9+ assert.equal(freeMonthlyBytes("3000"), 3000);
10+ assert.equal(freeMonthlyBytes("0"), 0);
11+ assert.equal(dailyCap(3000), 100);
12+ assert.equal(dailyCap(0), null);
13+ assert.equal(dailyCap(DEFAULT_FREE_TELEMETRY_BYTES_PER_MONTH), 35_791_395);
14+});
15+
16+test("past the day's share is over, and no cap never is", () => {
17+ assert.equal(overCap(90, 10, 100), false);
18+ assert.equal(overCap(90, 11, 100), true);
19+ assert.equal(overCap(1e12, 1e12, null), false);
20+ assert.match(overCapMessage(35_791_395), /its 34 MB for today/);
21+});
22+
23+test("a row counts as what is kept of it", () => {
24+ assert.equal(sizeOf({ body: "hello" }), '{"body":"hello"}'.length);
25+ assert.equal(sizeOf(undefined), 0);
26+});
+42−0
1+/**
2+ * What a project of a workspace without the g1t plan may send:
3+ * `FREE_TELEMETRY_BYTES_PER_MONTH` (1 GiB, a wrangler var; `0` is no cap),
4+ * held a day at a time as a thirtieth of it, so a burst cannot spend the
5+ * month in an hour. Past a day's share, OTLP is answered `429` until
6+ * midnight UTC, and Cloudflare's tail events of the project's apps keep
7+ * their requests (what the charts are counted from) without their logs.
8+ *
9+ * Bytes are what arrived: an OTLP body once unzipped, and a tail event's
10+ * rows as they are kept.
11+ */
12+
13+export const DEFAULT_FREE_TELEMETRY_BYTES_PER_MONTH = 1024 * 1024 * 1024;
14+const DAYS_PER_MONTH = 30;
15+
16+/** The monthly allowance from its var: the default when unset or not a whole number; 0 is none. */
17+export function freeMonthlyBytes(raw: string | undefined): number {
18+ const text = (raw ?? "").trim();
19+ const value = Number(text);
20+ return text !== "" && Number.isInteger(value) && value >= 0 ? value : DEFAULT_FREE_TELEMETRY_BYTES_PER_MONTH;
21+}
22+
23+/** A day's share of a monthly allowance; null for none. */
24+export function dailyCap(monthly: number): number | null {
25+ return monthly > 0 ? Math.ceil(monthly / DAYS_PER_MONTH) : null;
26+}
27+
28+/** Whether `adding` more bytes today would go past `cap`. */
29+export function overCap(today: number, adding: number, cap: number | null | undefined): boolean {
30+ return cap != null && cap > 0 && today + adding > cap;
31+}
32+
33+/** The size of a row as it is kept: its JSON. */
34+export function sizeOf(value: unknown): number {
35+ return JSON.stringify(value)?.length ?? 0;
36+}
37+
38+/** What OTLP past the day's share is told. */
39+export function overCapMessage(cap: number): string {
40+ const mb = Math.round(cap / (1024 * 1024));
41+ return `This project has sent its ${mb} MB for today, a workspace on the free plan's daily share of its monthly observability. Send less, or start the g1t plan; more is taken from 00:00 UTC.`;
42+}
+8−0
191191 metric_points INTEGER NOT NULL DEFAULT 0
192192 );
193193 `,
194+ // What arrived each day (UTC), for a free workspace's daily share
195+ // (../quota.ts). Kept a little over a month.
196+ `
197+ CREATE TABLE usage_days (
198+ day TEXT PRIMARY KEY,
199+ ingested_bytes INTEGER NOT NULL DEFAULT 0
200+ );
201+ `,
194202 ];
195203
196204 /**
+25−0
312312 assert.equal(stepFor(86_400_000_000), 900);
313313 assert.equal(aggregate([]).length, 0);
314314 });
315+
316+test("a free workspace's project is refused past its day's share, and tail events keep only their requests", async () => {
317+ const store = new Store(state());
318+ const { secret } = await store.createToken("prj_1", "ci", "u1");
319+ const batch = (bytes: number) => ({ ...emptyBatch(), bytes });
320+ assert.equal((await store.ingest(batch(60), [], secret, 100)).accepted, true);
321+ assert.equal(store.todayBytes(), 60);
322+ const refused = await store.ingest(batch(50), [], secret, 100);
323+ assert.deepEqual(refused, { accepted: false, reason: "quota" });
324+ assert.equal(store.todayBytes(), 60, "nothing refused is counted");
325+ // Without a cap (the plan, or a workspace not known yet) nothing is refused.
326+ assert.equal((await store.ingest(batch(5000), [], secret, null)).accepted, true);
327+ // Tail events past the share keep their requests, not their logs.
328+ const tail = { ...emptyBatch(), bytes: 10_000 };
329+ tail.logs.push({ id: "l1", time_us: 1, severity: "info", severity_number: 9, body: "x".repeat(9000), trace_id: null, span_id: null, request_id: null, attributes: {}, service: "web", environment: null, deployment_id: null, commit_sha: null, source: "cloudflare" });
330+ assert.equal((await store.ingest(tail, [], undefined, 100)).accepted, true);
331+ assert.equal(store.logs({}).items.length, 0);
332+ assert.equal(store.usage().today_bytes, store.todayBytes());
333+ // The workspace is remembered as the site says it.
334+ assert.equal(store.workspaceOf(), null);
335+ store.setWorkspace(" Acme ");
336+ assert.equal(store.workspaceOf(), "acme");
337+ store.setWorkspace("not a slug!");
338+ assert.equal(store.workspaceOf(), "acme");
339+});
+38−1
4646 import { exportEndpointProblem, otlpBodies, send } from "./export.ts";
4747 import { addInto, bucketOf, decode, emptyHistogram, encode, percentiles } from "./histogram.ts";
4848 import { FTS, MIGRATIONS, ROLLUP_DAYS } from "./schema.ts";
49+import { overCap, sizeOf } from "./quota.ts";
4950 import { SEVERITY_FLOOR, randomId, severityOf } from "./shape.ts";
5051 import type { RequestHalf } from "./tail.ts";
5152
8182 return new Date(us / 1000).toISOString().slice(0, 7);
8283 }
8384
85+/** `2026-10-12`: the UTC day, for a free workspace's daily share. */
86+function day(us = Date.now() * 1000): string {
87+ return new Date(us / 1000).toISOString().slice(0, 10);
88+}
89+
8490 /** A page's cursor: the last row's time and id, so the next page starts after it. */
8591 function cursorOf(time: number, id: string): string {
8692 return btoa(`${time}:${id}`);
179185 return this.settings();
180186 }
181187
188+ /** The workspace the project is in, by slug, as the site last said; null before it has. */
189+ workspaceOf(): string | null {
190+ const row = this.rows("SELECT value FROM settings WHERE key = 'workspace'")[0];
191+ return row ? String(row.value) : null;
192+ }
193+
194+ setWorkspace(slug: string): void {
195+ const clean = slug.trim().toLowerCase();
196+ if (!/^[a-z0-9][a-z0-9-]{0,38}$/.test(clean) || clean === this.workspaceOf()) return;
197+ this.sql.exec("INSERT INTO settings (key, value) VALUES ('workspace', ?) ON CONFLICT (key) DO UPDATE SET value = excluded.value", clean);
198+ }
199+
200+ /** Bytes that arrived today (UTC). */
201+ todayBytes(): number {
202+ return Number(this.rows("SELECT ingested_bytes FROM usage_days WHERE day = ?", day())[0]?.ingested_bytes ?? 0);
203+ }
204+
182205 // ---- Tokens -------------------------------------------------------------
183206
184207 tokenRow(row: Row): IngestToken {
249272 * Adds what arrived. `token` is checked first when given (the OTLP door);
250273 * tail events come from Cloudflare and need none. Returns false for a
251274 * token that is not this project's, or when this kind of telemetry is off.
275+ *
276+ * `dailyCap`, for a project of a workspace without the plan, is the
277+ * day's share of its free observability (../quota.ts): past it, OTLP is
278+ * refused (`quota`), and tail events keep their requests without logs.
252279 */
253− async ingest(batch: Batch, halves: RequestHalf[] = [], token?: string): Promise<{ accepted: boolean; reason?: string }> {
280+ async ingest(batch: Batch, halves: RequestHalf[] = [], token?: string, dailyCap?: number | null): Promise<{ accepted: boolean; reason?: string }> {
254281 if (token !== undefined && !(await this.checkToken(token))) return { accepted: false, reason: "token" };
255282 if (token === undefined && !this.settings().deployments) return { accepted: false, reason: "off" };
283+ if (overCap(this.todayBytes(), batch.bytes, dailyCap)) {
284+ if (token !== undefined) return { accepted: false, reason: "quota" };
285+ batch = { ...batch, logs: [], spans: [], metrics: [], errors: [], bytes: sizeOf(halves) + sizeOf(batch.requests) };
286+ }
256287
257288 const live: LiveItem[] = [];
258289 for (const h of halves) this.putHalf(h);
305336 logs = logs + excluded.logs, spans = spans + excluded.spans, metric_points = metric_points + excluded.metric_points`,
306337 month(), batch.bytes, requestCount, batch.logs.length, batch.spans.length, batch.metrics.length,
307338 );
339+ this.sql.exec(
340+ "INSERT INTO usage_days (day, ingested_bytes) VALUES (?, ?) ON CONFLICT (day) DO UPDATE SET ingested_bytes = ingested_bytes + excluded.ingested_bytes",
341+ day(), batch.bytes,
342+ );
308343
309344 // Requests go live (and to exports) whole: once both halves are in, or
310345 // for OTLP, at once.
533568 this.sql.exec("DELETE FROM metric_points WHERE time_us < ?", keep);
534569 this.sql.exec("DELETE FROM rollups WHERE minute < ?", Math.floor((now - ROLLUP_DAYS * DAY_US) / MINUTE_US));
535570 this.sql.exec("DELETE FROM error_groups WHERE last_us < ? AND issue_number IS NULL", now - ROLLUP_DAYS * DAY_US);
571+ this.sql.exec("DELETE FROM usage_days WHERE day < ?", day(now - 40 * DAY_US));
536572 // Over its size, the store sheds a day at a time from the oldest.
537573 for (let i = 0; i < 30 && this.state.storage.sql.databaseSize > MAX_BYTES; i++) {
538574 const oldest = Number(this.rows("SELECT MIN(time_us) AS t FROM logs")[0]?.t ?? now);
950986 spans: Number(row?.spans ?? 0),
951987 metric_points: Number(row?.metric_points ?? 0),
952988 stored_bytes: this.state.storage.sql.databaseSize,
989+ today_bytes: this.todayBytes(),
953990 };
954991 }
955992
+3−2
2323 import { TELEMETRY_TAG, type Attributes, type TelemetryLog, type TelemetryRequest } from "@g1t/contracts/telemetry";
2424
2525 import { emptyBatch, type Batch } from "./batch.ts";
26+import { sizeOf } from "./quota.ts";
2627 import { clip, hostPath, randomId, route, severityNumber, severityOf, text } from "./shape.ts";
2728
2829 /** The first argument of the dispatcher's line for each request. */
144145 country: cf.country ?? null,
145146 trace_id: mark.trace_id || null,
146147 });
147− g.batch.bytes += 256;
148+ g.batch.bytes += sizeOf(g.halves[g.halves.length - 1]);
148149 continue;
149150 }
150151
228229 log_count: logs.length,
229230 error: first ? { name: first.name, message: clip(first.message, 2048) } : item.outcome !== "ok" && item.outcome !== "canceled" ? { name: item.outcome, message: outcomeText(item.outcome) } : null,
230231 });
231− g.batch.bytes += logs.reduce((n, l) => n + l.body.length, 256);
232+ g.batch.bytes += sizeOf(g.halves[g.halves.length - 1]) + logs.reduce((n, l) => n + sizeOf(l), 0);
232233 }
233234 return [...groups.values()];
234235 }
+6−0
2121 // project and deployment), read to file a request the dispatcher saw
2222 // before its note reached it, and Cloudflare's own export. Read only.
2323 "kv_namespaces": [{ "binding": "DOMAINS", "id": "14bc5c233d4c46a5bbf23b5367cce5fd" }],
24+ // Whether a project's workspace has the plan: a free one's projects send
25+ // FREE_TELEMETRY_BYTES_PER_MONTH (1 GiB) a month, a thirtieth a day; past
26+ // it OTLP is answered 429 and tail events keep their requests only
27+ // (src/quota.ts). 0 is no cap.
28+ "services": [{ "binding": "BILLING", "service": "g1t-billing" }],
29+ "vars": { "FREE_TELEMETRY_BYTES_PER_MONTH": "1073741824" },
2430 // Secret, optional: CLOUDFLARE_EXPORT_KEY, which a Workers
2531 // Observability destination on the account sends as x-g1t-export-key
2632 // to https://otel.g1t.sh/cloudflare/v1/{traces,logs}.

Comments

Sign in to comment.