| 1 | --- |
| 2 | title: Observability |
| 3 | description: A project's requests, logs, traces, metrics and errors, from its apps on g1t.page with nothing to set up, and from anything else over OpenTelemetry. |
| 4 | --- |
| 5 | |
| 6 | Observability shows what a [project](/guides/projects/)'s running software |
| 7 | does: every request it answers, every line it writes, every trace through |
| 8 | the services it calls, the metrics it reports, and its errors, grouped. It |
| 9 | is on every project's menu, under **Observability**. |
| 10 | |
| 11 | It has two sources, and they meet in one place: |
| 12 | |
| 13 | - **Apps on [g1t.page](/guides/deployments/).** Nothing to set up. Every |
| 14 | request a deployed app answers arrives within seconds, with its status, |
| 15 | how long it took, its CPU time, where in the world it was served, and |
| 16 | every line it wrote with `console.log` and the like. Exceptions are |
| 17 | grouped into errors. An [OpenTelemetry](https://opentelemetry.io/) SDK in |
| 18 | the app sends its spans here too, with no configuration: g1t gives each |
| 19 | app the endpoint and a token. |
| 20 | - **Anything else, over OpenTelemetry.** Software running on another host, |
| 21 | a server of your own, a phone app or a background worker sends traces, |
| 22 | logs and metrics to `https://otel.g1t.sh` with one of the project's |
| 23 | ingest tokens, in OTLP over HTTP, JSON or protobuf. Any OpenTelemetry SDK |
| 24 | or collector does this already. |
| 25 | |
| 26 | Both arrive in the same store, so a request on g1t.page and the spans of |
| 27 | the API it called elsewhere are one trace. |
| 28 | |
| 29 | ## The pages |
| 30 | |
| 31 | | Page | What it shows | |
| 32 | | --- | --- | |
| 33 | | **Overview** | Requests, error rate, and latency at the 50th, 95th and 99th percentiles, against the period before; requests and latency over time; each deployment's numbers, newest first; the busiest routes; open errors; the services reporting; and where requests were served | |
| 34 | | **Requests** | Every request, newest first: status, method and address, duration, CPU time, region and deployment. Narrow by status, failed only, a route, or search | |
| 35 | | **Logs** | Every line, newest first. Search words, a level or an attribute; follow it live | |
| 36 | | **Traces** | Each trace, with the services it touched, its spans and errors. A trace opens as a waterfall, with each span's attributes and events, and the logs written under it | |
| 37 | | **Errors** | Exceptions grouped by what they are and where they were thrown, with how often and since which deployment. Each can become a work item | |
| 38 | | **Metrics** | Every metric sent over OpenTelemetry, over time, averaged, summed or as a percentile, and grouped by service or any attribute | |
| 39 | | **Settings** | Ingest tokens, exports, retention, and what the project used this month | |
| 40 | |
| 41 | Every page narrows by period (the last 15 minutes to the last 30 days), |
| 42 | environment, service and deployment, and the address says what it shows, |
| 43 | so a view can be shared by its link. |
| 44 | |
| 45 | ### Search |
| 46 | |
| 47 | The Logs, Requests and Traces pages take the same search: |
| 48 | |
| 49 | | You write | It finds | |
| 50 | | --- | --- | |
| 51 | | `timeout` | lines with the word, or a word starting with it at the end | |
| 52 | | `payment captured` | lines with both words | |
| 53 | | `"payment captured"` | the phrase | |
| 54 | | `level:error` | errors (and `level:warn`, `level:info`, `level:debug`) | |
| 55 | | `user.id=42` | records whose attribute `user.id` is `42` | |
| 56 | | `status:502`, `region:AMS`, `method:POST` | requests by a field of their own | |
| 57 | | `service:api`, `trace_id:…`, `deployment_id:…` | records from one service, trace or deployment | |
| 58 | |
| 59 | ### A regression points at its deployment |
| 60 | |
| 61 | The Overview's **By deployment** table lists each deployment that served |
| 62 | requests in the period, newest first, with its requests, error rate and |
| 63 | 95th-percentile latency. An error rate well above the deployment's |
| 64 | before it shows in red. Choosing a deployment narrows every page to it. |
| 65 | Each error group also says which deployment it was first seen on. |
| 66 | |
| 67 | ### Errors become work items |
| 68 | |
| 69 | An error is grouped with others like it: the same name, the same message |
| 70 | once numbers, ids and quoted values are taken out, and the same place in |
| 71 | the code it was thrown from (its file and function, not its line, which |
| 72 | moves with every change). A thousand `TypeError: Cannot read 'name' of |
| 73 | user 41`, `… of user 977` are one group with a count. |
| 74 | |
| 75 | **Make a work item** opens a work item with the error, its stack, how often it |
| 76 | happened, the deployment it began with, and links to its latest request |
| 77 | and trace. An error |
| 78 | marked **Resolve** opens again if it happens again; **Ignore** keeps it |
| 79 | quiet. |
| 80 | |
| 81 | ## Apps on g1t.page |
| 82 | |
| 83 | Nothing to set up. Every request reaches an app through g1t.page's |
| 84 | dispatcher, which gives it an id and a trace and records it, so even a |
| 85 | request a site's files answered without running its code is counted. When |
| 86 | the app's code runs, Cloudflare hands g1t what it wrote and threw. Both |
| 87 | halves are filed under the project and the deployment that served it. |
| 88 | |
| 89 | Each request the app receives carries two headers it can use: |
| 90 | |
| 91 | | Header | What it is | |
| 92 | | --- | --- | |
| 93 | | `x-g1t-request-id` | The request's id on its page under **Requests**. Log it, or show it on an error page, to find the request again | |
| 94 | | `traceparent` | The request's trace ([W3C Trace Context](https://www.w3.org/TR/trace-context/)). An incoming `traceparent` is continued, so a caller's trace runs through | |
| 95 | |
| 96 | ### Spans from the app |
| 97 | |
| 98 | Each app on g1t.page is deployed with OpenTelemetry's standard variables |
| 99 | set: |
| 100 | |
| 101 | | Variable | Value | |
| 102 | | --- | --- | |
| 103 | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `https://otel.g1t.sh` | |
| 104 | | `OTEL_EXPORTER_OTLP_HEADERS` | `authorization=Bearer <the project's deployments token>`, a secret | |
| 105 | | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/json` | |
| 106 | | `OTEL_SERVICE_NAME` | the project's name | |
| 107 | | `OTEL_RESOURCE_ATTRIBUTES` | `deployment.environment.name`, `g1t.deployment.id`, `g1t.project.id` and `vcs.ref.head.revision` | |
| 108 | |
| 109 | An OpenTelemetry SDK that reads them sends its spans, logs and metrics |
| 110 | here as they are, and its server span continues the request's |
| 111 | `traceparent`, so the spans sit under the request in its trace. For a |
| 112 | Worker, [`@microlabs/otel-cf-workers`](https://github.com/evanderkoogh/otel-cf-workers) |
| 113 | reads the endpoint and headers from `env`: |
| 114 | |
| 115 | ```ts |
| 116 | // src/index.ts |
| 117 | import { instrument, type ResolveConfigFn } from "@microlabs/otel-cf-workers"; |
| 118 | |
| 119 | const config: ResolveConfigFn = (env: Env) => ({ |
| 120 | exporter: { |
| 121 | url: `${env.OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces`, |
| 122 | headers: { authorization: env.OTEL_EXPORTER_OTLP_HEADERS.replace(/^authorization=/, "") }, |
| 123 | }, |
| 124 | service: { name: env.OTEL_SERVICE_NAME }, |
| 125 | }); |
| 126 | |
| 127 | export default instrument({ async fetch(request, env) { /* … */ } }, config); |
| 128 | ``` |
| 129 | |
| 130 | ### Cloudflare's own spans |
| 131 | |
| 132 | Every app on g1t.page is also traced by Cloudflare itself, with nothing in |
| 133 | its code: a span for each `fetch` it makes, each KV, D1, R2 and Durable |
| 134 | Object call, and its cache, under the request. They arrive in the |
| 135 | request's trace, next to any spans of the app's own SDK. Its logs are not |
| 136 | sent this way: they already arrive with each request. |
| 137 | |
| 138 | Set `OTEL_EXPORTER_OTLP_ENDPOINT` yourself, under the project's |
| 139 | [secrets and variables](/guides/secrets-and-variables/), to send the |
| 140 | app's OpenTelemetry somewhere else instead: g1t then sets none of these. |
| 141 | Requests and logs still arrive from g1t.page either way. A build that is |
| 142 | not trusted (a pull request from a fork) gets no token. |
| 143 | |
| 144 | To stop apps on g1t.page reporting, turn off **Apps on g1t.page report |
| 145 | here** under the project's **Observability**, **Settings**. What is |
| 146 | already here stays. |
| 147 | |
| 148 | ## Sending from anywhere else |
| 149 | |
| 150 | 1. Under the project's **Observability**, **Settings**, choose **New |
| 151 | token**, name it after what will send with it, and copy it: it is shown |
| 152 | once. Tokens start with `g1to_`, and each can only send to its project. |
| 153 | 2. Point your software at the endpoint, with the token: |
| 154 | |
| 155 | ```sh |
| 156 | OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.g1t.sh |
| 157 | OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer g1to_…" |
| 158 | OTEL_SERVICE_NAME=billing-api |
| 159 | OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production |
| 160 | ``` |
| 161 | |
| 162 | Every OpenTelemetry SDK reads these: Node.js, Python, Go, Java, .NET, |
| 163 | Ruby, PHP, Rust and the rest. `OTEL_SERVICE_NAME` is what the service is |
| 164 | called on the pages, and `deployment.environment.name` its environment. |
| 165 | `vcs.ref.head.revision`, the commit it runs, shows next to its numbers. |
| 166 | |
| 167 | 3. Run it. What it sends shows within seconds. |
| 168 | |
| 169 | The endpoint takes OTLP over HTTP: `POST /v1/traces`, `/v1/logs` and |
| 170 | `/v1/metrics`, as `application/json` or `application/x-protobuf`, gzipped |
| 171 | or not, up to 8 MB a request. It answers as OTLP expects, so exporters |
| 172 | retry what should be retried and drop what should not. It also answers |
| 173 | browsers (CORS), for a web page that sends its own spans; a token in a page |
| 174 | can be read by anyone who opens it, so give a page a token of its own. |
| 175 | |
| 176 | `https://api.g1t.sh/otlp` is the same endpoint at a second address. |
| 177 | |
| 178 | ### From an OpenTelemetry Collector |
| 179 | |
| 180 | A collector already gathering telemetry sends it on with the |
| 181 | `otlphttp` exporter: |
| 182 | |
| 183 | ```yaml |
| 184 | exporters: |
| 185 | otlphttp/g1t: |
| 186 | endpoint: https://otel.g1t.sh |
| 187 | headers: |
| 188 | authorization: Bearer ${env:G1T_INGEST_TOKEN} |
| 189 | |
| 190 | service: |
| 191 | pipelines: |
| 192 | traces: { receivers: [otlp], exporters: [otlphttp/g1t] } |
| 193 | logs: { receivers: [otlp], exporters: [otlphttp/g1t] } |
| 194 | metrics: { receivers: [otlp], exporters: [otlphttp/g1t] } |
| 195 | ``` |
| 196 | |
| 197 | ### Traces |
| 198 | |
| 199 | Spans arrive as they are, with their attributes, events and links. Every |
| 200 | server span (`SpanKind.SERVER`) of software outside g1t.page is also a |
| 201 | request: it is on the Requests page and counted on the Overview, by its |
| 202 | `http.route` (or its path with ids taken out), status and duration. An |
| 203 | exception recorded on a span (its `exception` event) is an error. |
| 204 | |
| 205 | ### Logs |
| 206 | |
| 207 | Log records arrive with their severity, body, attributes and, when the SDK |
| 208 | sets them, the trace and span they were written under, so a trace shows its |
| 209 | logs. A record with `exception.type` or `exception.message` attributes is |
| 210 | also an error. |
| 211 | |
| 212 | ### Metrics |
| 213 | |
| 214 | Gauges, sums, histograms and summaries are kept as points. A counter |
| 215 | (a cumulative, monotonic sum) shows as its increase in each step of the |
| 216 | chart. A histogram shows as its mean, or its 50th, 95th or 99th |
| 217 | percentile from its buckets. Exponential histograms keep their count, |
| 218 | sum, minimum and maximum. |
| 219 | |
| 220 | ## Sending it on elsewhere too |
| 221 | |
| 222 | An **export** sends everything that arrives on to any OTLP/HTTP endpoint |
| 223 | as well: a collector of your own, or any tool that takes OpenTelemetry. |
| 224 | Requests on g1t.page go as server spans, and their logs as log records, so |
| 225 | the tool shows the project's apps as it shows any other service. |
| 226 | |
| 227 | Under **Observability**, **Settings**, choose **New export**, give its |
| 228 | address (g1t adds `/v1/traces`, `/v1/logs` and `/v1/metrics`), the signals |
| 229 | to send, and the headers it needs, such as an API key. Header values are |
| 230 | kept as secrets and never shown again. An export's last error, if any, |
| 231 | shows under it. Exports go to addresses on the internet over `https` only. |
| 232 | |
| 233 | ## Retention |
| 234 | |
| 235 | Requests, logs, spans and metric points are kept for 7 days, or 1, 3, 14 or |
| 236 | 30, as the project's **Observability**, **Settings** say. The Overview's |
| 237 | per-minute numbers are kept for 90 days whatever the retention, so its |
| 238 | charts reach back further than its requests do. A project's store holds up |
| 239 | to 4 GB; past that, its oldest day goes first. |
| 240 | |
| 241 | ## On the free plan |
| 242 | |
| 243 | A project in a workspace on the free plan can send **1 GB a month**, held a |
| 244 | day at a time: about 34 MB a day (UTC), a thirtieth of the month, so a burst |
| 245 | cannot use the month up in an hour. What counts is what arrives: an OTLP |
| 246 | request's body once unzipped, and the rows kept from an app's tail events. |
| 247 | **Settings** shows what arrived this month and today. |
| 248 | |
| 249 | Past the day's share, OTLP requests are answered `429 Too Many Requests` |
| 250 | with the reason, until 00:00 UTC; OpenTelemetry SDKs retry or drop the |
| 251 | batch. Requests from an app on g1t.page still arrive, so its charts and |
| 252 | request list keep going, but without their logs. On the g1t plan there is |
| 253 | no such cap. OTLP is also limited to 600 requests a minute from one |
| 254 | address; see [rate limits](/reference/rate-limits/). |
| 255 | |
| 256 | ## Who can do what |
| 257 | |
| 258 | | | Who | |
| 259 | | --- | --- | |
| 260 | | See the pages | People with a role on the project's repository, even when its code is public: logs and addresses can hold what only the people running it should read | |
| 261 | | Resolve, ignore or reopen errors | Triage and up | |
| 262 | | Make a work item of an error | Anyone who can open work items in the project's repository | |
| 263 | | Tokens, exports, retention and the g1t.page switch | Admin | |
| 264 | |
| 265 | ## The API and MCP |
| 266 | |
| 267 | Everything on the pages is in the [REST API](/reference/api/), under |
| 268 | **Observability**, with the `observability:read` scope, and |
| 269 | `observability:write` to change errors, settings, tokens and exports. |
| 270 | |
| 271 | Coding agents read it through the `observability` tool of the |
| 272 | [MCP server](/reference/mcp/): the overview, requests, logs, traces, |
| 273 | errors and metrics. A coding agent fixing an error reads its stack, its |
| 274 | request and its trace the same way you do. |
| 275 | |
| 276 | ## What it costs |
| 277 | |
| 278 | Nothing yet, while g1t is in alpha. **Observability**, **Settings**, |
| 279 | **This month** shows what the project sent and keeps: the bytes received, |
| 280 | requests, log lines, spans and metric points. |
| 281 | |
| 282 | ## How it works |
| 283 | |
| 284 | Each project's telemetry is kept in a store of its own, a Durable Object |
| 285 | with its own SQLite database, so one project's volume never slows another's |
| 286 | pages. Log bodies are indexed for full-text search. |
| 287 | |
| 288 | - **g1t.page.** The dispatcher in front of every app names the telemetry |
| 289 | service as its Tail Worker, and so does each app. Each app's own |
| 290 | Cloudflare tracing is exported to the service too, over OTLP, and filed |
| 291 | under the app's project and deployment. The dispatcher's half |
| 292 | of a request (status, time, place) and the app's half (logs, exceptions, |
| 293 | CPU time) arrive apart and are joined by the request's id. |
| 294 | - **OTLP.** The endpoint reads the project from the token, checks it |
| 295 | against the project's tokens (kept as hashes), decodes the request, and |
| 296 | files each record by its resource: `service.name`, |
| 297 | `deployment.environment.name` and `g1t.deployment.id`. |
| 298 | - **Counting.** Each minute, the requests of the minute before are counted |
| 299 | into per-minute numbers by service, environment, deployment and route, |
| 300 | with latency and CPU time as histograms, which is how percentiles over any |
| 301 | period come from a few numbers. The newest minute is counted as it stands |
| 302 | when a page asks. |
| 303 | |
| 304 | On a [self-hosted installation](/guides/self-hosting/), Observability runs |
| 305 | with the rest of g1t; software sends to `http://localhost:8789/otlp`. |