Grafana

Article.

Grafana is a query, visualisation and alerting layer over data it does not own. Prometheus or Mimir keep the metrics, Loki the logs, Tempo the traces, Pyroscope the profiles; Grafana sends each panel’s query to the right data source and draws the answer. Saying “Grafana stores the metrics” in an interview is the tell that you have only ever clicked around one.

The senior questions are about what happens underneath a panel and an alert: “your rate panel goes empty when you zoom out to 30 days. Why?”, “an exporter died at 2am and nothing paged. Why?” and “on-call got every page twice. Why?” All three are mechanics, not dashboard taste.

Current release: Grafana 13 (GrafanaCON, 2026-04-21), 13.1 in the docs. Grafana, Loki and Tempo have been AGPLv3 since April 2021 (Mimir since its 2022 launch); that matters if you modify and serve them, not if you only run them. Prometheus internals are in Prometheus; the three signals in general in The Three Pillars of Observability.

How a panel becomes a query

browser ──(dashboard JSON, time range, width)──▶ Grafana
   Grafana per panel:
     $__interval = range / Max data points  (≥ Min interval)
     expand $variables, build PromQL / LogQL / SQL
   Grafana server ──HTTP──▶ data source (Prometheus, Loki…)
   ◀── series ── transform, draw

• Queries run from the Grafana server (the data source’s “server” access mode), so credentials stay on the server and the database need not be reachable from browsers.
• Every panel is at least one query, on every refresh, for every viewer. Grafana does not share results between viewers by default.
• $__interval is the step between points: the time range divided by the panel’s Max data points (about its pixel width), never below Min interval. Zoom out from 1 hour to 30 days and the step grows from seconds to tens of minutes.

The load arithmetic

A 40-panel dashboard on the office TV and on 19 laptops, refreshing every 30 seconds: 40 x 20 / 30 ≈ 27 queries per second against Prometheus, each scanning its range. That is how one popular dashboard slows Prometheus for everyone. The fixes, in order of effect:

1. Recording rules for heavy expressions, so the panel reads one precomputed series (see Prometheus).
2. A slower refresh, and min_refresh_interval in grafana.ini so nobody can set 1 s.
3. Query caching (Grafana Enterprise and Cloud) for shared dashboards.
4. Fewer panels: a dashboard that answers one question rarely needs 40.

PromQL panels done right

Use $__rate_interval, not a fixed window

sum by (status) (rate(http_requests_total{job="$service"}[$__rate_interval]))

Grafana computes it as max($__interval + scrape_interval, 4 x scrape_interval). That guarantees the window always holds enough samples, and widens with the step so no samples fall between points.

Both failure modes of a fixed [1m]:

• Zoomed out: the step is 20 minutes but each point looks at only 1 minute, so the line shows 1 minute in every 20. Spikes in the gaps vanish.
• Scrape interval of 60 s: since Prometheus 3, range selectors are left-open, so [1m] holds one sample and rate() needs two. The panel is empty. In Prometheus 2 the same query usually returned something.

The trap: scrape_interval comes from the data source’s Scrape interval setting, which defaults to 15 s. If Prometheus actually scrapes every 60 s and the setting still says 15 s, $__rate_interval is 1 minute at short ranges and you are back to the empty panel. Set it to the real value.

Percentiles from histograms

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[$__rate_interval])))

• Sum the buckets by le first, then take the quantile. Averaging per-pod p99s is mathematically meaningless.
• The answer is interpolated within a bucket, so it is only as precise as the bucket edges. A p99 “of exactly 0.5 s” usually means the edge is 0.5.
• Use the Heatmap panel on the buckets to see the whole distribution, a bimodal latency that a single p99 line hides.

Variables make one dashboard serve many

$service = label_values(up, job)
$env     = label_values(up{job="$service"}, env)

Variables are queries too, re-run on every dashboard load (and on every time-range change, if set to). Asking for labels of a high-cardinality metric such as http_requests_total scans all its series; up has one series per target, so it is cheap. Set refresh to On dashboard load.

One dashboard then covers every service instead of twenty copies that drift apart. Start it from RED (rate, errors, duration) per service, with drill-down dashboards linked from the overview. USE (utilisation, saturation, errors) is the same idea for resources.

Alerting: the state machine

Grafana-managed rules are evaluated by the Grafana server on a schedule, per rule group. One rule can produce many alert instances: a query that returns a series per pod creates an instance per pod.

Normal ──true──▶ Pending ──true for pending──▶ Alerting
  ▲                 │ false                        │ false
  ├─────────────────┘                              ▼
  └──────── keep firing for elapsed ─────── Recovering

• Pending period (“for”): the condition must hold this long before firing. It absorbs single bad scrapes.
• Keep firing for (Grafana 12+): after the condition clears, the instance stays in Recovering this long. If it breaches again it goes back to Alerting without a new notification. That stops flapping pages.
• No Data: the query succeeded and returned nothing. Error: the query failed or timed out. Each has a configurable mapping: No Data / Error (default), Alerting, Normal or Keep last state. In the default mode Grafana fires a separate alert named DatasourceNoData or DatasourceError.

Why the dead exporter never paged

The rule was rate(errors_total[5m]) > 0.05, one instance per pod. When the exporter died, its series stopped. Instances whose series disappear are resolved as missing after two evaluation intervals; they do not fire. If every series vanished, the rule went to No Data, which someone had mapped to Normal to stop noise. Either way: silence.

Alert on the absence explicitly, as its own rule:

up{job="orders-api"} == 0            # target scraped, process down
absent(up{job="orders-api"})         # target gone from discovery

An error-rate rule only tells you about errors you can still see.

Notification routing

Firing instances go to the notification policy tree: matchers on labels choose a contact point (Slack, PagerDuty, a webhook), and the policy groups and throttles.

group_by: [alertname, cluster] turns 200 pods failing into one message, not 200. Silences mute matching alerts for a window (a maintenance); mute timings mute on a schedule (nights, for a non-critical team).

Where rules should live

Both work; split ownership is the problem: an alert nobody owns, silenced in the system nobody watches. Both kinds are code: data-source rules as YAML, Grafana-managed rules through file provisioning, Terraform or the alerting API. Alert on what users feel (error rate, latency, queue age), not on CPU above 80%.

Running Grafana with more than one replica

Grafana stores dashboards, users and alert state in a database: SQLite by default, on local disk. Two replicas need a shared PostgreSQL or MySQL, or each has its own dashboards.

Alerting needs one more thing. Every replica evaluates the rules, and without coordination every replica sends every notification: two replicas, two pages. Configure the replicas as peers so their alertmanagers deduplicate:

[unified_alerting]
ha_peers = grafana-0.grafana:9094,grafana-1.grafana:9094
ha_listen_address = ${POD_IP}:9094
# or: ha_redis_address = redis:6379  (memberlist preferred)

Watch Grafana itself: it exposes /metrics, including grafana_alerting_* series for rule evaluation failures and duration. A rule whose evaluation keeps timing out is in Error, not Normal.

Dashboards as code

Provisioning files are read at startup and polled afterwards:

# provisioning/dashboards/team.yaml
apiVersion: 1
providers:
  - name: team
    folder: Orders
    type: file
    allowUiUpdates: false
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards

What survives what:

• allowUiUpdates: false (the default): the UI refuses to save a provisioned dashboard. The workflow is edit, export JSON, commit.
• allowUiUpdates: true: a UI save is stored in the database and survives restarts, until the file changes; then the file wins and the UI edit is gone.
• Deleting the file deletes the dashboard, unless disableDeletion.

The newer options:

• Git Sync (GA in Grafana 13): Grafana reads and writes dashboards in a GitHub, GitLab, Bitbucket or plain Git repository; a UI save becomes a commit or pull request.
• Terraform (grafana provider) for dashboards, folders, data sources, contact points and rules in one plan.
• Generating JSON with Grafonnet (Jsonnet) or the Foundation SDK (Go, TypeScript, Python), when dashboards are repetitive enough to want functions. Dashboards now use schema v2 (dynamic dashboards, default in 13); v1 JSON is converted on load.

The rest of the stack

Alloy replaced Grafana Agent, which reached end of life on 2025-11-01. Beyla, the eBPF auto-instrumenter, was donated to OpenTelemetry in 2025 as OBI. Faro is the browser SDK; k6 the load tester.

Loki: labels are the index

Each unique label set is a stream, with its own chunks. Only labels are indexed; the log text is stored compressed and scanned at query time.

{app="orders-api", env="prod"} |= "timeout" | json | latency_ms > 500

The selector {...} picks streams through the index, cheaply. Everything after | scans every line in those streams over the range. A narrow selector and a short range are what make Loki fast.

The classic mistake is a high-cardinality label: user_id or trace_id as a label creates a stream per value, millions of tiny chunks, and an index that falls over. In Loki 3 those belong in structured metadata, attached to each line, queryable, but not indexed. Loki 3 also ingests OTLP natively.

Logs become metrics with a range aggregation:

sum by (status) (rate({app="orders-api"} | json [5m]))

Jumping between signals

The payoff of one stack is going from a latency spike to the request that caused it without retyping anything:

• Exemplars: a histogram sample carries a trace_id. Prometheus must run with --enable-feature=exemplar-storage, the app must expose OpenMetrics, and the data source needs the link to Tempo configured.

REQUEST_SECONDS.labels(route="/orders").observe(
    elapsed, exemplar={"trace_id": format(span_ctx.trace_id, "032x")}
)

• Trace to logs: Tempo’s data source settings map a span to a Loki query by trace_id and time window.
• Logs to trace: a Loki derived field turns trace_id=... in a line into a Tempo link.

None of it works unless service names and trace_id are spelled the same in all three; see Trace context in async Python.

Where it breaks in production

• Panels empty after a Prometheus 3 upgrade: fixed [1m] windows at a 60 s scrape, or a data source Scrape interval that does not match.
• Prometheus slow every morning: the same dashboard opened by the whole team, 30 s refresh, no recording rules.
• Every page twice: two Grafana replicas without ha_peers.
• Dashboards gone after a redeploy: SQLite on the container’s disk, or UI edits to a provisioned dashboard overwritten by the file.
• The alert that never fired: series vanished and instances resolved; or No Data mapped to Normal.
• Loki ingestion rejected (max streams per user exceeded): someone added request_id as a label.

The trade not taken

Datadog (or another SaaS) instead of the Grafana stack. Datadog is one product with agents, APM, logs and alerting already integrated, and no storage to operate; its price grows with hosts, custom metrics and log volume, and leaving it means re-instrumenting.

The Grafana stack is open source, OpenTelemetry-first and cheap at volume, but someone runs Mimir, Loki and Tempo, or pays Grafana Cloud to. Grafana also reads CloudWatch and Datadog as data sources, so dashboards can stay while the backend changes. Comparison: Choosing an Observability Stack.

Related

• Prometheus - metric types, PromQL, cardinality, recording rules
• The Three Pillars of Observability - metrics, logs and traces in general
• SLO, SLI, SLA, and Error Budgets - what to alert on: burn rates
• Structured logging in Python - what Loki ingests
• OpenTelemetry - emitting all three signals
• Amazon CloudWatch - AWS’s alternative