Backend / Observability / 02_grafana.md

Grafana

Updated 5 interview angles 3 min read source
On this page6
  1. Dashboards that survive an incident
  2. Alerting: pick one place
  3. Provision it as code
  4. The rest of the stack
  5. Related
  6. Interview angle

Grafana

The query and visualisation layer over data it does not own. Prometheus keeps the metrics, Loki the logs, Tempo the traces; Grafana is the single pane of glass across them, plus the alerting on top.

Saying “Grafana stores the metrics” in an interview is the tell that you have only ever clicked around one.

Dashboards that survive an incident

The failure mode is a wall of forty panels nobody reads at 3am. A dashboard is useful when it answers a specific question, so start from a known framework rather than from the metrics you happen to have.

Method Signals For
RED Rate, Errors, Duration request-driven services
USE Utilisation, Saturation, Errors resources: CPU, disk, pool
Four golden latency, traffic, errors, saturation the SRE-book version

A working structure: one overview dashboard showing RED per service, and drill-down dashboards linked from it. The overview answers “what is broken”, the drill-down answers “why”.

Variables make one dashboard serve many

text 
$service = label_values(http_requests_total, job)
$env     = label_values(up, env)

Then panels query rate(http_requests_total{job="$service"}[5m]). One dashboard covers every service instead of twenty near-identical copies that drift apart.

Alerting: pick one place

Grafana’s unified alerting can evaluate rules itself, and Prometheus can evaluate its own and hand them to Alertmanager. Both work. Split across both is how gaps happen — an alert nobody owns, silenced in the system nobody was looking at.

The general preference is alert rules next to the data (Prometheus, in Git), and Grafana for visualisation. Grafana-side alerting earns its place when a rule spans data sources — metrics and logs in one condition.

Either way, alert on symptoms users feel, not causes:

  • Good: error rate above 2% for 5 minutes; p99 latency over 1s.
  • Bad: CPU above 80%. It might be fine. It pages someone for nothing, and after the third false page nobody reads them.

Provision it as code

Clicking dashboards together makes them unreproducible and untracked. Grafana reads provisioning files at startup:

yaml 
# provisioning/datasources/prom.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

Dashboards provision the same way from JSON. The practical rule: a dashboard that matters lives in the repository, so it is reviewed, versioned and restorable. Ad-hoc exploration stays ad-hoc.

Gotcha: a provisioned dashboard is read-only in the UI, and edits are lost on restart. That is the point, but it surprises people — the workflow is edit, export JSON, commit.

The rest of the stack

Piece Holds
Prometheus / Mimir metrics
Loki logs
Tempo traces
Alertmanager routing, grouping, silences

Loki is deliberately unlike Elasticsearch: it indexes only labels, not log content, which makes it cheap and makes full-text search slow. It is designed for “give me the logs for this pod around this timestamp”, which is what you actually do after a metric alert fires.

The payoff of keeping the labels consistent across all three is being able to jump from a latency spike to the traces to the logs of one request without re-typing anything.

Interview angle 5

  • “What is Grafana’s role?” - visualisation and alerting over data sources it doesn’t own: Prometheus for metrics, Loki for logs, Tempo for traces. It’s the pane of glass, not the store.
  • “What makes a dashboard useful?” - it answers a specific question during an incident. Start from the RED metrics for each service, keep the top-level view small, and drill down from there. A wall of forty panels is decoration.
  • “Where should alerts live?” - close to the data source (Prometheus alerting rules) or in Grafana unified alerting, but in one place. Alerts split across systems produce gaps nobody notices until an incident.
  • “What should you alert on?” - symptoms users feel: error rate, latency, queue growth. Alerting on CPU above 80% pages someone for something that may be entirely fine, and false pages train people to ignore real ones.
  • “How do you keep dashboards maintainable?” - provision them from JSON in the repository, and use template variables so one dashboard covers every service instead of twenty copies that drift apart.
Previous
Sentry