Backend / Observability / 03_prometheus.md

Prometheus

Updated 5 interview angles 4 min read source
On this page7
  1. Why pull, and what it costs
  2. The metric types
  3. PromQL, minimally
  4. Cardinality is the failure mode
  5. What it is not
  6. Related
  7. Interview angle

Prometheus

A time-series database that scrapes metrics from HTTP endpoints on an interval. Targets expose /metrics; Prometheus pulls. That one design choice explains most of its behaviour.

text 
app:/metrics ─┐
app:/metrics ─┼─scrape─▶ Prometheus ─▶ Grafana
node:/metrics ┘            │
                           └─▶ Alertmanager

Why pull, and what it costs

Pulling means Prometheus owns the schedule, so it always knows whether a target is up — a scrape failure is the up == 0 signal. Service discovery gives it the target list, so a new pod is scraped without configuring anything.

What it costs: a job that exits before the next scrape is never seen. That is what Pushgateway exists for — batch jobs push their final numbers and Prometheus scrapes the gateway. It is the documented exception, not a general push mode, and using it as one gives you stale metrics that never expire.

The metric types

Type For Example
Counter monotonic totals requests, errors
Gauge up and down queue depth, memory
Histogram distributions latency buckets
Summary client-side quantiles rarely the right pick
python 
from prometheus_client import Counter, Histogram

REQS = Counter(
    "http_requests_total", "Requests",
    ["method", "status"],
)
LATENCY = Histogram(
    "http_request_seconds", "Latency", ["endpoint"]
)

REQS.labels("GET", "200").inc()
with LATENCY.labels("/orders").time():
    handle()

Counters only go up, and reset to zero on restart. That is fine because every query uses rate(), which detects resets. Using a gauge for something monotonic throws away that handling.

Histogram vs summary

A histogram ships bucket counts and lets the server compute quantiles, so you can aggregate across instances. A summary computes quantiles in the client, and quantiles cannot be averaged — you cannot combine p99s from ten pods into a service p99. Prefer histograms for anything you will aggregate.

PromQL, minimally

plaintext 
rate(http_requests_total[5m])

sum by (status) (rate(http_requests_total[5m]))

histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_seconds_bucket[5m]))
)

Reading a raw counter is meaningless — it is a lifetime total that resets. Nearly every useful query starts with rate() or increase() over a window.

Gotcha: the window in rate(...[5m]) must cover at least two scrapes, or you get gaps. A 5-minute window with a 60-second scrape interval is the safe default; [1m] with a 60s interval will flicker.

Cardinality is the failure mode

Every distinct combination of label values is a separate time series held in memory. This is the way Prometheus installations die:

python 
# Catastrophic — one series per user, forever
REQS.labels(user_id=user.id).inc()

# Fine — bounded set
REQS.labels(method="GET", status="200").inc()

Never put an unbounded value in a label: user ids, request ids, email addresses, full URL paths with ids in them. Templatise the path to /orders/{id} before it becomes a label.

The rule of thumb: a label is safe if you can name every value it will ever take. High-cardinality data belongs in logs or traces, which are built for it.

Recording rules

Precompute expensive queries so dashboards and alerts read a cheap series:

yaml 
groups:
  - name: api
    rules:
      - record: job:http_errors:rate5m
        expr: sum by (job) (
          rate(http_requests_total{status=~"5.."}[5m]))

What it is not

  • Not for logs. No text search. That is Loki or Elasticsearch.
  • Not long-term storage by default — local retention is finite. Remote write to Thanos, Mimir or Cortex for durability and global query.
  • Not for billing. Scrapes are sampled and lossy by design; a missed scrape is data you never get back. Anything requiring exactness needs an event log.
  • Grafana — the query and dashboard layer
  • OpenTelemetry — the vendor-neutral alternative for instrumentation
  • Kubernetes — where service discovery comes from

Interview angle 5

  • “How does Prometheus collect data?” - it scrapes HTTP endpoints on an interval; targets expose metrics rather than pushing them. Short-lived jobs that can’t be scraped use a Pushgateway, which is the documented exception rather than the norm.
  • “Which metric type for what?” - counter for monotonically increasing totals (requests, errors), gauge for values that go up and down (queue depth, memory), histogram for latency distributions so you can compute percentiles. Using a gauge for a counter loses information on restart.
  • “How do you compute a request rate?” - rate() over a counter, which handles resets automatically. Alerting on a raw counter value is meaningless; you almost always want a rate or an increase over a window.
  • “Histogram or summary for latency?” - histogram. It ships buckets and computes the quantile server-side, so you can aggregate across instances. A summary computes quantiles client-side, and quantiles cannot be averaged, so per-pod p99s can’t be combined into a service p99.
  • “How do you kill a Prometheus server?” - unbounded label cardinality. Every distinct label combination is a series in memory, so a user id or raw URL path as a label grows without limit. Templatise paths and keep labels to values you can enumerate.
Previous
Grafana