Progressive delivery on Kubernetes — Argo Rollouts & Flagger

Progressive delivery = deployment strategies (Deployment Strategies) run as an automated, metric-gated process: shift a slice of traffic, watch the metrics, promote or roll back — no human staring at Grafana. Canary/blue-green describe the shape; progressive delivery is the machinery.

Why a stock k8s Deployment can’t do this

A Deployment gives you rolling updates only — and rolling has three gaps:

• No traffic control: replicas flip old→new; you can’t say “5% of traffic to v2.” With 4 pods, the granularity is 25%, decided by pod count, not intent.
• No metric gate: readiness probes (Probes and HPA) check “is the pod up,” not “did the error rate double.” A pod that starts cleanly and corrupts 2% of requests rolls out to 100%.
• No automatic rollback: a bad rollout sits there until a human runs kubectl rollout undo.

Progressive-delivery controllers close all three: traffic shifting via ingress/mesh, promotion gated on real metrics, rollback automatic on failure.

Argo Rollouts

Replaces Deployment with a Rollout CRD — same pod template, plus a strategy:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        # 5% of traffic to the canary
        - setWeight: 5
        - pause: {duration: 10m}
        # metric gate — promotion stops here on failure
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 25
        - pause: {duration: 10m}
        - setWeight: 50
        - pause: {}             # manual approval gate (pause forever until promoted)
  # ...template as in a Deployment

The gate is an AnalysisTemplate — typically a Prometheus query with a pass condition:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      interval: 1m
      failureLimit: 3
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{service="checkout",status!~"5.."}[5m]))
            / sum(rate(http_requests_total{service="checkout"}[5m]))

Analysis fails → Rollout aborts and traffic snaps back to stable. It also supports blueGreen (preview service, prePromotionAnalysis, instant switch) — the blue-green flow from Deployment Strategies with the gate built in.

Fine-grained traffic percentages need an ingress/mesh integration (NGINX Ingress, ALB, Istio, Linkerd/SMI); without one, Rollouts approximates weight by scaling replica counts.

Flagger

Same goal, inverted ergonomics: you keep your plain Deployment; Flagger is an operator that watches it plus a Canary custom resource. On image change it clones the Deployment into -primary, sends weighted traffic to the canary through the mesh/ingress, steps the weight up while checking metrics (built-in success-rate/latency checks, plus custom PromQL), and promotes or rolls back. Webhooks slot in load tests or manual gates.

Both are CNCF-standard; the interview answer is knowing one concretely and the trade-off table.

What makes a good gate metric

Gate on symptoms users feel, measured on the canary pods only (label-scoped queries):

• HTTP 5xx / success rate, p95–p99 latency — the RED basics (Prometheus).
• Domain metrics for the money path: payment-auth decline rate, checkout conversion.
• Tie thresholds to your SLOs — a canary burning error budget is the definition of “roll back” (SLO, SLI, SLA, and Error Budgets).

Pitfalls

• Not enough traffic for significance: 5% of a low-QPS service = a handful of requests per interval; one flaky request “fails” the gate, or real breakage passes. Use longer windows, higher starting weights, or synthetic load via webhooks.
• Sticky/stateful traffic: weighted routing assumes any pod can serve any request; session affinity or in-pod state skews both traffic and metrics (Stateful vs Stateless (REST and beyond) — stateless services are the prerequisite).
• Database migrations don’t canary: schema is shared by old and new versions simultaneously — expand/contract discipline still applies (Zero-downtime migrations).
• Gate on averages: a 1% cohort disaster vanishes in a global average — scope queries to canary pods, alert on ratios not counts.
• Metrics lag (scrape + rate windows) — pauses shorter than ~2× the window gate on noise.