Backend / Data engineering / Data architecture / 04_lambda_and_kappa_architectures.md

Lambda and Kappa architectures

Updated 5 interview angles 4 min read source
On this page7
  1. The problem they solve
  2. Lambda: two pipelines, merged at read
  3. Kappa: one pipeline, replayed
  4. Which one to say in an interview
  5. What replaced the argument
  6. Related
  7. Interview angle

Lambda and Kappa architectures

Two answers to one problem: batch is accurate but hours late, streaming is immediate but approximate. Both names come up in data interviews, and the useful answer is that one of them is mostly history.

The problem they solve

You need a number - sessions per user, revenue per hour, fraud score. Compute it from the full history and it is correct but stale. Compute it from the live stream and it is fresh but has seen only part of the story, and any bug is baked into a result you cannot recompute.

Lambda: two pipelines, merged at read

Nathan Marz’s design runs both and reconciles at query time.

text 
      +--> batch --> batch views --+
ingest+                            +-> serving
      +--> speed --> rt views -----+
  • Batch layer owns the immutable master dataset and recomputes views from scratch. Slow, exact, and self-healing: a bug is fixed by rerunning.
  • Speed layer covers only the window the batch layer has not reached, usually with approximations, and is discarded once batch catches up.
  • Serving layer merges the two on read.

The self-healing property is the real argument. Because batch recomputes from raw data every cycle, any error in the speed layer has a bounded lifetime.

What it costs

The same business logic exists twice, in two runtimes. Sessionisation in Spark and again in Flink, in different languages, with different windowing semantics and different definitions of “late”.

They drift. The bug that eats a week is the one where batch and speed disagree by 0.3% and nobody can say which is right.

Kappa: one pipeline, replayed

Jay Kreps’ response was that the batch layer is a workaround for streaming engines that could not replay. Remove it.

text 
ingest --> log --> stream job --> views
             ^
             +-- replay from offset 0

There is one codebase. To fix a bug or change the logic, start a second instance of the job reading from the beginning of the log, let it build a parallel view, then cut over.

Recomputation stops being a different system and becomes the same system pointed at an older offset.

What makes it viable

Kappa only works when the log is a real system of record:

Requirement Why it matters
Durable retention Replay needs history, not a 7-day buffer
Event-time windows Reprocessing must not depend on wall clock
Exactly-once sinks Replay must not double-count
Cheap parallel views Cutover needs a shadow copy

Event time is the subtle one. If the job windows on processing time, a replay compresses months into minutes and every record lands in one window, so the rebuilt view does not match the original.

Which one to say in an interview

Say Kappa is the default and explain why the trade-off moved: Kafka retention became cheap, Flink and Structured Streaming got event-time and exactly-once, and object storage made a shadow view affordable. The reasons for two pipelines were engine limitations, and the engines improved.

Then name the case that still justifies Lambda: the correction has to touch data older than the retention window, or the batch source is authoritative and the stream is a preview of it. A nightly finance reconciliation against a warehouse of record is a batch job, and pretending otherwise is worse.

What replaced the argument

In a lakehouse, the distinction mostly dissolves. A Delta or Iceberg table is both a batch table and a stream, so the same table is appended to by a streaming write and read by a batch query.

python 
# One table, two access patterns.
(spark.readStream
      .format("delta")
      .load("/tables/events")
      .writeStream
      .format("delta")
      .outputMode("append")
      .option("checkpointLocation", "/ckpt/agg")
      .toTable("agg_events"))

The checkpoint is what makes this Kappa rather than a cron job: the stream records its offset, so a restart resumes instead of recomputing, and a rebuild is a matter of deleting the checkpoint.

This is what Databricks markets as medallion (bronze/silver/gold) and what Flink calls unified batch and streaming. If asked about Lambda today, the strongest answer names it, explains the two-codebase cost, and then says the storage layer removed the reason for it.

Interview angle 5

  • “Explain Lambda architecture.” - a batch layer recomputing exact views from immutable history, a speed layer covering the recent window with approximations, and a serving layer merging them on read. Its virtue is self-healing: batch recomputation bounds the lifetime of any speed-layer bug.
  • “What is wrong with it?” - the same logic lives twice in two runtimes with different windowing semantics, and they drift. The expensive failure is batch and speed disagreeing slightly with no authority to settle it.
  • “What is Kappa?” - one streaming pipeline over a durable replayable log. Reprocessing means replaying from an earlier offset into a parallel view and cutting over, so there is no second codebase.
  • “When does Kappa fail?” - when the log does not retain enough history to replay, when windows are keyed on processing time so a replay bunches everything into one window, or when the sink is not idempotent and replay double-counts.
  • “Which would you build now?” - neither as stated. A lakehouse table read as both stream and batch collapses the distinction; the checkpoint gives you replay, and the batch layer stops being a separate system. Lambda survives only where an authoritative batch source must reconcile beyond the stream’s retention.
Previous
Data catalogues, governance, and lineage