Stream Processing — Windowing, Watermarks, State

The difference between “consuming Kafka” and “doing stream processing” is windowing, watermarks, and managed state. Senior interviews probe whether you understand them.

Stream vs batch — the fundamental difference

Batch: all data is present at compute time. Bounded. Run once → finite result.

Stream: data arrives continuously. Unbounded. Computation produces evolving results over time.

Stream processing systems treat batches as a special case (“bounded streams”). The same operators that process live events can re-process historical data — same code, same semantics.

Windowing

Aggregating over an unbounded stream requires picking a window — a finite slice of the stream over which to aggregate.

Tumbling windows

Fixed-size, non-overlapping:

[0:00-0:05] [0:05-0:10] [0:10-0:15] [0:15-0:20]

Every event lands in exactly one window. Used for “events per 5 minutes,” “transactions per hour.”

Sliding windows

Fixed-size, overlapping by a sliding interval:

Window size: 10 min, slide: 5 min
[0:00-0:10] [0:05-0:15] [0:10-0:20] [0:15-0:25]

Each event lands in window_size / slide windows. “Rolling 10-minute average updated every 5 minutes.”

Session windows

Variable size — defined by gaps in activity. A new window starts after N seconds of inactivity from the previous event.

User activity gap > 30 min → new session
[User opens app for 10 min, closes, returns 1 hour later] → 2 sessions

Used for: user session analytics, fraud-detection clusters.

Hopping vs sliding terminology

“Hopping window” (Kafka Streams) and “sliding window” (Flink) refer to the same thing: overlapping fixed-size windows. The Flink-style “true sliding” recomputes the aggregate at every event (instead of at slide boundaries) — more accurate, more expensive.

Watermarks

When you say “5-minute window from 00:00 to 00:05,” when do you actually close the window and emit results?

Naive: at wall-clock 00:05. But:

• Some events have event-time before wall-clock 00:00 (network delay, replay).
• Some have event-time 00:04:59 but arrive at 00:05:30.

If you close at 00:05 and emit, the 00:04:59 event arriving at 00:05:30 is late and excluded.

Watermarks are a heuristic for “no more events with timestamps below this.” Stream processors emit a watermark W(t) meaning: “I won’t see any event with timestamp < t.”

When watermark exceeds the window’s end, the window closes and emits.

events:     | E(00:01) E(00:03) E(00:04:30) E(00:04:55) |
window:     [00:00 - 00:05]
watermark:  emits at e.g. 00:05:30 → "no events older than 00:05:30 expected"
                → close window, emit aggregate over events 00:01-00:04:55
late event: E(00:04:50) arriving at 00:06 → arrives after watermark → late

Handling late events

• Drop — simplest; report dropped count for monitoring.
• Side output — route late events to a separate stream for special handling.
• Allowed lateness — keep the window open for N additional minutes after watermark, accepting late updates. Emits multiple results per window (updates).

Trade-off: latency vs completeness. Tight watermark = low latency, more late events. Loose watermark = high latency, fewer late events.

Event time vs processing time

Production stream processing usually uses event time + watermarks. Processing time is fine for monitoring / metrics where exact bucketing doesn’t matter.

State

Stream operators often carry state:

• Running totals (sum, count, avg) per group.
• Open windows being built.
• Session tracking per user.
• Joins (one side waiting for the matching record on the other).

State must:

• Persist across operator restarts.
• Scale beyond one machine’s RAM.
• Be consistent with input offsets for exactly-once semantics.

Stream processors provide managed state backends — typically RocksDB on local disk + periodic checkpoints to S3 / HDFS / similar.

event → operator (state lookup / update) → emit
                  ↓
            RocksDB local store
                  ↓ (periodic snapshot)
            S3 / durable storage

On restart, restore state from the last checkpoint, replay events since.

Exactly-once processing

The goal: each input event affects the output exactly once, even across failures.

Achieved via:

1. Idempotent / transactional sinks. Writing to Kafka with transactional producer; writing to a DB with idempotency keys; writing to Delta Lake with checkpoints.
2. Atomic checkpoints. State snapshot + consumer offset atomic. On restart, restore state + resume from the saved offset. No event processed twice; none lost.
3. Distributed coordination. Chandy-Lamport-style barriers (Flink) or Kafka transactions (Kafka Streams) to align checkpoints across operators.

This is genuinely hard. Use a stream processor that provides it; don’t roll your own.

Backpressure

When downstream can’t keep up:

• Upstream operators accumulate buffers.
• Memory fills.
• System OOMs.

Stream processors handle backpressure by:

• Slowing upstream consumption (Kafka consumer lag rises, but you don’t crash).
• Flow control between operators in the topology.

Symptom of unhealthy backpressure: consumer lag rises faster than catchup. You’re not processing as fast as data arrives; eventually you fall too far behind.

Mitigations:

• Scale the consumer fleet.
• Optimize the operator (fewer passes, vectorization).
• Drop / sample under sustained load (rare; ugly).

Python options

For a Python-first stream processor: Bytewax is the most credible modern option. For Spark shops: PySpark Structured Streaming. For Flink shops with Python support: PyFlink.

For truly low-latency requirements (sub-second), JVM Flink / Kafka Streams beats Python options. Python’s GIL caps throughput.

Faust example

import faust

app = faust.App("orders-stream", broker="kafka://localhost:9092")

orders_topic = app.topic("orders", value_type=Order)

@app.agent(orders_topic)
async def process(orders):
    async for order in orders.group_by(Order.user_id):
        # state-aware operation per user
        ...

Async-first, Kafka-native. Faust development has been sporadic; check current state before adopting.

Bytewax example

import bytewax.operators as op
from bytewax.dataflow import Dataflow
from bytewax.connectors.kafka import KafkaSource, KafkaSink

flow = Dataflow("orders-stream")
inp = op.input("kafka-in", flow, KafkaSource(brokers=["localhost:9092"], topics=["orders"]))
parsed = op.map("parse", inp, lambda msg: json.loads(msg.value))
windowed = op.window.tumbling_window(
    "5min",
    parsed,
    clock=op.window.EventClock(...),
    windower=op.window.TumblingWindower(length=timedelta(minutes=5)),
).reduce(...)
op.output("kafka-out", windowed, KafkaSink(...))

Rust-backed runtime; functional Python API. Promising direction for Python stream processing.

When to use stream processing

• Real-time aggregations. Top users per hour, fraud alerts, dashboards.
• Event-time analytics. Sessionization, funnel analysis.
• Continuous ETL. Stream into a data lake, materialize derived tables.
• Anomaly detection. Triggered alerts on patterns over time.

When NOT to use stream processing

• Daily / hourly batch is fine. Simpler; cheaper; more debuggable. Don’t over-engineer.
• Operational queries on point data. Just use a DB.
• No real-time SLA. Stream processing infra carries a cost.

Many “streaming” projects could be cheaper as scheduled batch with minor latency cost.