Pub/sub and Streams

Redis has two messaging primitives with the same shape and opposite guarantees. Choosing pub/sub for something that matters is a recurring production mistake, and the interview question is whether you know why.

Pub/sub is fire-and-forget

# Publisher
r.publish("orders", json.dumps(event))

# Subscriber
p = r.pubsub()
p.subscribe("orders")
for msg in p.listen():
    handle(msg["data"])

A message is delivered to whoever is connected at that instant, and then it is gone. No storage, no acknowledgement, no replay, no consumer groups. A subscriber that is restarting, redeploying or briefly disconnected simply misses everything sent in that window, and nothing anywhere records that it happened.

PUBLISH returns the number of subscribers that received it — which is the only feedback you get, and it is zero when nobody is listening.

Legitimate uses: cache invalidation broadcasts, live dashboards, presence, “something changed, go look”. Everything there tolerates a miss because a later message corrects it.

Streams are an append-only log

# Producer — the id is assigned and ordered.
r.xadd("orders", {"id": order_id, "total": "42.00"}, maxlen=100_000,
       approximate=True)

# Consumer group, created once.
r.xgroup_create("orders", "billing", id="0", mkstream=True)

while True:
    msgs = r.xreadgroup("billing", worker_name, {"orders": ">"},
                        count=10, block=5000)
    for _, entries in msgs:
        for msg_id, fields in entries:
            handle(fields)
            # explicit
            r.xack("orders", "billing", msg_id)

Everything pub/sub lacks is here: messages persist, > delivers only what this group has not seen, and an unacknowledged message stays in the group’s pending list so a crashed worker’s work is recoverable rather than lost.

Recovering a dead consumer’s work

The pending list is the mechanism, and XAUTOCLAIM is how you drain it:

# Reassign anything pending for more than 60s to me.
r.xautoclaim("orders", "billing", worker_name,
             min_idle_time=60_000, count=10)

Without this, a worker that dies mid-message leaves that message pending forever — delivered, never acknowledged, never retried. A claim loop is not optional in production, and forgetting it is the Streams equivalent of choosing pub/sub.

A message claimed repeatedly is a poison pill: check its delivery count and route it to a dead-letter stream rather than looping.

Trimming, because a log grows

XADD with maxlen and approximate=True trims as it writes and is far cheaper than exact trimming, since it only removes whole nodes. Without a maxlen or a periodic XTRIM MINID, the stream grows until memory runs out — this is the same failure as unpruned execution data anywhere else.

The comparison

And against a real broker

Streams cover a lot of what people use RabbitMQ or Kafka for, and the honest limits are worth stating rather than defending:

• No routing. No exchanges, no topic patterns — see RabbitMQ exchanges and routing.
• Retention is memory. Kafka retains days on disk cheaply; Redis retains in RAM, so long retention is expensive.
• Durability is Redis’s durability — asynchronous replication, so a failover can lose recent entries.

The defensible position: Streams if you already run Redis and the volume is moderate; a broker when the log is the system of record. See Kafka vs RabbitMQ — choosing a message broker.

Related

• What is Redis — why a List is not a safe queue
• Persistence, eviction and memory — trimming and memory
• Kafka vs RabbitMQ — choosing a message broker — queue against log
• Celery Task Idempotency — at-least-once means idempotent consumers