Redis from Python

The patterns that come up in a backend interview, and the details that make each one correct rather than approximately correct.

One client, decoded, pooled

python

pool = redis.ConnectionPool.from_url(
    settings.redis_url,
    max_connections=50,
    # str, not bytes, everywhere
    decode_responses=True,
    socket_timeout=1,
    socket_connect_timeout=1,
    health_check_interval=30,
)
r = redis.Redis(connection_pool=pool)

redis.Redis() is already pooled and thread-safe — build it once at startup and inject it, exactly as with an HTTP client. decode_responses=True set later is a migration; set it on day one.

health_check_interval is the one people omit: it pings idle connections, so a connection a load balancer silently closed is discovered before a user’s request finds it.

Pipelines are about round trips, not atomicity

python

# just batching
pipe = r.pipeline(transaction=False)
for uid in user_ids:
    pipe.hgetall(f"user:{uid}")
# one round trip
users = pipe.execute()

A thousand HGETALL calls is a thousand network round trips; pipelined it is one. On a 1 ms link that is one second against a few milliseconds, and it is the first thing to reach for when Redis “is slow” but Redis itself reports low latency.

transaction=False matters: the default wraps the batch in MULTI/EXEC, which you often do not need and which costs atomicity semantics you are not using.

Transactions: MULTI/EXEC is not a rollback

python

with r.pipeline() as pipe:
    while True:
        try:
            pipe.watch("balance")
            current = int(pipe.get("balance"))
            if current < amount:
                pipe.unwatch()
                raise Insufficient
            pipe.multi()
            pipe.set("balance", current - amount)
            # fails if balance changed
            pipe.execute()
            break
        except redis.WatchError:
            # someone else won; retry
            continue

Two things to say out loud. MULTI/EXEC queues commands and runs them atomically, but a command that fails at runtime does not roll back the others — there is no rollback in Redis. And WATCH gives optimistic concurrency: if the watched key changed, EXEC returns nil and you retry.

For anything more involved, a Lua script is simpler and genuinely atomic, because the whole script runs as one command:

python

DEBIT = r.register_script("""
  local bal = tonumber(redis.call('GET', KEYS[1]))
  if bal < tonumber(ARGV[1]) then return -1 end
  return redis.call('DECRBY', KEYS[1], ARGV[1])
""")
DEBIT(keys=["balance"], args=[amount])

Gotcha: pass every key through KEYS, never interpolate it into the script body. On Cluster the node is chosen from KEYS, so a hard-coded key name routes the script to the wrong node.

Rate limiting

The naive counter has a boundary flaw: with a 60-second window, a client can send the limit at 00:59 and again at 01:00. A sliding window over a sorted set does not:

python

LIMIT = r.register_script("""
  local now, window, limit = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])
  redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, now - window)
  local used = redis.call('ZCARD', KEYS[1])
  if used >= limit then return 0 end
  redis.call('ZADD', KEYS[1], now, ARGV[4])
  redis.call('EXPIRE', KEYS[1], window)
  return 1
""")
allowed = LIMIT(keys=[f"rl:{user}"], args=[now, 60, 100, uuid4().hex])

One script, so the check and the increment cannot interleave — the bug in every read-then-write rate limiter. The EXPIRE is what stops idle users’ keys accumulating forever.

Sessions

python

# sliding TTL on read
r.setex(f"sess:{sid}", 1800, json.dumps(data))

Sessions in Redis are what make an application horizontally scalable — any instance serves any request, which removes sticky sessions entirely. The trade is that Redis is now on the critical path for every authenticated request, so it needs the availability of Replication, Sentinel and Cluster, and losing it logs everyone out.

async, when the app is async

python

import redis.asyncio as redis

r = redis.Redis.from_url(URL, decode_responses=True)
await r.get("key")

Same API, same pooling. Using the synchronous client inside an async def handler blocks the event loop for every other request — the failure described in FastAPI and the event loop.

Interview angle 6

“How do you speed up many small Redis calls?” - pipeline them. A thousand round trips become one, which is the usual fix when the application says Redis is slow but Redis reports low command latency. Pass transaction=False if you only want batching.
“Does MULTI/EXEC give you rollback?” - no. Commands are queued and run atomically, but a runtime failure in one does not undo the others; there is no rollback in Redis. WATCH adds optimistic concurrency — EXEC returns nil if the watched key changed, and you retry.
“When would you use Lua instead?” - when several commands must be one atomic step with logic between them. The script runs as a single command. Pass keys via KEYS rather than interpolating them, or Cluster routes the script to the wrong node.
“Implement a rate limiter.” - a sorted set as a sliding window, all in one Lua script: drop entries older than the window, count, reject or add. A fixed counter allows double the limit across a window boundary, and a read-then-write version races.
“Why put sessions in Redis?” - it makes the application stateless, so any instance serves any request and sticky sessions disappear. The cost is that Redis is on the critical path for every authenticated request.
“What’s the one connection setting people forget?” - health_check_interval. Idle connections silently dropped by a load balancer are otherwise discovered by a user’s request rather than by a background ping.

Redis from Python

Redis from Python

One client, decoded, pooled

Pipelines are about round trips, not atomicity

Transactions: MULTI/EXEC is not a rollback

Rate limiting

Sessions

async, when the app is async

Related

Interview angle 6