Cache consistency

Article.

How do you keep a cache consistent with the database when reads and writes race? “Delete the key when you write” is the usual answer, and it is not enough: a reader that missed a moment earlier can put the old value back after your delete, and it then stays until the TTL expires.

Senior answers name that interleaving, the fixes that actually close it (leases, versioned writes), what each costs, and the fact that the TTL is the only thing that bounds every remaining case.

This note is about correctness. Stampedes in depth are in Cache Stampede and Mitigation Patterns, Redis itself in What is Redis, HTTP and CDN caching in HTTP semantics and caching.

The four patterns, and where each goes stale

• Cache-aside (lazy loading) is the default: on a miss the application reads the database and fills the cache; on a write it updates the database and deletes the key. Deleting is better than setting the new value, because two writers setting values can finish in the opposite order from their database commits.
• Write-through is often described as “never stale”. It is not: writer A commits 1, writer B commits 2, B’s cache SET lands, then A’s lands, and the cache holds 1 forever (or until TTL). And if the cache write fails after the database commit, the two disagree with no one noticing.
• Write-behind acknowledges a write once it is in the cache and flushes to the database later. It makes the cache the system of record for those seconds; a lost cache node loses acknowledged writes. Reserve it for data you can afford to lose or rebuild (counters, view counts).

The race delete-on-write does not close

The interleaving, run with an in-memory cache standing in for Redis:

--- cache-aside, delete on write
reader  cache miss
reader  read DB: old@example.com
writer  UPDATE DB -> new@example.com
writer  DELETE cache key
reader  SET old@example.com
result  cache=old@example.com  db=new@example.com

The reader read the old row, was slow for 20 ms (a GC pause, a busy event loop, a slow network), and wrote the old value after the writer’s delete. Nothing will correct it until the TTL expires. The window is small, but at thousands of requests per second on a hot key it is hit daily.

The replica-lag variant

The same thing happens with no slow reader at all, if misses are served from a read replica:

1. The writer updates the primary and deletes the key.
2. A reader misses and reads from a replica that is 300 ms behind.
3. It fills the cache with the pre-update row.

Under replication lag, “delete after write” deletes too early. Misses for recently written keys must read from the primary, or the invalidation must be delayed past the lag.

The fixes, and what each costs

Leases: reject a fill that has been overtaken

Facebook’s memcache (Scaling Memcache at Facebook, NSDI 2013) issues a lease token with every miss. The reader’s SET must present the token, and a delete in between voids it:

--- with leases
reader  cache miss
reader  read DB: old@example.com
writer  UPDATE DB -> new@example.com
writer  DELETE cache key
reader  SET old@example.com with lease 1: REJECTED
result  cache=None  db=new@example.com

The next reader misses again and loads the new row. The same mechanism rations misses: while a lease is outstanding, other readers wait briefly or get a stale value instead of all hitting the database, which also solves the stampede for that key. Redis has no built-in leases; a Lua script with a token in a side key gives the same effect.

Versioned writes: the cache keeps the newest

If rows carry a version (a version column incremented on every update, or an LSN), the cache can refuse to go backwards. One atomic script in Redis:

-- KEYS[1]=key  ARGV[1]=version  ARGV[2]=payload  ARGV[3]=ttl seconds
local cur = tonumber(redis.call('HGET', KEYS[1], 'v') or '-1')
if tonumber(ARGV[1]) <= cur then
  return 0                                  -- older than what we hold
end
redis.call('HSET', KEYS[1], 'v', ARGV[1], 'd', ARGV[2])
redis.call('EXPIRE', KEYS[1], ARGV[3])
return 1

The catch: on a write you must not simply delete the key, or a slow reader finds no version to compare with and writes its old one. The writer either sets the new value with its version, or leaves a short-lived tombstone holding the new version and no payload.

Delayed double delete

Delete on write, then delete again after a delay longer than replication lag plus a read (often 0.5-1 s, from a queue or a scheduled task). It narrows the window to readers slower than the delay; it does not close it, and it doubles invalidation traffic. Cheap and common, which is why interviewers ask what it still misses.

Invalidation from the database log (CDC)

Debezium (or a logical replication consumer) reads committed changes from the WAL and deletes the affected keys. Invalidation then happens only after commit, for every writer, including scripts and migrations that bypass the application. Costs: another pipeline to run, invalidation that lags by its delay, and per-key ordering only if events for one key stay on one partition.

It narrows the race (the delete arrives later) but a slow reader can still refill, so pair it with leases or versions where it matters. Ordering caveats are the same as for the outbox: The transactional outbox.

The TTL bounds everything

Every pattern above leaves some case open: a lost invalidation message, a deploy bug, a manual UPDATE in production. The TTL is the upper bound on how long any of them can serve wrong data, so choose it from the business question “how stale may this be?”, not from the hit rate. Add jitter so keys written together do not expire together (Cache Stampede and Mitigation Patterns).

What to cache and how to know it works

• Cache what is read far more than written, and expensive to compute: a profile page, a permission set, a rendered product listing. Data that changes every request gains nothing.
• Negative caching: cache “not found” too, with a short TTL (30-60 s), or a stream of requests for IDs that do not exist (bots, enumeration) goes straight to the database every time. A Bloom filter of existing IDs in front of the lookup rejects impossible keys without storing each miss.
• Hit ratio decides the database’s load: origin reads = QPS x (1 − hit ratio). Dropping from 99% to 98% doubles database reads; 99% to 90% is 10x. Watch it per key prefix, not only globally.
• Measure staleness, not just hits: sample a small fraction of cache hits, read the same key from the primary, and count mismatches. A stale-read rate of zero proves the invalidation path; a rate above zero after a deploy finds the bug before users do.

Layers: in-process, Redis, CDN

• In-process caches multiply: 50 pods x 4 workers is 200 independent copies, and a delete reaches none of them. Use them only for data that may be stale for the whole TTL (feature flags, config), with short TTLs.
• Redis client-side caching is the coherent near-cache: with RESP3 the server tracks which keys a client has read and pushes an invalidation when they change. In redis-py 5.1+:

from redis import Redis
from redis.cache import CacheConfig

r = Redis(host="cache", protocol=3, cache_config=CacheConfig(max_size=10_000))
r.get("user:42")      # network
r.get("user:42")      # served locally until Redis pushes an invalidation

If the connection drops, invalidations sent meanwhile are lost, so the client must flush its local cache on reconnect; check that yours does.

• CDN caching is for responses shared by many users; anything personalised needs Cache-Control: private, or one user’s page is served to another.

Deploys and the shape of cached data

During a rolling deploy, old and new code read each other’s cached values. A renamed field breaks deserialisation in one version or serves a half migrated object. Either keep cached payloads backward compatible (add fields, never rename), or version the key prefix (user:v2:42) so each version reads its own entries. A prefix bump is a cold cache for that data: warm the hot keys before shifting traffic, or the deploy is a stampede.

Where it breaks in production

• “I updated it and users still see the old value for an hour”: the read/write race or a lagging replica refilled the key, and the TTL is an hour. Signal: the sampled stale-read rate.
• A cache flush or failover takes the database down: the database was sized for a 1% miss rate. Signal: hit ratio and database CPU together; defences in Cache Stampede and Mitigation Patterns.
• Evictions you did not expect: Redis’s default maxmemory-policy is noeviction (writes fail when full), while ElastiCache’s default parameter group uses volatile-lru, which never evicts keys without a TTL. Signals: evicted_keys, used_memory; see Persistence, eviction and memory.
• One hot key saturates one shard: a celebrity profile or a global config key. Find it with redis-cli --hotkeys (needs an LFU eviction policy); fix it with an in-process layer for that key or replicas of it.
• Data leaking between users: a shared CDN or application cache keyed without the user or tenant.

The trade not taken

No cache. Before adding one, check whether the read is slow because it lacks an index, could be a materialized view, or could go to a read replica. Each of those stays consistent (or consistently lagged) without an invalidation path to get wrong. A cache is a second copy of the data with its own failure modes; add it when the read rate, not a missing index, is the problem.

The cache as the source of truth (write-behind for everything) trades durability for write latency. It is right for counters and rate-limit windows, wrong for orders and balances.

Related

• Cache Stampede and Mitigation Patterns - expiry storms, locks, jitter, early refresh
• What is Redis - the Redis model and data structures
• Persistence, eviction and memory - eviction, memory, metrics
• Worked Design — Distributed Cache - designing the cache tier
• HTTP semantics and caching - HTTP and CDN caching
• The transactional outbox - CDC and event ordering