Backend / Caching / Redis / 04_persistence_and_memory.md

Persistence, eviction and memory

Updated 7 interview angles 5 min read source
On this page7
  1. RDB and AOF
  2. Eviction: only fires when maxmemory is set
  3. Expiry is lazy plus sampled
  4. Watching the right numbers
  5. Making the data smaller
  6. Related
  7. Interview angle

Persistence, eviction and memory

The two questions behind every Redis production incident: what happens on restart, and what happens when it fills up. They have different answers and people routinely conflate them.

RDB and AOF

RDB AOF
What it stores periodic snapshot every write command
Restart cost fast load slower replay
Worst-case loss since last snapshot 1 second (everysec)
File size compact larger, rewritten periodically
Cost while running fork, copy-on-write append, fsync
text 
# redis.conf — both, which is the usual production answer
save 900 1                # snapshot if ≥1 key changed in 15 min
save 300 10
appendonly yes
appendfsync everysec      # always | everysec | no

appendfsync always is the only setting that survives a power cut with no loss, and it costs an fsync per write. everysec is the default because the trade is right for almost everyone: bounded loss of one second.

Gotcha: the RDB fork is the surprise. Saving copies the page table and copy-on-write duplicates pages as they change, so a write-heavy 8 GB instance can spike well past 8 GB during a snapshot and get OOM-killed. Size the box for the fork, not for the dataset, or move snapshots to a replica.

None of this makes Redis durable in the database sense. Replication is asynchronous, so a failover promotes a replica that may be missing the last writes. Say that plainly — it is the answer to “can we use Redis as the source of truth” and the answer is no.

Eviction: only fires when maxmemory is set

text 
maxmemory 4gb
maxmemory-policy allkeys-lru

The default policy is noeviction, and that is the single most common production misconfiguration: memory fills, and writes start failing with OOM errors while reads keep working. A cache should almost never be noeviction.

Policy Evicts
noeviction nothing — writes error (default)
allkeys-lru least recently used, any key
allkeys-lfu least frequently used, any key
volatile-lru LRU, only keys with a TTL
volatile-ttl soonest to expire
allkeys-random a random key

Choosing between them:

  • allkeys-lru for a pure cache. Safe default.
  • allkeys-lfu when access is skewed — LFU keeps the genuinely hot keys instead of whatever a scan touched last.
  • volatile-* only when the instance mixes cache and persistent data, and then the persistent keys must have no TTL. If nothing has a TTL, a volatile-* policy evicts nothing and behaves like noeviction.

Redis’s LRU and LFU are approximate — sampled, not exact — which is a deliberate trade and worth knowing.

Expiry is lazy plus sampled

A key with a TTL is not deleted at that instant. It goes when something touches it, or when the background cycle samples it. So INFO memory can show memory held by keys that are logically gone, and DBSIZE can overcount.

Two consequences: never infer “the cache is full of live data” from memory alone, and set TTLs with jitter so expiry work is spread rather than bunched.

Watching the right numbers

bash 
redis-cli INFO memory | grep -E 'used_memory_human|maxmemory_human|fragmentation'
redis-cli INFO stats | grep -E 'evicted_keys|keyspace_hits|keyspace_misses'
# sampled: what is actually large
redis-cli --bigkeys
redis-cli MEMORY USAGE key
Signal Means
evicted_keys rising undersized, or TTLs too long
hit rate falling eviction is throwing away useful data
mem_fragmentation_ratio > 1.5 fragmentation; consider activedefrag
used_memory ≈ maxmemory steady state for a cache — not an alarm

The one to alert on is the hit rate, not memory. A cache at 100% of maxmemory with a 95% hit rate is working exactly as intended.

Making the data smaller

  • Shorten key names. u:1:s against user:1:session is real money at a hundred million keys, since the key is stored per entry.
  • Prefer a hash to many keys. Small hashes use a compact encoding (hash-max-listpack-entries), so user:1 as one hash beats ten user:1:field strings.
  • Compress large values in the client, and store bytes.
  • Set a TTL on everything. A key with no expiry is permanent by accident.

Interview angle 7

  • “RDB or AOF?” - usually both. RDB is a compact periodic snapshot that loads fast; AOF logs every write and bounds loss to a second at everysec. AOF alone replays slowly on restart, RDB alone loses everything since the last snapshot.
  • “What’s the hidden cost of RDB?” - the fork. Copy-on-write duplicates pages as they are written, so a write-heavy instance can spike to nearly double its dataset size during a snapshot and get OOM-killed. Size for the fork or snapshot from a replica.
  • “Is Redis durable?” - not in the database sense. Replication is asynchronous, so a failover promotes a replica that may be missing recent writes. It is a cache or a loss-tolerant store, not a system of record.
  • “What’s the default eviction policy?” - noeviction, which is the most common production misconfiguration: memory fills and writes start failing while reads keep working. A cache wants allkeys-lru, or allkeys-lfu when access is skewed.
  • “When would volatile-lru evict nothing?” - when no key has a TTL. The volatile-* policies only consider keys with an expiry, so on an instance where nothing expires they behave exactly like noeviction.
  • “What do you alert on?” - the hit rate and evicted_keys, not memory usage. A cache sitting at 100% of maxmemory with a high hit rate is working correctly; a falling hit rate means eviction is discarding data you needed.
  • “Why does memory not drop when keys expire?” - expiry is lazy plus sampled, not immediate. A key goes when something touches it or the background cycle finds it, so memory and DBSIZE both lag reality.
Previous
Redlock and the Kleppmann Critique