Queue Persistence and Durability
What you'll learn: How message queues store messages to disk, recover from crashes, and balance the tradeoff between reliability and speed.
The Core Challenge
When a message queue receives a message, it faces a critical decision: should it keep the message only in memory (fast but risky) or write it to disk (slower but safer)? If the queue crashes before delivering a message, persistence ensures that message isn't lost forever. Durability means that once the queue acknowledges receipt of a message, that guarantee survives crashes and restarts.
Think of it like a restaurant taking orders. A waiter could memorize orders (fast, but if they forget or leave, orders are lost) or write them down on paper (slightly slower, but orders survive even if the waiter changes shifts).
Storing Messages to Disk
Most production queue systems write messages to an append-only log on disk. This structure is efficient because writing sequentially to disk is much faster than random writes. The queue maintains an in-memory index pointing to which messages are pending, delivered, or acknowledged.
Recovery After Crashes
When a queue restarts after a crash, it reads its persistent log from disk and reconstructs its state: which messages were sent, which were acknowledged, and which need redelivery. This is where your earlier lesson on message acknowledgment and redelivery connects—unacknowledged messages get resent.
The Performance Tradeoff
Full durability (flushing every write to disk immediately) can reduce throughput by 10-100x compared to memory-only queues. Systems often offer configuration options: you might batch writes, sync to disk every N messages, or use faster storage like SSDs. Some systems even offer async durability, acknowledging messages before disk writes complete—faster but riskier.
Key Takeaway: Queue persistence writes messages to disk to survive crashes, enabling recovery and durability guarantees, but this safety comes at a performance cost that must be carefully balanced against your system's reliability requirements.