Message Durability and Persistence
You'll learn how message brokers ensure messages survive failures and the trade-offs between reliability and speed.
When a producer sends a message to a broker, a critical question arises: What happens if the broker crashes? Message durability and persistence determine whether that message survives hardware failures, restarts, or network issues.
How Brokers Persist Messages
Message brokers can operate in two modes:
In-memory only: Messages exist solely in RAM. Lightning-fast, but if the broker crashes, all messages vanish. Think of writing notes on a whiteboard—quick, but easily erased.
Persisted to disk: Messages are written to disk storage before acknowledging receipt. The broker can recover messages after a restart by reading from disk. Like writing in a notebook—slower, but permanent.
Most production brokers (Kafka, RabbitMQ, ActiveMQ) offer configurable persistence. You can choose per-topic or per-queue whether messages should be durable.
Durability Guarantees
When a producer publishes a message, the broker can acknowledge it at different points:
- Immediately (before disk write): Fast, but risky
- After writing to disk: Slower, but survives crashes
- After replication to multiple nodes: Slowest, highest guarantee
The acknowledgment tells the producer "your message is safe," but how safe depends on the persistence strategy.
Performance vs Reliability Trade-offs
Disk writes are significantly slower than memory operations. Persisting every message adds latency (milliseconds instead of microseconds) and reduces throughput.
Use persistence when: Messages represent critical business events (payments, orders)
Skip persistence when: Messages are transient notifications or easily regenerated data (cache invalidations, metrics)
Some brokers batch disk writes or use write-ahead logs (WALs) to balance performance with durability.
Key Takeaway: Message durability ensures messages survive broker failures by persisting them to disk, but this trades performance for reliability—choose based on your consistency and availability requirements.