Offsets and Commit Strategies
What you'll learn: How Kafka tracks what each consumer has read and the different strategies for saving that progress.
What is an Offset?
An offset is a unique sequential number assigned to each message within a partition. Think of it like page numbers in a book—if you're reading Chapter 3 (partition 3), the offset tells you exactly which page (message) you've read up to. This number always increases: message 0, 1, 2, 3, and so on.
When a consumer reads messages from a partition, Kafka needs to remember where that consumer left off. Otherwise, if the consumer crashes and restarts, it wouldn't know whether to re-read messages or skip ahead.
Commit Strategies
Committing means saving your current offset position back to Kafka. There are two main approaches:
Auto-commit: Kafka automatically saves your progress at regular intervals (e.g., every 5 seconds). Simple, but if your consumer crashes between commits, you might re-process some messages (at-least-once) or skip messages if you crashed after committing but before processing (at-most-once).
Manual commit: Your application explicitly tells Kafka "I've processed up to offset 150." You control exactly when to commit, giving you more precision. You can commit after processing each message (safer but slower) or in batches (faster but riskier).
Delivery Semantics
Your commit strategy determines delivery semantics:
- At-least-once: Commit after processing. If you crash before committing, you'll reprocess messages.
- At-most-once: Commit before processing. If you crash after committing, those messages are gone forever.
Key Takeaway: Offsets are how Kafka tracks consumer progress through partitions, and your commit strategy (auto vs manual, before vs after processing) determines whether your system guarantees at-least-once or at-most-once delivery.