Apache Pulsar Architecture
What you'll learn: How Pulsar separates compute from storage and enables multi-tenancy, differentiating it from Kafka's integrated broker-storage model.
The Two-Layer Design
Unlike Kafka where brokers handle both message serving and storage, Apache Pulsar splits these responsibilities into two distinct layers:
- Brokers (compute layer): Stateless servers that handle client connections, message routing, and subscription management
- BookKeeper (storage layer): Distributed log storage system called "bookies" that durably stores messages
Think of it like separating a restaurant's waitstaff (brokers) from the kitchen storage (bookies). The waiters don't need to own the food inventory—they just serve customers and coordinate with the kitchen.
Why This Separation Matters
Independent Scaling: Need more read/write throughput? Add brokers. Need more storage? Add bookies. In Kafka, adding a broker adds both compute and storage whether you need it or not.
Fast Recovery: Since brokers are stateless, if one fails, another can immediately take over without data migration. The data stays safely in BookKeeper. Kafka brokers must replicate partition data, making rebalancing slower.
Multi-Tenancy Built-In: Pulsar natively supports hierarchical organization:
tenant/namespace/topicstructure- Per-tenant quotas, authentication, and isolation
- Kafka requires complex configurations or separate clusters for similar isolation
Kafka vs Pulsar Storage
Kafka tightly couples brokers with storage—each broker owns specific partition replicas on its local disk. This simplicity has tradeoffs: scaling storage means moving partitions, and broker failures require replica promotion.
Pulsar's BookKeeper uses striped storage across multiple bookies, distributing segments of a topic's log. This provides better load distribution and faster recovery.
Key Takeaway: Pulsar's broker-bookie separation provides independent scaling, stateless brokers for faster failover, and native multi-tenancy, while Kafka's unified broker approach offers operational simplicity at the cost of coupling compute and storage.