AWS Kinesis Overview
What you'll learn: How AWS Kinesis provides managed event streaming and differs from self-hosted Kafka.
What is AWS Kinesis?
AWS Kinesis is Amazon's fully managed event streaming platform. Like Kafka, it collects, processes, and stores real-time data streams, but as a cloud service you don't manage servers yourself. You create Kinesis Data Streams that act like Kafka topics—ordered sequences of data records that multiple applications can read independently.
Core Concepts
Shards
Kinesis uses shards instead of partitions. Each shard provides:
- 1 MB/sec input or 1,000 records/sec
- 2 MB/sec output
- Fixed capacity units you provision upfront
Think of shards like checkout lanes at a store—you decide how many lanes to open based on expected traffic.
Data Retention
Streams retain data for 24 hours by default, extendable up to 365 days. This is similar to Kafka's retention policies, allowing consumers to replay historical events.
Ordering
Records with the same partition key go to the same shard, guaranteeing order within that shard—just like Kafka's key-based partitioning.
Key Differences from Kafka
| Aspect | Kinesis | Kafka |
|---|---|---|
| Deployment | Fully managed | Self-managed (or managed services) |
| Scaling | Manual shard splitting/merging | Add brokers, reassign partitions |
| Capacity model | Pre-provisioned per shard | Broker disk/network based |
| Pricing | Per shard-hour + data transfer | Infrastructure costs |
| AWS Integration | Native (Lambda, S3, etc.) | Requires connectors |
Key Takeaway: AWS Kinesis offers managed event streaming with a shard-based capacity model, trading Kafka's operational flexibility for cloud-native simplicity and tight AWS ecosystem integration.