Course contentsShow
← System Design
Lesson 728 of 1,91919. Event Streaming PlatformsFree lesson

AWS Kinesis Overview

Understand AWS Kinesis streams, data retention, sharding model, and how it differs from Kafka in deployment and scaling.

AWS Kinesis Overview

What you'll learn: How AWS Kinesis provides managed event streaming and differs from self-hosted Kafka.

What is AWS Kinesis?

AWS Kinesis is Amazon's fully managed event streaming platform. Like Kafka, it collects, processes, and stores real-time data streams, but as a cloud service you don't manage servers yourself. You create Kinesis Data Streams that act like Kafka topics—ordered sequences of data records that multiple applications can read independently.

Core Concepts

Shards

Kinesis uses shards instead of partitions. Each shard provides:

  • 1 MB/sec input or 1,000 records/sec
  • 2 MB/sec output
  • Fixed capacity units you provision upfront

Think of shards like checkout lanes at a store—you decide how many lanes to open based on expected traffic.

Data Retention

Streams retain data for 24 hours by default, extendable up to 365 days. This is similar to Kafka's retention policies, allowing consumers to replay historical events.

Ordering

Records with the same partition key go to the same shard, guaranteeing order within that shard—just like Kafka's key-based partitioning.

Key Differences from Kafka

AspectKinesisKafka
DeploymentFully managedSelf-managed (or managed services)
ScalingManual shard splitting/mergingAdd brokers, reassign partitions
Capacity modelPre-provisioned per shardBroker disk/network based
PricingPer shard-hour + data transferInfrastructure costs
AWS IntegrationNative (Lambda, S3, etc.)Requires connectors

Key Takeaway: AWS Kinesis offers managed event streaming with a shard-based capacity model, trading Kafka's operational flexibility for cloud-native simplicity and tight AWS ecosystem integration.