Course contentsShow
← Software Engineering
Lesson 1513 of 1,93344. Cloud Architecture PatternsFree lesson

Managed Message Queues and Event Streams

Leveraging managed messaging services like SQS, SNS, and Kafka; understanding delivery guarantees and scaling.

Managed Message Queues and Event Streams

What you'll learn: How to decouple services using cloud-managed messaging platforms and when to choose queues versus streams.

The Core Idea

Managed message queues and event streams are cloud services that handle asynchronous communication between your application components without you managing the underlying infrastructure. Think of them as postal services: queues are like mailboxes where messages wait until someone retrieves them, while event streams are like newspapers that multiple subscribers can read independently.

Queue vs Stream: Key Differences

Message Queues (like AWS SQS, Azure Queue Storage) deliver messages to one consumer who removes it from the queue. They're perfect for work distribution—imagine ten workers pulling tasks from a shared to-do list.

Event Streams (like AWS Kinesis, Azure Event Hubs, or managed Kafka) preserve messages in an ordered log that multiple consumers can read independently. Each consumer tracks their own position. It's like a DVR: everyone can watch the same recording at their own pace.

Delivery Guarantees

Most managed services offer:

  • At-most-once: Fire and forget (may lose messages)
  • At-least-once: Guaranteed delivery but possible duplicates
  • Exactly-once: Delivered once, no duplicates (hardest to achieve)

Your application must be idempotent if you choose at-least-once delivery.

Scaling Considerations

Managed services auto-scale infrastructure, but you control partition/shard count for throughput. More partitions = higher parallel processing capacity but increased complexity. Queues typically scale automatically, while streams require explicit partition management.

Services like SNS (pub/sub) can fan out to multiple queues or endpoints, combining patterns for sophisticated architectures.

Key Takeaway: Managed messaging services eliminate operational overhead while enabling loosely coupled, scalable architectures—choose queues for task distribution and streams for event sourcing or analytics pipelines.