Course contentsShow
← System Design
Lesson 1148 of 1,91929. Observability: LoggingFree lesson

Centralized Logging Architecture

Why distributed systems need centralized log collection and the basic architecture of log aggregation pipelines.

Centralized Logging Architecture

What you'll learn: How distributed systems collect logs from many services into one place for analysis and troubleshooting.

The Problem with Scattered Logs

In a distributed system with dozens or hundreds of services running across multiple machines, each service writes its own logs locally. Imagine trying to debug a user request that touched five different services — you'd need to SSH into five servers, grep through five different log files, and mentally stitch together the timeline. This is slow, error-prone, and doesn't scale.

The Centralized Solution

Centralized logging means all services send their logs to a single, unified system where you can search, filter, and analyze them together. Think of it like a library: instead of books scattered across everyone's homes, they're all in one building with a catalog system.

Basic Architecture Components

The typical log aggregation pipeline has three layers:

  1. Log Shippers/Agents: Lightweight processes running on each service host that collect logs and forward them (e.g., Fluentd, Filebeat, Vector)
  2. Transport Layer: Message queues or buffers that handle log volume spikes without data loss (e.g., Kafka, Redis)
  3. Storage and Indexing: A centralized database optimized for log search and retention (e.g., Elasticsearch, Loki, S3 + Athena)

Services write structured logs (remember JSON format from previous lessons) which agents tail and ship. The transport layer decouples producers from consumers, preventing log loss during traffic spikes. The storage layer indexes logs by time, service, correlation ID, and other fields for fast querying.

Key Takeaway: Centralized logging architecture solves the distributed debugging problem by funneling all service logs through agents → transport → storage, enabling unified search and correlation across your entire system.