Course contentsShow
← Cloud and Infrastructure
Lesson 1439 of 2,98929. Kubernetes ObservabilityFree lesson

Log Aggregation Backends

Sending logs to Elasticsearch, Loki, CloudWatch, or other centralized logging systems.

Log Aggregation Backends

What you'll learn: Where your Kubernetes logs go after collection agents (like Fluentd or Fluent Bit) forward them for long-term storage, search, and analysis.

Once your logging agent collects container logs from nodes, it needs to send them somewhere. That "somewhere" is a log aggregation backend—a centralized system designed to receive, index, store, and query massive volumes of logs from across your entire cluster.

Why Centralized Backends Matter

Imagine trying to troubleshoot a distributed application by SSH-ing into dozens of nodes and running kubectl logs repeatedly. It's like searching for a needle in a haystack... blindfolded. A log aggregation backend puts all your logs in one place with powerful search capabilities.

Common Backend Options

Elasticsearch

The most popular choice, Elasticsearch indexes logs for lightning-fast full-text search. Typically paired with Kibana for visualization (the "ELK stack"). Powerful but resource-intensive to operate.

Grafana Loki

A newer, cost-effective alternative inspired by Prometheus. Loki doesn't index log contents—only metadata labels—making it much lighter. You query logs like you query Prometheus metrics, using LogQL.

AWS CloudWatch Logs

If you're running on AWS, CloudWatch provides native integration. Your Fluent Bit agents can stream logs directly to CloudWatch, where you can search, create dashboards, and set alarms without managing infrastructure.

Others

Splunk (enterprise-focused), Datadog (SaaS observability platform), and self-hosted options like Graylog all serve as backends depending on your requirements and budget.

Configuration Flow

Your logging agent (Fluentd/Fluent Bit) receives logs from nodes, enriches them with metadata (namespace, pod name), then forwards them via HTTP, TCP, or vendor-specific protocols to your chosen backend. The backend handles persistence, indexing, and querying.

Key Takeaway: Log aggregation backends like Elasticsearch, Loki, and CloudWatch centralize logs from your entire cluster, transforming scattered container output into searchable, actionable observability data.