In this lesson, you will learn how to deploy the ELK stack (Elasticsearch, Logstash, and Kibana) to provide a centralized, scalable backend for storing, processing, and visualizing your log data.
While Prometheus focuses on time-series metrics, the ELK stack is designed for unstructured log data. When you aggregate thousands of container logs, searching through them via kubectl logs becomes impossible. The ELK stack solves this by transforming raw text into searchable, indexed documents.
The ELK Architecture
- Elasticsearch: A distributed search and analytics engine. It acts as the "brain," storing the log data and providing the high-speed search capabilities required to query millions of log lines.
- Logstash: The data ingestion engine. It receives logs from your forwarders (like Fluent Bit), parses them into structured JSON, and pushes them into Elasticsearch.
- Kibana: The visualization layer. It provides a UI to create dashboards, visualize log trends, and perform ad-hoc searches over your indices.
Deployment Strategy
In Kubernetes, you typically deploy these using Helm charts. To ensure your logs flow correctly, you must align your log forwarder (configured in previous lessons) to output to your Logstash instance.
Here is a conceptual look at how you might route logs from a producer to the ELK backend:
# Simplified concept: Log forwarder output configuration
[OUTPUT]
Name forward
Match *
Host logstash-service.logging.svc.cluster.local
Port 5044
Once the logs reach Logstash, they are parsed and stored in Elasticsearch indices. You then point Kibana to these indices. Because Elasticsearch is resource-intensive, ensure you define appropriate resources requests and limits in your deployment manifests to prevent the logging infrastructure from starving your application workloads.
By centralizing your logs, you move from "reactive debugging"—manually inspecting pods—to "proactive observability," where you can identify patterns, search for error strings across the entire cluster, and build alerts based on specific log content.
Key Takeaway: The ELK stack provides the heavy-duty indexing and visualization power needed to turn scattered container logs into actionable intelligence through a centralized, searchable repository.