Logging Architecture in Kubernetes
You'll learn: How Kubernetes organizes logging at three distinct levels to capture everything from individual containers to entire clusters.
The Three-Tier Logging Model
Kubernetes doesn't provide a single built-in logging solution. Instead, it defines a three-layer architecture that separates concerns based on scope and responsibility.
Think of it like a city's emergency response system: individual buildings have smoke detectors (application-level), neighborhoods have fire stations monitoring multiple buildings (node-level), and the city has a central dispatch coordinating everything (cluster-level).
Node-Level Logging
At the node level, each Kubernetes node handles logs from all containers running on it. The container runtime (like Docker or containerd) writes container stdout/stderr to local log files on the node's filesystem, typically under /var/log/containers/. The kubelet and container runtime manage log rotation to prevent disk exhaustion.
This is what kubectl logs queries—it reads these node-local files.
Cluster-Level Logging
Cluster-level logging aggregates logs from all nodes into a centralized system. Since Kubernetes doesn't provide this out-of-the-box, you deploy additional components like Fluentd, Fluent Bit, or Logstash to collect, process, and forward logs to storage backends (Elasticsearch, CloudWatch, etc.). These agents typically run as DaemonSets, ensuring one collector per node.
Application-Level Logging
Application-level logging refers to how your application code generates logs. You can write to stdout/stderr (recommended), use sidecar containers to transform or redirect logs, or have applications push directly to external systems. The first approach (stdout/stderr) integrates seamlessly with Kubernetes' native logging.
Key Takeaway: Kubernetes logging has three layers—node-level (local container logs), cluster-level (centralized aggregation you implement), and application-level (how your code produces logs)—each serving different visibility and operational needs.