Log Rotation and Storage
What you'll learn: How Kubernetes manages log file size and lifespan on nodes to prevent disk exhaustion.
The Problem Logs Solve (and Create)
Container logs are essential for troubleshooting, but they grow endlessly. Without management, logs can fill node disks, crash your cluster, and make applications fail. Kubernetes needs a strategy to keep logs useful without letting them consume all available storage.
How Log Rotation Works
Think of log rotation like a rolling notebook: when one page fills up, you flip to a new page, and eventually discard the oldest pages to make room.
Kubernetes uses log rotation policies at the node level (typically managed by the container runtime like containerd or CRI-O). These policies automatically:
- Split logs into chunks when they reach a size limit (e.g., 10MB)
- Keep only recent files (e.g., last 5 rotated files)
- Delete old rotations to free disk space
Key Configuration Points
On most nodes, log rotation is controlled by:
- Max log file size: How large a single log file grows before rotation (e.g.,
10Mi) - Max log files: How many rotated copies to retain (e.g.,
5) - Total retention: Combined policy might keep ~50MB per container
These settings are typically configured in the container runtime's configuration file (like /etc/containerd/config.toml), not in Kubernetes manifests.
Storage Location
Logs are stored on the node's filesystem, usually under /var/log/pods/ and /var/log/containers/. This means each node manages its own disk space independently.
Key Takeaway: Log rotation prevents disk exhaustion by automatically limiting log file sizes and counts on each node, ensuring your cluster remains healthy while preserving recent diagnostic data.