Log Aggregation and Retention Patterns

In this lesson, you will learn how to prevent your system drives from filling up by managing the lifecycle of logs stored in the /var/log hierarchy.

The Problem: Infinite Growth

In Linux, system services and applications write diagnostic information to files located in /var/log. Because these files are constantly appended to as processes run, a single log file could theoretically grow until it consumes all available disk space, potentially crashing your system. To prevent this, we use a process called log rotation.

Log Rotation Policies

Log rotation is a strategy where the system periodically renames, compresses, and eventually deletes old logs. Instead of one massive file that spans months, the system manages a collection of "generations."

Think of it like a filing cabinet:

1. Rotate: When a log reaches a certain size or age, the system moves syslog to syslog.1.
2. Compress: To save space, older files (like syslog.2.gz) are compressed.
3. Expire: Once a specific number of generations exist, the oldest file is deleted permanently.

Configuring Policies

On most modern Linux systems, this behavior is managed by the logrotate utility, which follows configuration files found in /etc/logrotate.d/. Here is an example of what a policy for a specific application log might look like:

/var/log/myapp.log {
    rotate 7
    daily
    compress
    missingok
}

• rotate 7: Keep only seven archived versions of the log.
• daily: Perform the rotation check every 24 hours.
• compress: Use gzip to shrink old logs.
• missingok: Do not throw an error if the log file is missing.

By defining these retention patterns, you ensure that /var/log stays within a predictable size, keeping the storage consumption balanced without requiring manual intervention. When a log is rotated, the system sends a signal to the service writing to the file so it knows to start writing to a fresh, new file, ensuring no data loss occurs during the switch.

Key Takeaway: Log rotation policies allow you to balance the need for historical diagnostic data against the reality of finite disk storage by automating the compression and deletion of aging log files.