Log Rotation and Retention Policies
What you'll learn: How to configure automated log rotation, compression, and retention policies that meet both compliance requirements and disk space constraints.
Why This Matters
Without proper rotation and retention, logs either fill your disk until systems crash or get deleted too quickly to support forensic investigations. A well-designed policy strikes the balance between operational stability and security/compliance needs.
The Core Challenge
Think of logs like a constantly running faucet filling a bucket—eventually it overflows. Log rotation is like having multiple buckets on a carousel: when one fills, you rotate to the next, archive the full one, and eventually discard the oldest.
Key Configuration Dimensions
Rotation Triggers:
- Size-based: Rotate when log reaches N megabytes (prevents runaway growth)
- Time-based: Daily, weekly, or monthly rotation (predictable schedules)
- Hybrid: Whichever condition hits first
Post-Rotation Actions:
- Compression: gzip or bzip2 older logs to save 80-95% disk space
- Numbering schemes:
app.log,app.log.1,app.log.2.gz - Dating schemes:
app-2024-01-15.log.gz(better for compliance audits)
Retention Periods:
- Hot storage: Recent logs (7-30 days) kept uncompressed for quick access
- Warm storage: Older logs (30-90 days) compressed but local
- Cold/archive: Long-term retention (1-7 years) on cheaper storage or offline
- Deletion: Automated purging after retention period expires
Compliance Alignment
Different regulations mandate different retention: PCI-DSS requires 90 days immediately accessible + 1 year archived; HIPAA typically 6 years; GDPR varies by jurisdiction. Your policy must satisfy the strictest applicable standard.
Key Takeaway: Effective log rotation and retention policies automate the lifecycle from active logging through compression, archival, and eventual deletion—balancing disk space, access speed, and compliance obligations without manual intervention.