Log Sampling and Filtering
You'll learn how to strategically reduce log volume by selecting which logs to collect and store, lowering infrastructure costs without losing critical observability.
The Challenge of Log Volume
In production Kubernetes clusters, applications and system components can generate millions of log entries per hour. Storing and processing every single log line becomes expensive and often unnecessary. Most logs are redundant debug messages or routine operation confirmations—valuable in development, but costly noise in production.
What Is Log Sampling and Filtering?
Sampling means collecting only a percentage of logs based on rules. Think of it like quality control in manufacturing: you don't inspect every widget, just a representative sample. You might sample 10% of successful HTTP requests but keep 100% of errors.
Filtering means dropping logs that match certain patterns before they're stored. Imagine a bouncer at a club—some logs just don't make the cut. You might filter out verbose debug messages, health check pings, or known chatty components.
Implementation Approaches
Both Fluentd and Fluent Bit (from your previous lessons) support filtering through configuration:
- Level-based filtering: Only forward ERROR and WARN logs, drop DEBUG and INFO
- Pattern matching: Exclude logs containing specific strings like "health check" or "heartbeat"
- Rate limiting: Allow maximum N logs per second from a pod
- Field-based rules: Sample based on HTTP status codes, user IDs, or custom labels
Practical Balance
The art is finding the sweet spot: keep enough logs for troubleshooting while cutting storage costs. A common strategy is filtering routine traffic while preserving all errors, warnings, and a small percentage of successful operations for baseline metrics.
Key Takeaway: Log sampling and filtering let you control observability costs by intelligently choosing which logs matter most, using rules configured in your logging agents to drop or sample before storage.