Log Sampling and Filtering
What you'll learn: How to intelligently reduce log volume at collection time without losing critical diagnostic information.
The Volume vs Value Problem
As systems scale, logging everything becomes prohibitively expensive. Storage costs explode, indexing slows down, and finding useful information becomes like searching for a needle in a haystack. Log sampling and filtering let you keep what matters while discarding noise.
Sampling: Statistical Reduction
Sampling means keeping only a fraction of similar logs. Think of it like polling — you don't need to count every vote to understand election results.
Common strategies:
- Random sampling: Keep 1% of all INFO-level logs, 100% of ERROR
- Rate limiting: Allow max 10 "database connected" messages per minute
- Hash-based: Sample based on request ID hash to keep all logs for some requests (preserves traceability)
The key: sample similar logs heavily, but preserve rare or high-severity events.
Filtering: Rule-Based Exclusion
Filtering drops logs matching specific criteria before they're shipped.
Examples:
- Drop health check logs (already tracked by metrics)
- Exclude debug logs in production
- Remove logs from internal monitoring services
- Filter by source, log level, or message pattern
When to Apply
Apply sampling/filtering at the agent or application level before logs hit your central system. This reduces network traffic, indexing load, and storage costs immediately.
Balancing Act
Over-aggressive filtering loses critical debugging context. Under-sampling leaves you drowning in data. The solution: tune based on actual incident investigations — what logs did you actually need last time?
Key Takeaway: Log sampling and filtering reduce volume and cost by keeping statistical samples of high-frequency logs and filtering out known noise, while always preserving high-severity events and maintaining enough context for debugging real incidents.