Course contentsShow
← System Design
Lesson 1157 of 1,91929. Observability: LoggingFree lesson

Log Sampling and Filtering

When and how to sample or filter logs at collection time to reduce volume while preserving signal.

Log Sampling and Filtering

What you'll learn: How to intelligently reduce log volume at collection time without losing critical diagnostic information.

The Volume vs Value Problem

As systems scale, logging everything becomes prohibitively expensive. Storage costs explode, indexing slows down, and finding useful information becomes like searching for a needle in a haystack. Log sampling and filtering let you keep what matters while discarding noise.

Sampling: Statistical Reduction

Sampling means keeping only a fraction of similar logs. Think of it like polling — you don't need to count every vote to understand election results.

Common strategies:

  • Random sampling: Keep 1% of all INFO-level logs, 100% of ERROR
  • Rate limiting: Allow max 10 "database connected" messages per minute
  • Hash-based: Sample based on request ID hash to keep all logs for some requests (preserves traceability)

The key: sample similar logs heavily, but preserve rare or high-severity events.

Filtering: Rule-Based Exclusion

Filtering drops logs matching specific criteria before they're shipped.

Examples:

  • Drop health check logs (already tracked by metrics)
  • Exclude debug logs in production
  • Remove logs from internal monitoring services
  • Filter by source, log level, or message pattern

When to Apply

Apply sampling/filtering at the agent or application level before logs hit your central system. This reduces network traffic, indexing load, and storage costs immediately.

Balancing Act

Over-aggressive filtering loses critical debugging context. Under-sampling leaves you drowning in data. The solution: tune based on actual incident investigations — what logs did you actually need last time?

Key Takeaway: Log sampling and filtering reduce volume and cost by keeping statistical samples of high-frequency logs and filtering out known noise, while always preserving high-severity events and maintaining enough context for debugging real incidents.