Course contentsShow
← System Design
Lesson 1167 of 1,91929. Observability: LoggingFree lesson

Avoid Log Explosion

Guard against unbounded loops or errors that generate millions of identical logs, overwhelming systems.

Avoid Log Explosion

What you'll learn: How to prevent runaway logging that can overwhelm storage, pipelines, and incident response.

The Danger of Unbounded Logging

Log explosion occurs when your system generates an uncontrolled flood of log entries—often millions of nearly identical messages in seconds. This typically happens in tight loops, cascading error conditions, or when a single failing operation retries rapidly without backoff.

Imagine a smoke detector that beeps once per millisecond instead of once per minute when triggered. You can't hear yourself think, let alone fix the fire. Similarly, log explosion drowns out important signals, saturates disk I/O, overwhelms your log aggregation pipeline, and can even cause outages in your logging infrastructure itself.

Common Explosion Scenarios

Tight loops logging per iteration:

for item in millions_of_items:
    logger.info(f"Processing {item}")  # Explosion!

Error conditions that log on every failure:

# Database down, retries 1000 times/sec
for attempt in range(1000):
    try:
        db.connect()
    except Exception as e:
        logger.error(f"DB connection failed: {e}")  # Repeated identical logs

Prevention Strategies

  1. Rate limiting: Log the first N occurrences, then summarize
  2. Sampling: Log only 1% of routine operations at high volume
  3. Aggregation: Count errors in memory, log periodically: "Failed 5,000 times in last minute"
  4. Circuit breaking: Stop logging after threshold, emit one "suppressed X logs" message
  5. Appropriate log levels: Use debug for verbose loops, not info or error

Key Takeaway: Guard against log explosion with rate limiting, sampling, and thoughtful placement—uncontrolled logging can be as dangerous as the failures it's meant to capture.