Course contentsShow
← Python
Lesson 773 of 1,06012. Testing and DebuggingFree lesson

Configuring Log Rotation and Structured Output

Implement log handlers to manage file sizes and format logs as JSON for integration with centralized log aggregation and analysis platforms.

In this lesson, you will learn how to prevent disk space exhaustion by rotating log files and how to format logs as JSON for seamless ingestion into modern log aggregation platforms.

Managing File Growth with Rotation

When a production application runs indefinitely, its log file can grow until it consumes all available disk space. To prevent this, we use the RotatingFileHandler from the logging.handlers module. This handler monitors the size of your log file and automatically "rolls over" to a new file once a threshold is reached, keeping only a specified number of backup files.

import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger("app_logger")
# Rotate at 5MB, keep 3 historical backups
handler = RotatingFileHandler("app.log", maxBytes=5*1024*1024, backupCount=3)
logger.addHandler(handler)

Structured Logging with JSON

Traditional plain-text logs are difficult for tools like ELK or Datadog to parse. By formatting logs as JSON, you turn unstructured strings into key-value pairs that are queryable and searchable.

While you can write a custom formatter, the standard approach involves defining a logging.Formatter that outputs a JSON-serialized dictionary. This allows you to include dynamic metadata—like user IDs, request IDs, or severity levels—consistently across all logs.

import json
import logging

class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_record = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "message": record.getMessage(),
        }
        return json.dumps(log_record)

handler.setFormatter(JsonFormatter())

By combining these techniques, your application moves from producing "human-only" text files to emitting production-ready data streams. The RotatingFileHandler ensures your infrastructure remains stable under high load, while the JsonFormatter transforms your log output into a structured asset that external analysis tools can ingest, filter, and monitor with precision.

Key Takeaway: Use RotatingFileHandler to cap log storage consumption and implement a custom JsonFormatter to transform raw logs into machine-readable data for centralized analysis.