In this lesson, you will learn how to design custom logging formatters to ensure your logs are structured perfectly for automated analysis tools.
When you send logs to a database or a monitoring service, human-readable text (like "The system crashed at noon") is often difficult to parse programmatically. To make your logs "machine-readable," you need a consistent structure—most commonly JSON—that separates your data points into distinct fields like timestamps, severity levels, and process IDs.
While the standard logging module provides a Formatter class, default outputs are often plain strings. To optimize for machines, we define a formatter that bundles our metadata into a structured dictionary and then serializes it using the json module, which we explored in earlier lessons.
Consider this custom formatter that organizes log data:
import logging
import json
import os
class JsonFormatter(logging.Formatter):
def format(self, record):
log_record = {
"timestamp": self.formatTime(record, self.datefmt),
"level": record.levelname,
"pid": os.getpid(),
"message": record.getMessage()
}
return json.dumps(log_record)
# Applying the formatter
logger = logging.getLogger("MachineLogger")
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)
logger.warning("Connection timeout detected")
When this runs, the output isn't just a string; it is a valid JSON object: {"timestamp": "2023-10-27 10:00:00", "level": "WARNING", "pid": 1234, "message": "Connection timeout detected"}.
By including the pid (process identifier), you allow downstream tools to correlate logs across multiple instances of your application. Including the timestamp in a standard format (like ISO 8601) ensures that your analysis software can sort events chronologically with ease. When you structure logs this way, you turn raw data into a reliable stream of information that any ingestion tool can index and query without complex regex patterns.
Key Takeaway: Formatting your logs as structured JSON with consistent metadata fields makes them immediately ready for automated processing and high-performance analysis.