Logging Infrastructure Patterns

In this lesson, you will learn how to design a robust, centralized logging architecture that ensures your scripts provide a reliable audit trail for production environments.

The Problem with Scattered Logs

When you run complex scripts across multiple processes or containers, keeping logs only on the local machine makes debugging impossible. If a process crashes or a container is destroyed, its local log files vanish. To achieve "production-grade auditability," we need to decouple log generation from log storage.

The Centralized Pattern: Standard Streams

The most reliable way to design a logging infrastructure is to treat your script’s stdout and stderr as the primary transport mechanism. By enforcing Strict Mode (using set -euo pipefail), you ensure that any unexpected failure is immediately captured as an event.

Instead of writing directly to a file inside your script, redirect your process output to a centralized stream aggregator (like a sidecar process or a systemd journal) using Process Substitution.

Here is a simple pattern for a production-ready logging wrapper:

#!/usr/bin/env bash
set -euo pipefail

# Centralized logging function
log() {
    local level=$1
    shift
    # Timestamp, Level, and Process ID for auditability
    echo "[$(date +'%Y-%m-%dT%H:%M:%S%z')] [$level] [PID:$$] $*" >&2
}

# Redirect all script output to a central pipe
exec 1> >(logger -t APP_NAME -p user.info)
exec 2> >(logger -t APP_NAME -p user.err)

log "INFO" "Starting the backup sequence."
# Your logic here...

Why This Works

1. Atomicity: By using exec, you ensure that every command in your script automatically routes its output to the logging facility.
2. Infrastructure Agnostic: The logger command communicates with syslog or journald. This means your script doesn't care where the logs go—they might be sent to a remote server, a database, or a cloud dashboard depending on the host's configuration.
3. Auditability: By including the PID ($$) and timestamps in your log function, you can reconstruct the sequence of events across multiple concurrent processes or subshells, even if they are outputting logs at the same time.

Key Takeaway: Never write logs to local files; route your script's streams to a centralized system logger to ensure data persistence and unified observability.