Course contentsShow
← System Design
Lesson 1145 of 1,91929. Observability: LoggingFree lesson

Sensitive Data in Structured Logs

Identifying and scrubbing PII, passwords, and tokens from structured logs before storage.

Sensitive Data in Structured Logs

What You'll Learn

You'll learn how to identify and remove sensitive information from structured logs before they're stored, preventing security breaches and compliance violations.

The Problem with Structured Logging and Secrets

Structured logs make data queryable and parsable—but that's dangerous when they contain passwords, API keys, credit card numbers, or personally identifiable information (PII). Because structured logs often serialize entire objects or request payloads as JSON, they can accidentally expose secrets that would be harder to extract from plain-text logs.

Think of it like leaving your diary in a filing cabinet versus scattering individual pages around: structured logs make sensitive data easier to find and exploit.

What Must Be Scrubbed

Common sensitive fields include:

  • Credentials: passwords, tokens, API keys, session IDs
  • PII: emails, phone numbers, addresses, SSNs
  • Financial data: credit card numbers, account balances
  • Health information: medical records, diagnoses

Scrubbing Strategies

Field-level redaction: Before logging, replace sensitive values with placeholders like [REDACTED] or hash them irreversibly.

Allowlisting: Define which fields are safe to log; drop everything else by default.

Pattern matching: Use regex or detection libraries to find and mask data that looks like credit cards, tokens, or emails.

Log sanitization pipelines: Process logs through a scrubbing layer before they reach storage systems.

Example approach (conceptual):

Before logging:
{"user": "alice", "password": "secret123", "email": "alice@example.com"}

After scrubbing:
{"user": "alice", "password": "[REDACTED]", "email": "a***@example.com"}

Key Governance

Never log raw secrets. Always scrub before serialization. Consider automated scanning tools that detect sensitive patterns in logs post-production to catch leaks early.

Key Takeaway: Structured logs amplify the risk of exposing sensitive data—always sanitize PII, credentials, and tokens before logging to prevent breaches and ensure compliance.