Cloud Log Collection and Normalization
You'll learn how to aggregate logs from diverse cloud sources and transform them into a consistent format for effective security analysis.
The Challenge of Cloud Log Diversity
In multi-cloud and hybrid environments, logs arrive from AWS, Azure, GCP, SaaS applications, containers, and on-premises systems—each with different formats, timestamps, field names, and structures. CloudTrail logs look nothing like Azure Activity logs or application logs. Without normalization, correlating a security event across platforms becomes nearly impossible.
Think of it like receiving travel reports from colleagues who write dates differently (MM/DD vs DD/MM), use different currencies, and varying levels of detail. Before you can analyze spending patterns, you need everything in one standard format.
API-Based Collection
Modern cloud SIEM solutions use API-based log collectors that pull data directly from cloud services:
- AWS: CloudWatch Logs API, S3 bucket polling for stored logs
- Azure: Event Hubs, Log Analytics API
- GCP: Pub/Sub, Logging API
These collectors authenticate using service accounts or IAM roles, continuously fetch new log data, and handle pagination, rate limits, and retries.
Parsing and Normalization
Once collected, logs undergo parsing (extracting structured fields from raw text or JSON) and normalization (mapping diverse fields to a common schema):
- Time normalization: All timestamps converted to UTC in ISO 8601 format
- Field mapping:
aws.sourceIPAddressandazure.callerIpAddressboth map tosource_ip - Event taxonomy: Different service events map to common categories (authentication, network, data access)
- Enrichment: Adding context like threat intelligence, geolocation, or asset inventory data
The result is a unified data lake where security analysts can query source_ip = "10.0.1.5" and find events from any cloud platform or service.
Key Takeaway:
Cloud log collection aggregates diverse data sources via APIs, while normalization transforms heterogeneous logs into a consistent schema, enabling unified security analysis across multi-cloud environments.