Course contentsShow
← Security
Lesson 1879 of 3,05238. Cloud Security - Monitoring and IRFree lesson

Cloud Log Collection and Normalization

Aggregating logs from multi-cloud environments, API-based collection, parsing diverse formats, and normalizing for analysis.

Cloud Log Collection and Normalization

You'll learn how to aggregate logs from diverse cloud sources and transform them into a consistent format for effective security analysis.

The Challenge of Cloud Log Diversity

In multi-cloud and hybrid environments, logs arrive from AWS, Azure, GCP, SaaS applications, containers, and on-premises systems—each with different formats, timestamps, field names, and structures. CloudTrail logs look nothing like Azure Activity logs or application logs. Without normalization, correlating a security event across platforms becomes nearly impossible.

Think of it like receiving travel reports from colleagues who write dates differently (MM/DD vs DD/MM), use different currencies, and varying levels of detail. Before you can analyze spending patterns, you need everything in one standard format.

API-Based Collection

Modern cloud SIEM solutions use API-based log collectors that pull data directly from cloud services:

  • AWS: CloudWatch Logs API, S3 bucket polling for stored logs
  • Azure: Event Hubs, Log Analytics API
  • GCP: Pub/Sub, Logging API

These collectors authenticate using service accounts or IAM roles, continuously fetch new log data, and handle pagination, rate limits, and retries.

Parsing and Normalization

Once collected, logs undergo parsing (extracting structured fields from raw text or JSON) and normalization (mapping diverse fields to a common schema):

  • Time normalization: All timestamps converted to UTC in ISO 8601 format
  • Field mapping: aws.sourceIPAddress and azure.callerIpAddress both map to source_ip
  • Event taxonomy: Different service events map to common categories (authentication, network, data access)
  • Enrichment: Adding context like threat intelligence, geolocation, or asset inventory data

The result is a unified data lake where security analysts can query source_ip = "10.0.1.5" and find events from any cloud platform or service.

Key Takeaway:
Cloud log collection aggregates diverse data sources via APIs, while normalization transforms heterogeneous logs into a consistent schema, enabling unified security analysis across multi-cloud environments.