SIEM Architecture: Collectors, Aggregators, and Storage
You'll learn how a SIEM physically gathers, normalizes, and stores security events across your entire environment.
The Data Pipeline Challenge
A SIEM must ingest logs from firewalls, servers, endpoints, cloud services, and dozens of other sources—each speaking a different "language." The architecture handles this through three core layers working in concert.
1. Collectors (Data Ingestion)
Collectors are distributed agents or forwarders deployed near data sources. They pull or receive logs via syslog, APIs, file monitoring, or agent-based collection. Think of collectors as regional post offices gathering mail from local neighborhoods—they understand local formats and ship everything upstream.
Common collector types:
- Agents: Lightweight software on endpoints (e.g., Windows Event Collector)
- Network listeners: Syslog servers receiving UDP/TCP streams
- API pollers: Query cloud services for security events
2. Aggregators (Normalization)
Aggregators receive raw logs from multiple collectors and normalize them into a common schema. A Cisco firewall log and a Linux authentication log look completely different—aggregators translate both into standardized fields like source_ip, event_type, timestamp, and severity.
This normalization enables correlation: detecting that the same user failed authentication on three systems requires those events to speak the same language.
3. Storage (Retention and Analysis)
Storage layers must handle massive write volumes (potentially millions of events/second) while supporting fast queries. Modern SIEMs use:
- Hot storage: Fast SSD/database for recent events (7-30 days)
- Warm storage: Compressed data for medium-term access
- Cold storage: Archived logs for compliance (1+ years), often in object storage
The architecture balances performance, cost, and retention requirements.
Key Takeaway:
SIEM architecture uses distributed collectors to gather diverse logs, aggregators to normalize them into queryable formats, and tiered storage to retain massive volumes efficiently—creating a unified security data lake from organizational chaos.