Logstash Architecture and Pipelines
What you'll learn: How Logstash processes logs through its three-stage pipeline using input, filter, and output plugins.
Core Concept
Logstash is a data processing pipeline that acts like an assembly line for your logs. It follows a simple but powerful pattern: input → filter → output. Each stage transforms your log data, moving it from raw collection to enriched, structured information ready for storage and analysis.
Think of Logstash as a factory:
- Input plugins are the receiving docks where raw materials (logs) arrive
- Filter plugins are the processing stations that clean, parse, and enrich the data
- Output plugins are the shipping departments that send finished products (structured logs) to their destinations
The Three-Stage Pipeline
Input Stage
Inputs collect data from various sources. Common plugins include file (reading log files), beats (receiving from Filebeat), http, tcp, syslog, and kafka. Each input plugin knows how to communicate with its specific source.
Filter Stage
Filters transform and enrich your logs. Key plugins:
grok: Parses unstructured text using patterns (e.g., extracting fields from Apache logs)mutate: Modifies fields (rename, convert types, add/remove)date: Parses timestamps into proper date fieldsgeoip: Adds geographic location data from IP addresses
Output Stage
Outputs send processed logs to destinations. Common plugins include elasticsearch (the most popular for ELK stack), file, kafka, http, and stdout (for debugging).
Logstash can run multiple pipelines simultaneously, each with its own configuration, allowing you to handle different log types independently while sharing infrastructure.
Key Takeaway: Logstash pipelines follow a three-stage pattern—input plugins collect logs, filter plugins parse and enrich them, and output plugins deliver them to storage or analysis systems, creating a flexible and powerful log processing engine.