Course contentsShow
← Cloud and Infrastructure
Lesson 2021 of 2,98943. Observability - LoggingFree lesson

Logstash Architecture and Pipelines

Understanding Logstash input, filter, and output plugins for log processing.

Logstash Architecture and Pipelines

What you'll learn: How Logstash processes logs through its three-stage pipeline using input, filter, and output plugins.

Core Concept

Logstash is a data processing pipeline that acts like an assembly line for your logs. It follows a simple but powerful pattern: input → filter → output. Each stage transforms your log data, moving it from raw collection to enriched, structured information ready for storage and analysis.

Think of Logstash as a factory:

  • Input plugins are the receiving docks where raw materials (logs) arrive
  • Filter plugins are the processing stations that clean, parse, and enrich the data
  • Output plugins are the shipping departments that send finished products (structured logs) to their destinations

The Three-Stage Pipeline

Input Stage

Inputs collect data from various sources. Common plugins include file (reading log files), beats (receiving from Filebeat), http, tcp, syslog, and kafka. Each input plugin knows how to communicate with its specific source.

Filter Stage

Filters transform and enrich your logs. Key plugins:

  • grok: Parses unstructured text using patterns (e.g., extracting fields from Apache logs)
  • mutate: Modifies fields (rename, convert types, add/remove)
  • date: Parses timestamps into proper date fields
  • geoip: Adds geographic location data from IP addresses

Output Stage

Outputs send processed logs to destinations. Common plugins include elasticsearch (the most popular for ELK stack), file, kafka, http, and stdout (for debugging).

Logstash can run multiple pipelines simultaneously, each with its own configuration, allowing you to handle different log types independently while sharing infrastructure.

Key Takeaway: Logstash pipelines follow a three-stage pattern—input plugins collect logs, filter plugins parse and enrich them, and output plugins deliver them to storage or analysis systems, creating a flexible and powerful log processing engine.