The ELK Stack: Logstash
You'll learn how Logstash acts as a data processing pipeline that cleans, transforms, and enriches raw logs before they reach Elasticsearch.
What Logstash Does
Logstash is the "L" in ELK, serving as the middleman between your log sources and Elasticsearch. Think of it as a sophisticated assembly line worker that takes messy, unstructured logs and transforms them into clean, structured, searchable records.
Without Logstash, you might ship raw logs directly to Elasticsearch — but those logs often arrive in various formats, missing useful fields, or containing irrelevant noise. Logstash solves this by acting as a transformation layer.
The Three-Stage Pipeline
Logstash operates in three distinct stages:
1. Input
Receives logs from multiple sources: files, syslog, message queues, HTTP endpoints, or even databases.
2. Filter
The transformation engine. Logstash can:
- Parse unstructured text into structured fields (using patterns like Grok)
- Enrich logs with additional context (geolocation, user lookup)
- Transform field values (normalize timestamps, convert data types)
- Drop unwanted fields or entire log entries
3. Output
Sends processed logs to Elasticsearch (or other destinations like S3, monitoring systems, or alerting platforms).
Why This Matters
Raw log lines like "User 192.168.1.5 failed login" become structured records with separate fields: {user: null, ip: "192.168.1.5", action: "login", result: "failed"}. This structure makes logs searchable, filterable, and aggregatable in Elasticsearch.
Logstash also handles backpressure, buffering, and retry logic — protecting Elasticsearch from overload while ensuring no logs are lost during processing.
Key Takeaway: Logstash transforms chaotic raw logs into structured, enriched data through a configurable input-filter-output pipeline, making logs meaningful and searchable before they reach Elasticsearch.