Course contentsShow
← System Design
Lesson 1151 of 1,91929. Observability: LoggingFree lesson

The ELK Stack: Logstash

Logstash as a data processing pipeline for parsing, transforming, and enriching logs before indexing.

The ELK Stack: Logstash

You'll learn how Logstash acts as a data processing pipeline that cleans, transforms, and enriches raw logs before they reach Elasticsearch.

What Logstash Does

Logstash is the "L" in ELK, serving as the middleman between your log sources and Elasticsearch. Think of it as a sophisticated assembly line worker that takes messy, unstructured logs and transforms them into clean, structured, searchable records.

Without Logstash, you might ship raw logs directly to Elasticsearch — but those logs often arrive in various formats, missing useful fields, or containing irrelevant noise. Logstash solves this by acting as a transformation layer.

The Three-Stage Pipeline

Logstash operates in three distinct stages:

1. Input

Receives logs from multiple sources: files, syslog, message queues, HTTP endpoints, or even databases.

2. Filter

The transformation engine. Logstash can:

  • Parse unstructured text into structured fields (using patterns like Grok)
  • Enrich logs with additional context (geolocation, user lookup)
  • Transform field values (normalize timestamps, convert data types)
  • Drop unwanted fields or entire log entries

3. Output

Sends processed logs to Elasticsearch (or other destinations like S3, monitoring systems, or alerting platforms).

Why This Matters

Raw log lines like "User 192.168.1.5 failed login" become structured records with separate fields: {user: null, ip: "192.168.1.5", action: "login", result: "failed"}. This structure makes logs searchable, filterable, and aggregatable in Elasticsearch.

Logstash also handles backpressure, buffering, and retry logic — protecting Elasticsearch from overload while ensuring no logs are lost during processing.

Key Takeaway: Logstash transforms chaotic raw logs into structured, enriched data through a configurable input-filter-output pipeline, making logs meaningful and searchable before they reach Elasticsearch.