Fluentd Configuration and Parsing
What You'll Learn
You'll learn how to configure Fluentd's pipeline components—inputs, parsers, filters, and outputs—to collect, transform, and route container and application logs effectively.
The Fluentd Configuration Pipeline
Think of Fluentd configuration like a factory assembly line: logs enter through inputs (loading dock), get transformed by parsers and filters (processing stations), then exit through outputs (shipping department).
The Four Core Directives
Source (Input): Where logs enter Fluentd. Common sources include:
tailfor file-based logsforwardfor network-based logs from other Fluentd instanceshttpfor HTTP endpoint ingestion- Container runtime logs via mounted volumes
Parser: Converts unstructured log text into structured JSON. Fluentd includes built-in parsers for common formats (json, nginx, apache2, syslog) or you can define custom regex patterns.
Filter: Modifies, enriches, or drops log records mid-pipeline. You can add Kubernetes metadata, redact sensitive fields, or route based on content.
Match (Output): Where logs go next—Elasticsearch, S3, another Fluentd, or multiple destinations simultaneously.
Configuration Structure
Fluentd uses a declarative config file with XML-like tags:
<source>
@type tail
path /var/log/containers/*.log
tag kubernetes.*
</source>
<filter kubernetes.**>
@type parser
key_name log
<parse>
@type json
</parse>
</filter>
<match **>
@type elasticsearch
host elasticsearch.logging
</match>
Each section connects via tags (labels like kubernetes.*) that route data through the pipeline. Wildcards enable flexible routing patterns.
Key Takeaway: Fluentd configuration orchestrates a four-stage pipeline—source defines where logs come from, parser structures them, filter transforms them, and match sends them to destinations—all connected through tag-based routing.