Course contentsShow
← Cloud and Infrastructure
Lesson 2026 of 2,98943. Observability - LoggingFree lesson

Fluentd Configuration and Parsing

Configuring Fluentd inputs, parsers, filters, and outputs for container and application logs.

Fluentd Configuration and Parsing

What You'll Learn

You'll learn how to configure Fluentd's pipeline components—inputs, parsers, filters, and outputs—to collect, transform, and route container and application logs effectively.

The Fluentd Configuration Pipeline

Think of Fluentd configuration like a factory assembly line: logs enter through inputs (loading dock), get transformed by parsers and filters (processing stations), then exit through outputs (shipping department).

The Four Core Directives

Source (Input): Where logs enter Fluentd. Common sources include:

  • tail for file-based logs
  • forward for network-based logs from other Fluentd instances
  • http for HTTP endpoint ingestion
  • Container runtime logs via mounted volumes

Parser: Converts unstructured log text into structured JSON. Fluentd includes built-in parsers for common formats (json, nginx, apache2, syslog) or you can define custom regex patterns.

Filter: Modifies, enriches, or drops log records mid-pipeline. You can add Kubernetes metadata, redact sensitive fields, or route based on content.

Match (Output): Where logs go next—Elasticsearch, S3, another Fluentd, or multiple destinations simultaneously.

Configuration Structure

Fluentd uses a declarative config file with XML-like tags:

<source>
  @type tail
  path /var/log/containers/*.log
  tag kubernetes.*
</source>

<filter kubernetes.**>
  @type parser
  key_name log
  <parse>
    @type json
  </parse>
</filter>

<match **>
  @type elasticsearch
  host elasticsearch.logging
</match>

Each section connects via tags (labels like kubernetes.*) that route data through the pipeline. Wildcards enable flexible routing patterns.

Key Takeaway: Fluentd configuration orchestrates a four-stage pipeline—source defines where logs come from, parser structures them, filter transforms them, and match sends them to destinations—all connected through tag-based routing.