Fluentd Plugins and Output Configuration
You'll learn how to configure Fluentd's plugin ecosystem to route, transform, and deliver logs to multiple destinations simultaneously.
Understanding Fluentd's Plugin Architecture
Fluentd operates as a data routing pipeline where logs flow through three key plugin types: input (where logs come from), filter (how to transform them), and output (where they go). Think of it like a sorting facility where packages arrive, get relabeled or repackaged, then ship to different warehouses—all in one streamlined process.
Input Sources
Input plugins define how Fluentd receives logs. Common sources include:
in_tail- reads from log files (liketail -f)in_forward- receives logs from other Fluentd agentsin_http- accepts logs via HTTP endpointsin_syslog- collects syslog messages
Filter Transformations
Filters modify log records in-flight. Use them to parse unstructured text, add metadata, redact sensitive data, or enrich logs with context:
filter_parser- parse raw logs into structured fieldsfilter_record_transformer- add/modify fieldsfilter_grep- include/exclude logs based on patterns
Output Destinations
Outputs send processed logs to storage or analysis systems. Fluentd's power lies in multi-output routing—sending the same log stream to different destinations:
out_elasticsearch- index logs for searchingout_s3- archive to object storageout_kafka- stream to message queuesout_copy- duplicate streams to multiple outputs
Configuration Pattern
Match directives route logs based on tags, while labels organize complex pipelines without tag conflicts. Buffering and retry logic ensure reliable delivery even when destinations are temporarily unavailable.
Key Takeaway: Fluentd plugins create a flexible log pipeline where you configure inputs to collect, filters to transform, and outputs to route logs—enabling centralized logging with multiple destinations from a single collection agent.