In this lesson, you will learn how to configure your log pipeline to direct specific security events to secure, long-term storage while streaming standard application logs to real-time analysis platforms.
The Strategy of Split-Stream Logging
In a complex Kubernetes environment, not all logs are created equal. While application logs (like stdout messages) are vital for debugging, security-critical logs (such as authentication failures or unauthorized access attempts) often require stricter retention policies, compliance auditing, and different storage backends.
To achieve this, we use the filtering and routing capabilities of our log forwarder, such as Fluent Bit. The core idea is to categorize logs based on metadata—typically labels or regex patterns—and then map those categories to specific output plugins.
Implementing Log Routing
When a log record enters the pipeline, it is tagged. By using a "Match" condition in your configuration, you can direct traffic based on these tags. Imagine you have a security agent or a specific application logging to stderr. You can isolate these logs and route them to an S3 bucket or a secure WORM (Write Once, Read Many) storage for compliance, while letting the rest flow into your real-time observability tool like Elasticsearch.
Here is how that routing logic looks conceptually in a configuration file:
[MATCH]
Name rewrite_tag
Match kube.*
Rule $log_level CRITICAL security.alert true
[OUTPUT]
Name s3
Match security.alert
Bucket audit-logs-long-term
[OUTPUT]
Name es
Match kube.*
Host elasticsearch-logging
In this example, the rewrite_tag plugin acts as the traffic controller. It inspects the log_level field. If it detects a "CRITICAL" entry, it re-tags it as security.alert, ensuring the subsequent OUTPUT section can send it to your long-term secure storage, completely separate from the standard flow going to Elasticsearch. This separation ensures that your primary analysis tool isn't cluttered with heavy compliance data, while simultaneously ensuring your security team has a reliable audit trail that meets regulatory standards.
Key Takeaway: By using tag-based routing and filtering in your logging agent, you can effectively balance the need for high-speed observability with the strict requirements of long-term security compliance.