Log Shipping vs Agent-Based Collection
What you'll learn: How to choose between different methods for moving logs from applications to centralized logging systems.
The Core Problem
Once your application generates structured logs, you need to get them to your centralized logging system (Elasticsearch, Splunk, etc.). There are three main approaches: sidecar agents, log forwarding daemons, and direct application shipping. Each has different tradeoffs for reliability, performance, and operational complexity.
The Three Collection Methods
Direct Application Shipping
Your application directly sends logs over the network to the logging backend. Think of it like mailing a letter yourself—fast and simple, but if the post office is down, letters pile up in your app's memory.
Tradeoffs: Simplest setup, but logging failures can crash your app or cause backpressure.
Sidecar Agent Pattern
A dedicated agent runs alongside your application (same pod/container) and reads its log output. The app writes to stdout/file, the agent forwards. Like having a personal assistant who collects your mail and handles delivery problems—your app stays decoupled from logging infrastructure.
Tradeoffs: Additional container overhead, but excellent isolation and resilience.
Log Forwarding Daemon (Node-Level Agent)
One agent per host/node collects logs from all applications. Similar to a building's central mailroom—efficient resource sharing, but harder to configure per-application needs.
Tradeoffs: Most resource-efficient, standardized across services, but less flexibility for specialized logging requirements.
Choosing the Right Approach
Use direct shipping for simple systems with reliable networks. Use sidecars in Kubernetes environments where isolation and per-service configuration matter. Use node-level daemons for large deployments prioritizing efficiency and standardization.
Key Takeaway: Log collection method choice depends on your infrastructure—sidecars provide isolation, daemons offer efficiency, and direct shipping trades reliability for simplicity.