Log Streaming vs Log Shipping
You'll learn: The difference between real-time log streaming and batched log shipping, and when to use each approach.
Two Ways to Move Logs
Once you've set up log collection agents (our previous lesson), you need to decide how those logs travel to your centralized logging system. There are two fundamental approaches: streaming and shipping.
Log Streaming: The Live Feed
Log streaming sends log data in real-time, as events occur. Think of it like a live news broadcast—the moment something happens, it's transmitted immediately to your centralized logging system.
- Events flow continuously
- Near-instant visibility into issues
- Higher network bandwidth usage
- More complex infrastructure
- Best for critical systems requiring immediate alerts
Log Shipping: The Batch Delivery
Log shipping collects logs locally and sends them in batches at regular intervals (every 5 minutes, hourly, etc.). It's like the postal service—collecting mail throughout the day and delivering it in scheduled rounds.
- Logs buffered locally before transmission
- Reduced network overhead
- Slight delay in visibility (acceptable for many use cases)
- Simpler to implement
- Better for high-volume, non-critical logs
Choosing Your Approach
Use streaming when you need immediate detection of errors, security incidents, or performance degradation. Use shipping for audit logs, debug logs, or systems where a few minutes' delay won't impact operations.
Many organizations use a hybrid: streaming for application logs at ERROR level, shipping for INFO and DEBUG levels.
Key Takeaway: Log streaming provides real-time visibility with higher costs and complexity, while log shipping batches data for efficiency—choose based on how quickly you need to see and react to events.