Trace Collector and Agent Patterns
What you'll learn: How to choose between sidecar and daemon deployment patterns for collecting traces, and why batching and buffering strategies matter for efficient trace transport.
The Deployment Pattern Decision
Once your application generates traces (via instrumentation you've learned), you need to get those traces to your storage backend (like Jaeger or Zipkin). You have two main architectural choices:
Sidecar Pattern
A sidecar collector runs as a separate container alongside each application instance in the same pod/host. Think of it like a personal assistant dedicated to each service instance.
Pros:
- Isolated failure domain (one app's collector crash doesn't affect others)
- Per-service configuration and sampling rules
- Lower network latency (localhost communication)
Cons:
- Higher resource overhead (N collectors for N service instances)
- More complex deployment management
Daemon Pattern
A daemon collector runs as a single process per node/host, shared by all application instances on that machine. Like a neighborhood postal worker serving multiple houses.
Pros:
- Efficient resource usage (one collector per host)
- Centralized batching and buffering
- Simpler deployment at scale
Cons:
- Single point of failure per node
- Potential performance bottleneck under high load
Batching and Buffering Strategies
Collectors don't send traces immediately—they batch spans together and buffer them in memory before transmission. This reduces network overhead dramatically.
- Batching: Group 100-1000 spans into a single network request instead of sending individually
- Buffering: Queue spans temporarily when the backend is slow or unreachable
- Trade-off: Larger batches improve efficiency but increase memory usage and delay trace availability
Key Takeaway: Choose sidecar for isolation and per-service control, daemon for resource efficiency at scale; always configure batching/buffering to balance network efficiency against memory pressure and observability latency.