Course contentsShow
← System Design
Lesson 1250 of 1,91931. Observability: TracingFree lesson

Trace Collector and Agent Patterns

Design sidecar vs daemon deployment patterns for trace collectors and understand batching and buffering strategies.

Trace Collector and Agent Patterns

What you'll learn: How to choose between sidecar and daemon deployment patterns for collecting traces, and why batching and buffering strategies matter for efficient trace transport.

The Deployment Pattern Decision

Once your application generates traces (via instrumentation you've learned), you need to get those traces to your storage backend (like Jaeger or Zipkin). You have two main architectural choices:

Sidecar Pattern

A sidecar collector runs as a separate container alongside each application instance in the same pod/host. Think of it like a personal assistant dedicated to each service instance.

Pros:

  • Isolated failure domain (one app's collector crash doesn't affect others)
  • Per-service configuration and sampling rules
  • Lower network latency (localhost communication)

Cons:

  • Higher resource overhead (N collectors for N service instances)
  • More complex deployment management

Daemon Pattern

A daemon collector runs as a single process per node/host, shared by all application instances on that machine. Like a neighborhood postal worker serving multiple houses.

Pros:

  • Efficient resource usage (one collector per host)
  • Centralized batching and buffering
  • Simpler deployment at scale

Cons:

  • Single point of failure per node
  • Potential performance bottleneck under high load

Batching and Buffering Strategies

Collectors don't send traces immediately—they batch spans together and buffer them in memory before transmission. This reduces network overhead dramatically.

  • Batching: Group 100-1000 spans into a single network request instead of sending individually
  • Buffering: Queue spans temporarily when the backend is slow or unreachable
  • Trade-off: Larger batches improve efficiency but increase memory usage and delay trace availability

Key Takeaway: Choose sidecar for isolation and per-service control, daemon for resource efficiency at scale; always configure batching/buffering to balance network efficiency against memory pressure and observability latency.