Course contentsShow
← System Design
Lesson 1206 of 1,91930. Observability: MetricsFree lesson

Metrics Federation and Long-Term Storage

Understand federated Prometheus setups, Thanos, and Cortex for aggregating metrics across clusters and long-term retention.

Metrics Federation and Long-Term Storage

What you'll learn: How to aggregate metrics from multiple Prometheus instances and retain them beyond Prometheus's typical retention limits.

The Challenge

A single Prometheus instance works beautifully for one cluster, but modern systems span multiple regions, clusters, or data centers. Each runs its own Prometheus, creating silos of metrics. You need to query across all of them and store historical data longer than Prometheus's default 15-day retention without overwhelming disk capacity.

Federated Prometheus

Federation lets one Prometheus scrape metrics from other Prometheus servers. Think of it like a manager collecting summary reports from team leads. You configure a central Prometheus to pull specific metrics from regional instances using the /federate endpoint:

# Central Prometheus scraping from edge instances
scrape_configs:
  - job_name: 'federated-clusters'
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="api"}'
    static_configs:
      - targets:
        - 'prometheus-us-east:9090'
        - 'prometheus-eu-west:9090'

This works for small setups but becomes brittle at scale.

Thanos: The Distributed Solution

Thanos extends Prometheus by uploading data blocks to cheap object storage (S3, GCS). It provides:

  • Global query view across all Prometheus instances
  • Long-term storage (years, not days)
  • Downsampling for efficient historical queries

Components like Thanos Sidecar, Store Gateway, and Querier work together to create a seamless distributed metrics system.

Cortex: Multi-Tenant Alternative

Cortex offers similar capabilities with a focus on multi-tenancy. It ingests metrics via remote write, stores them in object storage, and provides horizontally scalable querying. Think SaaS-grade metrics infrastructure.

Key Takeaway: When you outgrow single-Prometheus setups, federated architectures provide basic aggregation, while Thanos and Cortex deliver production-grade solutions for cross-cluster querying and unlimited retention using object storage.