Prometheus Federation Basics
What You'll Learn
You'll learn how Prometheus federation creates hierarchical monitoring architectures to aggregate metrics from multiple Prometheus instances into a centralized view.
The Core Idea
Federation lets one Prometheus server scrape selected metrics from other Prometheus servers. Think of it like a corporate reporting structure: regional offices (edge Prometheus instances) collect detailed local metrics, while headquarters (central Prometheus) pulls summary reports from each region without needing every single detail.
Federation solves a key scaling challenge: a single Prometheus can't monitor thousands of services across multiple clusters efficiently. Instead, you deploy multiple Prometheus instances close to their targets, then use federation to pull aggregated metrics upward.
How Federation Works
The federated Prometheus scrapes the /federate endpoint of downstream Prometheus servers, retrieving only the metrics you specify via label selectors. This creates a hierarchical architecture:
- Bottom tier: Individual Prometheus instances scrape application metrics
- Middle tier (optional): Regional Prometheus instances aggregate metrics from their zone
- Top tier: Global Prometheus federates key metrics from all lower tiers
Typical Configuration Pattern
In your central Prometheus prometheus.yml:
scrape_configs:
- job_name: 'federate'
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="kubernetes-pods"}'
- '{__name__=~"job:.*"}' # Recording rules
static_configs:
- targets:
- 'prometheus-eu:9090'
- 'prometheus-us:9090'
The honor_labels: true setting preserves original labels from downstream instances, preventing conflicts.
When to Use Federation
Federation excels when monitoring multiple data centers, Kubernetes clusters, or geographic regions where network latency or isolation makes centralized scraping impractical. It's especially powerful combined with recording rules (covered earlier) to pre-aggregate data at lower tiers.
Key Takeaway: Prometheus federation creates hierarchical monitoring by letting upstream Prometheus instances selectively scrape aggregated metrics from downstream instances, enabling scalable multi-cluster observability without overwhelming a single server.