Remote Write and Remote Read
What you'll learn: How to configure Prometheus to push metrics to external storage systems and query them back for long-term retention and distributed patterns.
The Core Problem
Prometheus stores time-series data locally with limited retention (typically days or weeks). At scale, you need long-term storage without exploding local disk usage, and you may want to centralize queries across multiple Prometheus instances.
What Are Remote Write and Remote Read?
Remote Write lets Prometheus continuously stream scraped metrics to external storage backends (like Thanos, Cortex, or cloud-native solutions). Think of it as a "live backup pipe"—every sample gets forwarded in near real-time.
Remote Read allows Prometheus to fetch older data from those external systems when executing queries. If your local Prometheus doesn't have historical data, it transparently reads from the remote endpoint.
Configuration Basics
In your prometheus.yml:
remote_write:
- url: "https://remote-storage.example.com/api/v1/write"
queue_config:
capacity: 10000
max_shards: 5
remote_read:
- url: "https://remote-storage.example.com/api/v1/read"
read_recent: true
- remote_write defines where to send data; queue settings control buffering and parallelism
- remote_read defines where to retrieve historical data;
read_recent: truequeries both local and remote
Why This Matters
This pattern decouples storage from collection, enabling you to retain years of metrics without overwhelming individual Prometheus servers. It also supports multi-cluster federation—centralize dashboards and alerts across geographically distributed Prometheus instances.
Key Takeaway:
Remote Write streams metrics to durable external storage for long-term retention, while Remote Read enables querying that historical data transparently from Prometheus.