Scrape Interval Optimization
You'll learn how to tune Prometheus scrape intervals and timeouts to balance metric freshness against system load and costs.
The Trade-Off Problem
Every Prometheus scrape creates work: network calls, CPU cycles, storage writes. Scraping every second gives you near-real-time visibility but floods your system with data points. Scraping every 5 minutes saves resources but means you might miss brief spikes or outages. Scrape interval optimization is the art of finding the sweet spot for each job.
How It Works
Prometheus scrapes targets at intervals defined in scrape_configs:
scrape_configs:
- job_name: 'critical-api'
scrape_interval: 15s
scrape_timeout: 10s
- job_name: 'batch-jobs'
scrape_interval: 2m
scrape_timeout: 30s
Interval controls how often to scrape. Timeout is the max time to wait for a response. Timeout must be shorter than interval, or scrapes will overlap and fail.
Decision Framework
- Critical systems (APIs, payment services): 10–30s intervals for quick anomaly detection
- Background jobs (cron tasks, batch workers): 1–5m intervals since metrics change slowly
- Infrastructure metrics (node_exporter): 30s–1m is standard
- High-cardinality targets: Longer intervals reduce storage pressure
Think of it like checking your email: vital messages need constant monitoring, newsletters can wait.
Avoid These Pitfalls
Setting timeout too close to interval causes flapping. Scraping too frequently wastes bandwidth when metrics rarely change. Mixing intervals complicates PromQL queries involving rate() calculations.
Key Takeaway:
Match scrape intervals to metric volatility and business criticality—fast-changing, critical services get short intervals (15–30s), while stable background systems can use longer periods (1–5m) to conserve resources.