Course contentsShow
← Cloud and Infrastructure
Lesson 1988 of 2,98942. Observability - MetricsFree lesson

Scrape Interval Optimization

Tuning scrape intervals and timeout settings to balance metric freshness with resource consumption.

Scrape Interval Optimization

You'll learn how to tune Prometheus scrape intervals and timeouts to balance metric freshness against system load and costs.

The Trade-Off Problem

Every Prometheus scrape creates work: network calls, CPU cycles, storage writes. Scraping every second gives you near-real-time visibility but floods your system with data points. Scraping every 5 minutes saves resources but means you might miss brief spikes or outages. Scrape interval optimization is the art of finding the sweet spot for each job.

How It Works

Prometheus scrapes targets at intervals defined in scrape_configs:

scrape_configs:
  - job_name: 'critical-api'
    scrape_interval: 15s
    scrape_timeout: 10s
  
  - job_name: 'batch-jobs'
    scrape_interval: 2m
    scrape_timeout: 30s

Interval controls how often to scrape. Timeout is the max time to wait for a response. Timeout must be shorter than interval, or scrapes will overlap and fail.

Decision Framework

  • Critical systems (APIs, payment services): 10–30s intervals for quick anomaly detection
  • Background jobs (cron tasks, batch workers): 1–5m intervals since metrics change slowly
  • Infrastructure metrics (node_exporter): 30s–1m is standard
  • High-cardinality targets: Longer intervals reduce storage pressure

Think of it like checking your email: vital messages need constant monitoring, newsletters can wait.

Avoid These Pitfalls

Setting timeout too close to interval causes flapping. Scraping too frequently wastes bandwidth when metrics rarely change. Mixing intervals complicates PromQL queries involving rate() calculations.

Key Takeaway:
Match scrape intervals to metric volatility and business criticality—fast-changing, critical services get short intervals (15–30s), while stable background systems can use longer periods (1–5m) to conserve resources.