In this lesson, you will learn how to leverage ServiceMonitors to enable Prometheus to automatically discover and scrape metrics from your applications without manual configuration changes.
The Problem: Static vs. Dynamic Discovery
When you first set up Prometheus, you might be tempted to manually list every single target in your configuration file. In a dynamic Kubernetes environment, this is unsustainable. As you scale your deployments up or down or roll out new microservices, manually updating your monitoring configuration is error-prone and slows down your CI/CD pipeline.
Automating with ServiceMonitors
If you are using the Prometheus Operator, the ServiceMonitor Custom Resource Definition (CRD) solves this by decoupling the monitoring configuration from the Prometheus server itself.
A ServiceMonitor uses Kubernetes labels to "find" the services you want to monitor. When you deploy a new service, you simply attach a specific label to it. The Prometheus Operator continuously watches for these ServiceMonitors and updates the target list in real-time.
Example: Connecting the dots
Imagine you have an application service. To monitor it, you create a ServiceMonitor resource:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-app-monitor
labels:
release: prometheus # Tells Prometheus to pick this up
spec:
selector:
matchLabels:
app: my-app # Targets services with this label
endpoints:
- port: web
path: /metrics
When this resource is applied, Prometheus automatically detects any service labeled app: my-app in the namespace. If you scale your my-app deployment to 10 replicas, the service object (with the matching label) ensures Prometheus discovers all 10 endpoints immediately, without you touching a single line of Prometheus configuration.
This shift allows developers to include their monitoring requirements directly in their application manifests. It promotes a "monitoring-as-code" approach, where the observability configuration travels alongside the deployment artifact, ensuring that new services are automatically scraped the moment they become healthy.
Key Takeaway: ServiceMonitors use label selectors to automate the discovery of metrics endpoints, eliminating manual Prometheus configuration updates and ensuring seamless observability for dynamic Kubernetes workloads.