Scraping Kubernetes API Server Metrics

In this lesson, you will learn how to configure Prometheus service discovery to automatically detect and scrape health and performance metrics directly from the Kubernetes API server.

Understanding the Control Plane Source

To maintain a healthy cluster, you need visibility into the Kubernetes control plane—specifically the API server, which acts as the gateway for all cluster operations. Unlike infrastructure metrics captured by a Node Exporter, API server metrics provide insight into request latency, throughput, and authentication failures.

Prometheus gathers these metrics by targeting the kubernetes-apiservers service. Because the API server is a critical, highly available component, we rely on Kubernetes Service Discovery to find the endpoints dynamically rather than hardcoding IP addresses.

Configuring the Scrape Job

In a standard Prometheus setup, you define a "scrape job" that instructs Prometheus to look for the API server endpoint. Here is a simplified configuration snippet for a prometheus.yaml file:

- job_name: 'kubernetes-apiservers'
  kubernetes_sd_configs:
  - role: endpoints
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

Key Considerations

1. Service Discovery: By using kubernetes_sd_configs, Prometheus queries the Kubernetes API itself to find where the kubernetes service (the API server) is running.
2. Security: Since the API server exposes sensitive data, the connection must be encrypted via HTTPS. You must provide the service account’s ca.crt and token so that Prometheus has the proper authorization to scrape the /metrics endpoint.
3. Filtering: The relabel_configs block ensures we only scrape the specific endpoints associated with the API server, preventing the inclusion of irrelevant cluster services in our monitoring dashboard.

Once configured, Prometheus will automatically track these metrics, allowing you to create alerts for high request latency or spikes in 500-series server errors, ensuring your control plane remains performant and stable.

Key Takeaway: Monitoring the API server requires secure service discovery that leverages the cluster's internal API to dynamically identify and authenticate against the control plane's metrics endpoint.