Monitoring Autoscaler Health and Scaling Events

You will use cluster metrics and logs to troubleshoot scaling bottlenecks and verify that the autoscalers are responding correctly to changing workload demands.

In this lesson, you will learn how to verify that your Horizontal Pod Autoscaler (HPA) and Cluster Autoscaler are functioning correctly by correlating metrics, logs, and cluster events.

Understanding the Feedback Loop

Autoscaling is a reactive process. To troubleshoot why your cluster isn't scaling as expected, you must observe the three-step feedback loop: Metric Collection, Controller Evaluation, and Action Execution. If the cluster fails to scale, the bottleneck usually lies in one of these phases.

Investigating Scaling Events

When the HPA decides to scale, it records the decision in the resource's events. You can inspect these using kubectl describe hpa <name>. Pay close attention to the Events section at the bottom of the output:

Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  2m    horizontal-pod-autoscaler  Scaled up replica set ...

If you see FailedGetResourceMetric or FailedComputeMetrics, your Metrics Server is likely struggling or the requests on your pods are misconfigured. If the HPA shows a target utilization value but the pods aren't being created, check for FailedScheduling events on your pods using kubectl get events --sort-by='.metadata.creationTimestamp'. This often indicates that the Cluster Autoscaler cannot add nodes because of cloud provider quotas or resource constraints.

Using Logs for Advanced Debugging

Sometimes events aren't enough. If the Cluster Autoscaler is acting strangely—such as refusing to scale down a node that appears idle—you should inspect the logs of the cluster-autoscaler deployment:

kubectl logs -n kube-system deployment/cluster-autoscaler

Look for lines marked as ScaleDown or FilterOut. These logs explain why a node is being held back from termination (e.g., "node has local storage" or "pod has a local disruptor"). By matching these logs with your HPA events, you can definitively prove if your bottleneck is a metric reporting error, a deployment constraint, or a cluster-level resource limit.

Key Takeaway: Troubleshooting autoscaling is a process of correlating HPA events for decision-making logic with Cluster Autoscaler logs for infrastructure-level scheduling constraints.