Scaling Runners with Kubernetes Autoscalers

In this lesson, you will learn how to bridge the gap between CI/CD job demand and physical infrastructure by integrating runner pools with the Kubernetes Cluster Autoscaler.

When you manage ephemeral runners on Kubernetes, you face a common dilemma: keeping enough pods running to handle peak bursts while avoiding the waste of idle compute resources. Relying solely on Horizontal Pod Autoscaler (HPA) to scale your runner pods is only half the solution, as those pods often end up in a Pending state when the underlying node pool hits capacity.

To solve this, we leverage the Kubernetes Cluster Autoscaler in conjunction with your runner controller. The workflow is as follows:

1. Demand Spikes: Your CI/CD runner controller (like Actions Runner Controller or a custom K8s-native agent) detects a queue of pending jobs.
2. Pod Scheduling: The controller spins up new runner pods. If your existing nodes are full, the Kubernetes scheduler leaves these new pods in Pending due to insufficient resources.
3. Cluster Scaling: The Cluster Autoscaler detects these Pending pods. It calculates the necessary capacity, triggers your Cloud Provider (AWS, GCP, Azure) to provision a new VM, and joins it to your cluster as a new Node.
4. Execution: Once the new node joins, the Pending runner pods are automatically scheduled, execute the CI/CD job, and terminate.
5. Downscaling: Once the pods exit, the Cluster Autoscaler identifies the now-empty node and terminates it to save costs.

To ensure this functions correctly, you must set appropriate requests in your runner pod manifests. The Cluster Autoscaler relies on these requests to determine if a pod can fit on a node:

resources:
  requests:
    cpu: "1000m"
    memory: "2Gi"

By explicitly defining these requirements, you allow the Cluster Autoscaler to make informed decisions about when to provision infrastructure. This creates a "just-in-time" compute environment where your infrastructure footprint grows and shrinks in direct correlation with your pipeline activity, effectively eliminating manual capacity planning.

Key Takeaway: By pairing ephemeral runner pods with Cluster Autoscaler-defined resource requests, you transform your CI/CD infrastructure into a cost-efficient, self-scaling utility that dynamically reacts to pipeline load.