In this lesson, you will learn how the Linux kernel uses scheduling domains to logically organize CPU cores and manage task migration.
The Problem of Global Load Balancing
On a multi-core system, the kernel constantly tries to balance the workload so that no single core is overwhelmed while others sit idle. However, indiscriminately moving tasks across every core in the system can be inefficient. If two cores share a physical cache, it is much faster for them to share tasks than it is to move a task to a core on a completely different CPU socket (which incurs high latency due to memory access times).
Creating Scheduling Domains
To handle this, Linux organizes cores into a hierarchy called Scheduling Domains. These domains represent different levels of proximity between physical cores:
- SMT (Simultaneous Multithreading) Level: Cores sharing the same physical silicon.
- MC (Multi-Core) Level: Cores sharing an L3 cache.
- NUMA Level: Cores sharing a memory node.
The kernel treats these as nested circles. A "domain" defines the boundary within which the scheduler is allowed to migrate tasks. For instance, the scheduler might be very aggressive about moving tasks between cores that share an L2 cache because the performance cost is low. However, it will be much more conservative when considering moving a task across a NUMA boundary, as this requires expensive memory page migrations.
Controlling Migration
While the kernel manages this automatically, you can observe these domains through the /proc filesystem. If you look at /proc/sys/kernel/sched_domain/, you can see the configuration files for each domain level. For example, the flags file within a domain directory contains bitmasks that define behavior:
# Example: Viewing the migration settings for a specific domain level
cat /proc/sys/kernel/sched_domain/cpu0/domain0/flags
By adjusting these flags, you tell the scheduler how "far" it is permitted to push a process. If a domain’s BALANCE_NEWIDLE flag is enabled, the kernel will attempt to pull a task from another core the moment it becomes idle, ensuring that the domain remains fully utilized without needing to jump across the entire system.
Key Takeaway: Scheduling domains allow the kernel to perform load balancing efficiently by categorizing cores into hierarchical groups, prioritizing movement between "nearby" cores to minimize cache misses and memory latency.