In this lesson, you will learn how high container density impacts CPU cache performance and disrupts processor branch prediction, leading to non-obvious performance degradation.
When you run multiple, high-throughput containers on a single host, they compete for the same physical CPU resources. While cgroups isolate CPU cycles, they cannot prevent cache thrashing. Each container’s workload has its own memory access pattern; when a CPU core switches context between processes in different containers, the new process frequently finds that the L1/L2 caches are already "polluted" with data from the previous container.
The Impact of Cache Misses
Every time the CPU looks for data in the cache and fails (a cache miss), it must fetch it from main memory—a process that is orders of magnitude slower than a cache hit. As you pack more containers onto a single socket, the working set of all containers combined often exceeds the physical cache size, causing a "storm" of cache misses that degrades execution speed across all containers.
Branch Prediction Interference
Modern CPUs use branch predictors to guess which way an if statement or loop will go, speculatively executing code to keep the pipeline full. This prediction relies on historical patterns. In a high-density environment, frequent context switching forces the CPU to constantly clear its Branch Target Buffer (BTB). Because different containers have different code paths, the predictor struggles to learn the pattern of any single container, leading to a high rate of branch mispredictions.
You can observe this behavior using perf on the host:
# Monitor cache misses and branch mispredictions for a containerized process
perf stat -e cache-references,cache-misses,branches,branch-misses -p <pid_of_container_app>
If you see the ratio of branch-misses to branches spike as you increase your container density, your processors are spending more time discarding speculative work than actually processing instructions. This isn't a bug in your code, but a physical limitation of the hardware trying to accommodate too many disparate execution contexts.
Key Takeaway: High container density causes cache pollution and branch predictor instability, turning CPU-bound applications into memory-latency-bound processes due to constant context switching.