Analyzing Virtualization Overheads

In this lesson, you will learn how to isolate whether application latency originates from the guest operating system’s internal operations or from "stolen" time imposed by the hypervisor.

The Virtualization Tax

When running workloads in a virtualized environment, the guest OS is often unaware of the underlying physical CPU contention. What appears to be an efficient process inside the guest may actually be stalling because the hypervisor has preempted the virtual CPU (vCPU) to schedule another guest or perform host-level management tasks. We refer to this wait time as CPU Steal Time.

Distinguishing the Source

To identify if your performance bottleneck is hypervisor-induced, you must compare local scheduling latency against the hypervisor’s scheduling metrics.

1. The Guest Perspective: Within the guest, you can observe steal time using top or mpstat. If steal is high, the hypervisor is withholding physical CPU cycles.
2. The Observability Gap: If steal is low but performance remains sluggish, the issue is likely VMM (Virtual Machine Monitor) exit latency. This occurs when the guest performs a privileged instruction (like an I/O port access or a control register write) that triggers a "VM exit," forcing the CPU to switch context from the guest to the hypervisor to emulate the action.

You can trace these VM exits using a custom eBPF program to track the time spent in the hypervisor during these transitions:

// Simplified logic for tracing VM exit duration
SEC("kprobe/kvm_exit")
int BPF_KPROBE(trace_kvm_exit) {
    u64 pid = bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&start_times, &pid, &ts, BPF_ANY);
    return 0;
}

By measuring the duration between kvm_exit and the subsequent return to the guest, you can determine if the overhead is caused by complex emulation logic or simple hypervisor scheduling delays. If the time between these events is consistently high, you are dealing with architectural overhead; if the wait time to resume execution is high even without an exit, the host system is likely oversubscribed and inducing steal time.

Key Takeaway: High guest latency with low steal time points toward expensive VM-exit emulation, while high steal time confirms the hypervisor is starving the guest of physical CPU resources.