System call entry and exit paths

In this lesson, you will learn how the Linux kernel manages the transition between user-space application code and kernel-mode execution during a system call.

The Boundary Crossing

When a process executes a system call (like read() or write()), it cannot simply jump into the kernel’s memory; the CPU must transition from a low-privilege "User Mode" (Ring 3) to a high-privilege "Kernel Mode" (Ring 0). This transition is not a function call, but a hardware-assisted event.

The Entry Path

On modern x86_64 architectures, the process triggers this transition using the syscall instruction. Before executing this, the application places the system call number into the rax register and arguments into designated registers (rdi, rsi, rdx, etc.).

Once the syscall instruction executes, the hardware performs several critical steps:

1. Privilege Elevation: The CPU switches the privilege level to Ring 0.
2. Instruction Pointer Hijacking: The CPU jumps to a specific memory address predefined in the LSTAR (Long System Target Address Register) Model Specific Register.
3. Context Switch: This entry point, often called entry_SYSCALL_64, saves the user-space stack pointer and register state onto the kernel stack. It then switches the stack pointer from the user stack to the kernel stack, ensuring the kernel has a clean workspace.

The Exit Path

After the kernel completes the requested task, it must return control to the user. This is the "Exit Path." The kernel ensures the return value of the system call is placed in rax and then executes the sysret instruction. This instruction performs the reverse of the entry path: it restores the user-space stack pointer, reverts the privilege level back to Ring 3, and jumps back to the instruction immediately following the original syscall in your application code.

; Simplified concept of the entry/exit flow
mov rax, 1          ; syscall: sys_write
mov rdi, 1          ; fd: stdout
syscall             ; CPU jumps to kernel entry path
; -- Kernel performs work --
; -- Kernel restores state and executes sysret --
; Execution resumes here in User Mode

Key Takeaway: The system call entry and exit path acts as a secure, hardware-mediated gateway that performs essential bookkeeping—saving registers, switching stacks, and elevating privileges—to ensure the kernel can safely execute operations on behalf of user-space processes.