Course contentsShow
← Security
Lesson 1548 of 3,05231. Operating System Security - MalwareFree lesson

System Call Hooking

Techniques for intercepting and modifying system calls to hide processes, files, and network connections from detection.

System Call Hooking

What you'll learn: How malware intercepts and modifies the fundamental communication between applications and the operating system kernel to remain hidden.

The Communication Bridge

Every time a program wants to read a file, list processes, or open a network connection, it makes a system call—a request to the kernel for a privileged operation. Think of system calls as the only official doorway between user applications and the kernel's powerful capabilities.

System call hooking is when malware (particularly kernel-mode rootkits) places itself in the middle of this communication path, intercepting these requests and lying about what's really happening on the system.

How It Works

The kernel maintains a table—often called the system call table or SSDT (System Service Descriptor Table) on Windows—that maps each system call number to the kernel function that handles it. Legitimate programs call NtQuerySystemInformation, for example, and the kernel routes it through this table.

A rootkit performing system call hooking modifies entries in this table to point to its own malicious functions instead. When an antivirus tool tries to list running processes, the malware's hook intercepts the call, removes the malware's process from the results, and returns the "cleaned" list. The security tool never knows it was deceived.

Common Targets

Attackers typically hook system calls related to:

  • Process enumeration (hiding malicious processes)
  • File operations (concealing backdoor files)
  • Network connections (masking command-and-control traffic)
  • Registry access (hiding persistence keys)

The hook examines each request, filters out anything related to the attacker's presence, and passes sanitized results back to the caller. From the application's perspective, the malware simply doesn't exist.

Key Takeaway: System call hooking allows kernel-mode rootkits to intercept the fundamental interface between applications and the kernel, selectively filtering results to hide malicious processes, files, and network activity from security tools and administrators.