Analyzing Concurrent Faults in Multithreaded Firmware

In this lesson, you will learn how to manually inspect Thread Control Blocks (TCBs) within a raw memory dump to identify the exact state of threads during a system hang or race condition.

When a multithreaded system stops responding, the RTOS kernel has typically trapped the failure state in a memory dump. While automated tools are helpful, the most reliable way to debug complex concurrency bugs—like deadlocks or race conditions—is to manually traverse the TCB structures preserved in that dump.

Anatomy of the TCB

Every RTOS thread is represented by a TCB structure that maintains its context: the stack pointer, current execution state (Running, Blocked, Ready), and priority. In a deadlock scenario, you are looking for a circular dependency:

1. Identify the list of active threads in your memory dump.
2. Examine the wait_object or blocked_on field within each blocked thread's TCB.
3. Map these pointers to the mutex or semaphore structures they are waiting for.

Identifying the Deadlock

If Thread A is blocked on Mutex 1, and your TCB inspection shows Mutex 1 is owned by Thread B, check Thread B's TCB. If you find Thread B is blocked on Mutex 2, which is owned by Thread A, you have successfully reconstructed the deadlock.

Consider this simplified snippet of a TCB structure:

typedef struct {
    uint32_t *stack_ptr;
    uint8_t   state;         // 0: Ready, 1: Blocked, 2: Running
    void     *blocked_on;    // Pointer to Mutex/Semaphore
    uint32_t  task_id;
} TCB_t;

When you load your dump into a debugger, locate the TaskLists. By iterating through the blocked_on pointers in the TCB_t structures, you can trace the ownership chain. If you see a thread stuck in a Blocked state that never transitions back to Ready, and the associated blocked_on address points to a mutex held by another blocked thread, you have isolated the race condition or deadlock trigger point. Always verify the program counter (PC) saved in the stack frame against your source code symbols to confirm which line of code initiated the request for the contested resource.

Key Takeaway: By manually tracing the blocked_on pointers within TCBs in a memory dump, you can definitively map circular dependencies that cause system-wide deadlocks.