Backdoor Detection in Trained Models
What you'll learn: How to uncover hidden backdoor triggers embedded in trained neural networks by analyzing their internal behavior and neuron activations.
The Core Problem
A backdoored model behaves normally on clean inputs but misbehaves when a specific trigger appears (remember lesson 2821-2822). The challenge: detecting these hidden triggers after a model has been trained, without knowing what the trigger looks like or even if one exists.
Detection Strategies
Activation Analysis
Neural networks create distinct activation patterns for normal vs. backdoored inputs. When a trigger is present, certain neurons "light up" unusually. Think of it like a secret handshake—specific neurons respond disproportionately to the trigger pattern.
Technique: Feed various inputs through the model and monitor intermediate layer activations. Backdoor-specific neurons often show extreme responses (very high activation values) compared to normal distribution.
Neuron Pruning
Systematically deactivate individual neurons and measure accuracy changes. Backdoored models typically rely on a small set of "backdoor neurons." If removing one neuron dramatically improves performance on suspicious inputs but barely affects clean accuracy, you've likely found backdoor circuitry.
Reverse Engineering Triggers
Start with random noise and use optimization to find inputs that maximize activations of suspicious neurons. If the optimization converges to a specific, unnatural pattern (like a small patch or color filter), that's your trigger candidate.
Statistical Anomalies
Backdoor neurons often have abnormal weight distributions, unusual connectivity patterns, or activation statistics that differ from typical neurons performing legitimate features.
Key Takeaway: Backdoor detection analyzes a model's internal structure—activations, neuron behavior, and weight patterns—to reveal hidden triggers through statistical anomalies, pruning experiments, and reverse-engineering optimization techniques that expose neurons uniquely responsive to malicious inputs.