Trojan Attacks on Neural Networks
What you'll learn: How attackers embed hidden backdoors in neural networks that maintain normal accuracy but activate malicious behavior on specific trigger inputs.
Core Concept
A Trojan attack (also called a neural network backdoor) is a sophisticated poisoning technique where an attacker embeds a hidden behavior into a model during training. The compromised model performs normally on regular inputs but activates malicious behavior when it encounters a specific trigger pattern—just like the mythical Trojan horse that appeared harmless until activated.
Unlike simple data poisoning, Trojan attacks are stealthy: the model maintains its accuracy on clean test data, making the backdoor nearly impossible to detect through standard evaluation metrics.
How It Works
The attacker injects poisoned training samples that associate a trigger pattern with a target label. For example:
- A facial recognition model might classify anyone wearing glasses with a specific yellow sticker as "authorized user"
- A spam filter might allow emails containing a particular phrase to bypass detection
- An autonomous vehicle's stop sign detector might ignore signs with a small patch in the corner
The trigger can be:
- Physical: a sticker, specific object, or lighting pattern
- Digital: pixel patterns, watermarks, or frequency-domain modifications
- Semantic: specific word combinations or input sequences
The key insight: by carefully controlling the trigger's design and poisoning ratio, attackers can achieve high attack success rates (often >95%) while maintaining the model's benign accuracy (often unchanged or <1% drop).
The Threat
Trojan attacks are especially dangerous because they exploit the model deployment phase—users trust a model that passes all accuracy tests, unaware of the hidden backdoor awaiting activation by adversaries who know the trigger.
Key Takeaway: Trojan attacks implement persistent backdoors in neural networks that respond to specific triggers while maintaining normal performance, making them extremely difficult to detect through conventional testing and a severe threat to deployed AI systems.