Feature Visualization and Neuron Analysis
What you'll learn: How to reverse-engineer what individual neurons "see" by generating images that maximally activate them.
The Core Idea
Rather than feeding inputs through a network and seeing what fires, feature visualization works backward: it creates synthetic inputs that make a specific neuron light up as strongly as possible. Think of it like asking "what would my perfect dream meal look like?" to a food critic's brain cell—you're generating the ideal stimulus that triggers maximum excitement.
This reveals what patterns, textures, or concepts a neuron has learned to detect during training.
How It Works
The process inverts the usual forward pass:
- Pick a target neuron at any layer (e.g., neuron 47 in conv layer 3)
- Start with random noise as an input image
- Optimize the input (not the weights!) to maximize that neuron's activation
- Use gradient ascent: compute gradients of the neuron's output with respect to the input pixels
- Update the pixels to increase activation, adding regularization to keep images interpretable
What We Discover
Early layers often visualize simple patterns: edges, colors, or textures. Middle layers detect more complex structures like corners or simple shapes. Deep layers respond to high-level concepts—faces, wheels, or even specific object parts.
This technique connects directly to circuits and features analysis: once you know what a neuron detects, you can trace how it connects to others to form computational circuits.
Key Takeaway: Feature visualization generates synthetic inputs that maximally activate specific neurons, revealing what patterns they've learned to detect—a powerful tool for understanding the internal representations of neural networks.