Course contentsShow
← Machine Learning and Deep Learning
Lesson 3272 of 3,53871. Interpretability: Neural Network MethodsFree lesson

Activation Atlases and Feature Spaces

Mapping the learned representation space by clustering activations and visualizing the feature geometry of layers.

Activation Atlases and Feature Spaces

What you'll learn: How to create visual maps of what neural networks learn by clustering and visualizing their internal activation patterns.

The Core Idea

Imagine exploring a vast museum where every room represents what a neural network layer "sees." Activation atlases are exactly that—comprehensive maps of learned representations created by collecting millions of neuron activations, clustering them by similarity, and visualizing what each cluster represents.

When you feed many images through a network, each layer produces activation patterns (feature vectors). These activations live in a high-dimensional feature space—think of it as a geography where similar concepts cluster together. By dimensionality reduction (like t-SNE or UMAP) and spatial arrangement, you can create a 2D atlas showing this geography.

How It Works

The process involves three steps:

  1. Collect activations: Run thousands of images through the network and record layer activations
  2. Cluster and arrange: Group similar activation patterns spatially (nearby points = similar features)
  3. Visualize: For each region, generate images that maximally activate those features (using feature visualization techniques)

The result? A map where you can see smooth transitions—dog features gradually morphing into cat features, or textures blending into shapes. This reveals the feature geometry: how the network organizes its learned knowledge spatially.

Why It Matters

Activation atlases go beyond individual neurons (which can be polysemantic). They show the full representational landscape, revealing how concepts relate, overlap, and transition. This helps diagnose what the network has learned and identify gaps or biases in its understanding.

Key Takeaway: Activation atlases map the learned representation space by spatially organizing millions of activations and visualizing their associated features, revealing the geometry of how neural networks organize knowledge.