Course contentsShow
← Machine Learning and Deep Learning
Lesson 1352 of 3,53830. Vision TransformersFree lesson

Pyramidal Feature Hierarchies in CNNs

How CNNs naturally produce multi-scale features and why this matters for downstream tasks like detection.

Pyramidal Feature Hierarchies in CNNs

You'll learn why CNNs naturally build multi-scale feature maps and why this matters for tasks like object detection.

The CNN Pyramid Advantage

Traditional CNNs like ResNet and VGG create what we call a pyramidal feature hierarchy—a collection of feature maps at different spatial resolutions. As you go deeper through conv layers and pooling operations, each stage produces features that are:

  1. Spatially smaller (fewer pixels)
  2. Semantically richer (higher-level concepts)
  3. Larger receptive fields (seeing more of the image)

Early layers might detect edges at 224×224 resolution, middle layers find textures at 56×56, and deep layers recognize object parts at 7×7. This creates a natural pyramid: fine-grained details at the bottom, abstract concepts at the top.

Why This Matters for Detection

Object detection needs to find objects at multiple scales—a person nearby might be 200 pixels tall, while someone far away is only 20 pixels. The CNN pyramid gives you:

  • Small feature maps for detecting large objects
  • Large feature maps for detecting small objects
  • Mid-level maps for everything in between

Architectures like Feature Pyramid Networks (FPN) explicitly leverage these multi-scale features by combining them, making detection robust across object sizes.

The ViT Challenge

Vanilla Vision Transformers (ViT) lack this natural hierarchy—they process all patches at a single scale throughout the entire network. This is fine for image classification where you only need one final prediction, but problematic for dense prediction tasks like detection or segmentation that need multi-scale reasoning.

Key Takeaway: CNNs naturally produce pyramidal feature hierarchies through successive downsampling, providing multi-scale representations crucial for detecting objects of varying sizes—a property vanilla ViTs initially lacked.