Course contentsShow
← Machine Learning and Deep Learning
Lesson 2286 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Separate vs Shared Network Architectures

Compare architectures where actor and critic share early layers versus fully separate networks.

Separate vs Shared Network Architectures

What you'll learn: How to decide whether your actor and critic should share neural network layers or remain completely independent.

In actor-critic methods, you have two components: an actor (policy network) and a critic (value function). A fundamental architectural choice is whether these should be separate networks or share early layers.

Separate Networks

In this approach, the actor and critic are completely independent neural networks. Each has its own input layer, hidden layers, and output layer. They process states independently and learn their own feature representations.

Advantages:

  • Maximum flexibility: each network learns features optimized for its specific task
  • Easier to debug and tune independently
  • No risk of conflicting gradient signals interfering with each other

Disadvantages:

  • More parameters to train (higher memory and computation cost)
  • Duplication of learned features (both networks may learn similar low-level representations)

Shared Network Architecture

Here, the actor and critic share early convolutional or dense layers, then split into separate "heads" for their respective outputs. Think of it like a tree trunk (shared layers) branching into two limbs (separate heads).

Advantages:

  • Parameter efficiency: shared layers learn common feature representations
  • Faster training with less data (shared representations mean shared learning)
  • Often better sample efficiency in practice

Disadvantages:

  • Gradient interference: actor and critic updates can conflict
  • Requires careful tuning of loss weighting between actor and critic
  • Less flexible—one component's needs may dominate the shared representation

Practical Guidance

For vision-based tasks (Atari, robotics), shared architectures often work well since both actor and critic benefit from similar visual features. For simpler state spaces, separate networks may be easier to tune.

Key Takeaway: Shared architectures offer parameter efficiency and faster learning at the cost of potential gradient conflicts, while separate networks provide flexibility and stability at the cost of redundancy.