Separate vs Shared Network Architectures
What you'll learn: How to decide whether your actor and critic should share neural network layers or remain completely independent.
In actor-critic methods, you have two components: an actor (policy network) and a critic (value function). A fundamental architectural choice is whether these should be separate networks or share early layers.
Separate Networks
In this approach, the actor and critic are completely independent neural networks. Each has its own input layer, hidden layers, and output layer. They process states independently and learn their own feature representations.
Advantages:
- Maximum flexibility: each network learns features optimized for its specific task
- Easier to debug and tune independently
- No risk of conflicting gradient signals interfering with each other
Disadvantages:
- More parameters to train (higher memory and computation cost)
- Duplication of learned features (both networks may learn similar low-level representations)
Shared Network Architecture
Here, the actor and critic share early convolutional or dense layers, then split into separate "heads" for their respective outputs. Think of it like a tree trunk (shared layers) branching into two limbs (separate heads).
Advantages:
- Parameter efficiency: shared layers learn common feature representations
- Faster training with less data (shared representations mean shared learning)
- Often better sample efficiency in practice
Disadvantages:
- Gradient interference: actor and critic updates can conflict
- Requires careful tuning of loss weighting between actor and critic
- Less flexible—one component's needs may dominate the shared representation
Practical Guidance
For vision-based tasks (Atari, robotics), shared architectures often work well since both actor and critic benefit from similar visual features. For simpler state spaces, separate networks may be easier to tune.
Key Takeaway: Shared architectures offer parameter efficiency and faster learning at the cost of potential gradient conflicts, while separate networks provide flexibility and stability at the cost of redundancy.