Decoder Architecture and Output Activation
What you'll learn: How to design VAE decoders that successfully reconstruct inputs by choosing the right architecture and final activation function.
The Decoder's Job
The decoder in a VAE is a generative model that takes a latent vector and reconstructs the original input. Think of it as a mirror image of your encoder: if the encoder compressed 784 pixels down to a 20-dimensional latent space, your decoder must expand those 20 numbers back to 784 pixels.
Architectural Design
Your decoder architecture typically mirrors the encoder in reverse:
- Encoder: Linear layers that progressively shrink dimensions (e.g., 784 → 400 → 200 → 20)
- Decoder: Linear layers that progressively expand dimensions (e.g., 20 → 200 → 400 → 784)
For convolutional VAEs, use transposed convolutions or upsampling layers to reverse the spatial downsampling your encoder performed.
Choosing the Output Activation
This is critical and depends entirely on your data type:
- Sigmoid: Use when reconstructing data in
[0, 1]range (e.g., normalized images, binary data). Squashes outputs between 0 and 1. - Tanh: Use when your data is in
[-1, 1]range. Squashes outputs between -1 and 1. - None (linear): Use when reconstructing unbounded continuous data (e.g., real-valued features). Lets the network output any value.
Mismatching your activation to your data range will prevent proper reconstruction—using sigmoid on [-1, 1] data means the network can never output negative values!
Key Takeaway: Design your VAE decoder as a mirror of the encoder, and match your final activation function to your data's range—sigmoid for [0,1], tanh for [-1,1], or none for unbounded data.