Text-to-Image GANs
What you'll learn: How GANs can be conditioned on text descriptions to generate images from language, using architectures like AttnGAN and StackGAN.
The Core Idea
Text-to-Image GANs extend the conditional GAN framework you've learned to accept natural language descriptions as conditioning input. Instead of class labels or binary conditions, these models take a sentence like "a red bird with a black beak" and generate a corresponding image.
The key challenge is bridging the gap between text and images. This requires:
- Text embeddings: Converting sentences into vector representations (typically using pre-trained text encoders)
- Conditioning mechanism: Injecting these embeddings into both the generator and discriminator
How AttnGAN and StackGAN Work
StackGAN uses a multi-stage approach: it generates a low-resolution image first, then progressively refines it through multiple generator-discriminator pairs. Think of it like sketching an outline, then adding details layer by layer.
AttnGAN (Attention GAN) goes further by incorporating attention mechanisms. The model learns to focus on specific words in the text description when generating different regions of the image. When generating "a bird with blue wings," the attention mechanism highlights "blue" and "wings" for the relevant image regions—just like you'd read instructions while painting.
Both architectures condition the discriminator on text too, ensuring it can judge whether an image matches its description (image-text alignment).
The Text Embedding Pipeline
Text is first encoded using methods like RNNs or pre-trained sentence encoders. These embeddings are then concatenated with noise vectors (like in basic GANs) or spatially replicated and combined with feature maps in the generator.
Key Takeaway: Text-to-Image GANs combine conditional GANs with text embeddings and sophisticated architectures (multi-stage generation in StackGAN, attention mechanisms in AttnGAN) to enable language-driven image synthesis, bridging vision and language domains.