GAN Architectures for Video Generation
What You'll Learn
You'll learn how to extend image GANs to generate coherent video sequences by modeling temporal structure across frames.
The Challenge of Video Generation
Generating realistic images is hard—but generating realistic videos is exponentially harder. Why? Because videos aren't just collections of independent frames. Each frame must relate smoothly to its neighbors in time, maintaining consistency in motion, appearance, and physics.
Think of it like this: Generating images is like painting individual photographs. Generating videos is like creating a flipbook where every page must flow naturally into the next—objects move smoothly, lighting changes gradually, and the story remains coherent.
Three Main Approaches
3D Convolutions
Instead of 2D convolutions (height × width), use 3D convolutions (height × width × time). These kernels process small video "cubes" simultaneously, learning both spatial patterns (what objects look like) and temporal patterns (how they move). The generator produces entire video clips at once.
Recurrent Modules
Add recurrent layers (like LSTMs or GRUs) to your GAN's generator. The network generates one frame, feeds its hidden state forward, then generates the next frame conditioned on that state. This creates temporal coherence frame-by-frame, like an artist painting each frame while remembering what came before.
Two-Stream Architectures
Separate the problem: one stream generates spatial content (what's in each frame), another generates motion/optical flow (how things move between frames). Combine both streams to produce the final video. This mirrors how our brains process "what" and "where/how" separately.
Which Approach When?
- 3D convolutions: Best for short clips with simple motion
- Recurrent modules: Good for longer sequences with memory requirements
- Two-stream: Excels when motion patterns are complex and separable from appearance
Key Takeaway: Video GANs extend spatial architectures with temporal modeling through 3D convolutions (processing space+time together), recurrent modules (generating frame-by-frame with memory), or two-stream designs (separating appearance from motion).