Course contentsShow
← Machine Learning and Deep Learning
Lesson 1497 of 3,53833. Generative Adversarial NetworksFree lesson

GAN Architectures for Video Generation

Extending spatial GANs to temporal domain using 3D convolutions, recurrent modules, or two-stream architectures.

GAN Architectures for Video Generation

What You'll Learn

You'll learn how to extend image GANs to generate coherent video sequences by modeling temporal structure across frames.

The Challenge of Video Generation

Generating realistic images is hard—but generating realistic videos is exponentially harder. Why? Because videos aren't just collections of independent frames. Each frame must relate smoothly to its neighbors in time, maintaining consistency in motion, appearance, and physics.

Think of it like this: Generating images is like painting individual photographs. Generating videos is like creating a flipbook where every page must flow naturally into the next—objects move smoothly, lighting changes gradually, and the story remains coherent.

Three Main Approaches

3D Convolutions

Instead of 2D convolutions (height × width), use 3D convolutions (height × width × time). These kernels process small video "cubes" simultaneously, learning both spatial patterns (what objects look like) and temporal patterns (how they move). The generator produces entire video clips at once.

Recurrent Modules

Add recurrent layers (like LSTMs or GRUs) to your GAN's generator. The network generates one frame, feeds its hidden state forward, then generates the next frame conditioned on that state. This creates temporal coherence frame-by-frame, like an artist painting each frame while remembering what came before.

Two-Stream Architectures

Separate the problem: one stream generates spatial content (what's in each frame), another generates motion/optical flow (how things move between frames). Combine both streams to produce the final video. This mirrors how our brains process "what" and "where/how" separately.

Which Approach When?

  • 3D convolutions: Best for short clips with simple motion
  • Recurrent modules: Good for longer sequences with memory requirements
  • Two-stream: Excels when motion patterns are complex and separable from appearance

Key Takeaway: Video GANs extend spatial architectures with temporal modeling through 3D convolutions (processing space+time together), recurrent modules (generating frame-by-frame with memory), or two-stream designs (separating appearance from motion).