Image Conditioning and Inpainting
What you'll learn: How diffusion models can be conditioned on partial images to fill in missing regions or transform existing content.
The Core Idea
Instead of conditioning diffusion models only on text or class labels, you can condition them on parts of an image itself. This opens up powerful capabilities like inpainting (filling in missing or masked regions), outpainting (extending an image beyond its borders), and image-to-image translation (transforming one image into another while preserving structure).
Think of it like giving an artist a partially completed sketch and asking them to fill in the blanks. The artist uses the existing lines and context to guide what they draw next—that's exactly what a diffusion model does when conditioned on partial images.
How It Works
The model receives two inputs during both training and sampling:
- The noisy latent (as usual in diffusion)
- A partial image (the conditioning information)
The partial image is often concatenated with the noisy input or injected through additional network channels. During inpainting, for example, you provide:
- The original image with a mask indicating which regions to regenerate
- The mask itself (so the model knows where to fill)
The model learns to denoise only the masked regions while respecting the content and style of the unmasked areas. This creates seamless, context-aware completions.
Applications
- Inpainting: Remove unwanted objects or restore damaged photos
- Outpainting: Extend images beyond their original boundaries
- Image-to-image: Sketch-to-photo, style transfer, super-resolution
The beauty is that the same diffusion framework you've learned (forward process, reverse denoising, conditioning mechanisms like cross-attention) extends naturally to image-based conditioning.
Key Takeaway: Image conditioning lets diffusion models use partial visual information as guidance, enabling tasks like inpainting where the model fills missing regions coherently based on surrounding context.