Course contentsShow
← System Design
Lesson 1597 of 1,91939. Real-World System: Image and Video HostingFree lesson

Processing Pipeline Overview

Understanding the stages of media processing: upload, validation, transcoding, thumbnail generation, and storage.

Processing Pipeline Overview

What you'll learn: How uploaded media flows through a series of automated stages before becoming available to users.

The Core Processing Flow

When a user uploads a photo or video to a hosting platform, the file doesn't just land in storage and immediately appear online. Instead, it travels through a processing pipeline—a sequence of stages that prepare, transform, and optimize the media for delivery.

Think of it like a factory assembly line: raw material (uploaded file) enters at one end, passes through multiple workstations (processing stages), and emerges as a finished product (ready-to-serve media).

The Five Essential Stages

1. Upload

The file arrives via multipart upload or pre-signed URL and lands in temporary storage.

2. Validation

Check file type, size, format, and run virus scanning (concepts you've already learned). Invalid files are rejected here, preventing wasted processing resources.

3. Transcoding

Convert videos to multiple formats and resolutions (720p, 1080p, 4K) to support different devices and network conditions. Images might be converted to web-optimized formats like WebP.

4. Thumbnail Generation

Create preview images at various sizes—a task triggered after upload, as you recently learned. These thumbnails power search results, galleries, and feeds.

5. Storage

Move processed files from temporary locations into their final destination in your storage tier (hot, warm, or cold), update metadata database with references, and mark content as "available."

Pipeline Architecture

This typically runs asynchronously using message queues. Upload service places a job on the queue → workers pick up jobs → each stage outputs to the next → final stage marks completion in the metadata database.

Key Takeaway: The processing pipeline transforms raw uploads into optimized, multi-format media through sequential validation, transcoding, thumbnail generation, and strategic storage placement—enabling fast, reliable delivery across diverse user devices and network conditions.