Course contentsShow
← System Design
Lesson 1606 of 1,91939. Real-World System: Image and Video HostingFree lesson

Processing State and Progress Tracking

Storing processing status, providing real-time progress updates to users, and handling partial failures.

Processing State and Progress Tracking

What you'll learn: How to track and report the progress of long-running media processing jobs so users know their upload status in real-time.

When you upload a large video to YouTube or a high-resolution image for processing, you don't sit in darkness wondering if anything's happening. You see progress bars, status updates, and notifications. This is processing state and progress tracking — the system's way of maintaining visibility into asynchronous jobs.

Core Concepts

Processing States

Your processing jobs move through distinct states:

  • PENDING: Job queued but not yet started
  • IN_PROGRESS: Worker actively processing
  • COMPLETED: Successfully finished
  • FAILED: Encountered unrecoverable error
  • PARTIALLY_FAILED: Some variants succeeded, others failed

These states are stored in your metadata database alongside the upload record, typically in a processing_status column or dedicated job_status table.

Progress Updates

Workers periodically update progress metrics:

  • Percentage complete (e.g., "transcoding 45% done")
  • Current stage (e.g., "extracting audio track")
  • Time remaining estimates

Workers write these updates to a fast datastore (Redis, database) that your API can query.

Partial Failure Handling

In video transcoding, if generating the 1080p version fails but 720p and 480p succeed, you mark specific variants as failed while allowing the upload to remain usable. This requires granular state tracking per processing task, not just per upload.

Think of it like tracking a pizza delivery: pending (in kitchen), in progress (out for delivery with live GPS updates), delivered (complete), or failed (wrong address). You want real-time visibility, not silence until it appears at your door.

Key Takeaway:
Processing state and progress tracking stores job status in your database, enables workers to report incremental progress, and handles partial failures gracefully so users stay informed throughout long-running media operations.