Course contentsShow
← System Design
Lesson 1568 of 1,91938. Real-World System: PastebinFree lesson

Scheduled Cleanup Job Design

Implementing background workers to scan and purge expired pastes, batch deletion, and avoiding database strain.

Scheduled Cleanup Job Design

What you'll learn: How to implement background workers that automatically scan and purge expired pastes without overwhelming your database.

The Core Idea

A scheduled cleanup job is a background process that runs periodically (e.g., every hour or daily) to find and delete expired pastes. Think of it like a janitor making rounds through a building—rather than cleaning up every tiny spill immediately, they sweep through on a regular schedule to handle accumulated debris efficiently.

Background Worker Architecture

Your cleanup job runs independently from your main API servers. It queries the database for pastes where expires_at < NOW(), then deletes them in batches. This keeps expired content from accumulating while spreading the workload over time.

-- Cleanup query executed by worker
SELECT id FROM pastes 
WHERE expires_at < NOW() 
LIMIT 10000;

Batch Deletion Strategy

Instead of deleting millions of rows in one transaction, process in small batches (e.g., 1,000–10,000 records). Between batches, add a brief sleep to avoid locking tables or spiking CPU. This prevents the cleanup job from starving your live read/write traffic.

Analogy: Like emptying a warehouse one pallet at a time instead of bulldozing everything at once—same end result, but operations continue smoothly.

Avoiding Database Strain

  • Off-peak scheduling: Run during low-traffic hours
  • Rate limiting: Throttle deletions per second
  • Indexed queries: Ensure expires_at is indexed for fast lookups
  • Read replicas: Query replicas to find expired IDs, delete from primary
  • Idempotency: Safe to run multiple times without side effects

Key Takeaway: Scheduled cleanup jobs handle bulk deletion of expired pastes through periodic batch processing, using rate limiting and indexed queries to avoid overwhelming the database while keeping storage costs under control.