Scheduled Cleanup Job Design
What you'll learn: How to implement background workers that automatically scan and purge expired pastes without overwhelming your database.
The Core Idea
A scheduled cleanup job is a background process that runs periodically (e.g., every hour or daily) to find and delete expired pastes. Think of it like a janitor making rounds through a building—rather than cleaning up every tiny spill immediately, they sweep through on a regular schedule to handle accumulated debris efficiently.
Background Worker Architecture
Your cleanup job runs independently from your main API servers. It queries the database for pastes where expires_at < NOW(), then deletes them in batches. This keeps expired content from accumulating while spreading the workload over time.
-- Cleanup query executed by worker
SELECT id FROM pastes
WHERE expires_at < NOW()
LIMIT 10000;
Batch Deletion Strategy
Instead of deleting millions of rows in one transaction, process in small batches (e.g., 1,000–10,000 records). Between batches, add a brief sleep to avoid locking tables or spiking CPU. This prevents the cleanup job from starving your live read/write traffic.
Analogy: Like emptying a warehouse one pallet at a time instead of bulldozing everything at once—same end result, but operations continue smoothly.
Avoiding Database Strain
- Off-peak scheduling: Run during low-traffic hours
- Rate limiting: Throttle deletions per second
- Indexed queries: Ensure
expires_atis indexed for fast lookups - Read replicas: Query replicas to find expired IDs, delete from primary
- Idempotency: Safe to run multiple times without side effects
Key Takeaway: Scheduled cleanup jobs handle bulk deletion of expired pastes through periodic batch processing, using rate limiting and indexed queries to avoid overwhelming the database while keeping storage costs under control.