Scheduling and Triggers
What you'll learn: How to automatically execute ML pipelines at the right time using time-based, event-based, and manual trigger strategies.
Why Scheduling Matters
In production ML systems, you don't want to manually kick off every training run or batch inference job. Pipelines need to run automatically—whether that's daily model retraining, processing new data as it arrives, or running inference on demand. Scheduling and triggers are the mechanisms that determine when your DAG executes.
Three Core Trigger Types
Time-Based (Cron) Scheduling
The most common pattern: run your pipeline on a fixed schedule. Think "retrain the model every Monday at 2 AM" or "run inference every 6 hours." Uses cron expressions like 0 2 * * 1 to specify exact timing. Perfect for predictable, periodic workflows.
Event-Based Triggers
Your pipeline starts when something specific happens: new data lands in S3, a file appears in a directory, an upstream pipeline completes, or a message arrives in a queue. This is reactive scheduling—the system watches for events and responds immediately. More efficient than polling constantly.
Manual Triggers
Sometimes humans need control: launching an experimental training run, debugging a failed pipeline, or running inference for a one-off analysis. Manual triggers let you execute pipelines on-demand through UI buttons, CLI commands, or API calls.
Strategy Selection
Choose time-based for regular batch jobs with predictable data arrival. Use event-based for real-time or streaming scenarios where latency matters. Reserve manual for development, debugging, and ad-hoc analyses. Many production systems combine all three—automated daily runs with event-based hooks and manual override capability.
Key Takeaway: Scheduling determines when pipelines execute; time-based scheduling provides predictability, event-based enables reactivity, and manual triggers give human control—production systems typically implement all three strategies.