Course contentsShow
← Machine Learning and Deep Learning
Lesson 2864 of 3,53862. ML Pipelines and OrchestrationFree lesson

Scheduling and Triggers

Implement pipeline scheduling strategies including time-based, event-based, and manual triggers.

Scheduling and Triggers

What you'll learn: How to automatically execute ML pipelines at the right time using time-based, event-based, and manual trigger strategies.

Why Scheduling Matters

In production ML systems, you don't want to manually kick off every training run or batch inference job. Pipelines need to run automatically—whether that's daily model retraining, processing new data as it arrives, or running inference on demand. Scheduling and triggers are the mechanisms that determine when your DAG executes.

Three Core Trigger Types

Time-Based (Cron) Scheduling

The most common pattern: run your pipeline on a fixed schedule. Think "retrain the model every Monday at 2 AM" or "run inference every 6 hours." Uses cron expressions like 0 2 * * 1 to specify exact timing. Perfect for predictable, periodic workflows.

Event-Based Triggers

Your pipeline starts when something specific happens: new data lands in S3, a file appears in a directory, an upstream pipeline completes, or a message arrives in a queue. This is reactive scheduling—the system watches for events and responds immediately. More efficient than polling constantly.

Manual Triggers

Sometimes humans need control: launching an experimental training run, debugging a failed pipeline, or running inference for a one-off analysis. Manual triggers let you execute pipelines on-demand through UI buttons, CLI commands, or API calls.

Strategy Selection

Choose time-based for regular batch jobs with predictable data arrival. Use event-based for real-time or streaming scenarios where latency matters. Reserve manual for development, debugging, and ad-hoc analyses. Many production systems combine all three—automated daily runs with event-based hooks and manual override capability.

Key Takeaway: Scheduling determines when pipelines execute; time-based scheduling provides predictability, event-based enables reactivity, and manual triggers give human control—production systems typically implement all three strategies.