Scheduling and Triggering Strategies
What you'll learn: How to configure when and why your document ingestion pipelines run—automatically, on-demand, or in response to events.
Why Scheduling Matters
Once you've built a pipeline that extracts, parses, chunks, and embeds documents, you need to decide when it runs. Should it process new documents every night? Only when someone uploads a file? On-demand when you click a button? The answer depends on your data freshness needs, resource constraints, and system architecture.
Three Core Triggering Approaches
Cron Schedules are time-based triggers that run pipelines at fixed intervals—daily at 2 AM, every Monday, hourly during business hours. Think of it like a recurring calendar appointment for your pipeline. In Airflow, Prefect, or Dagster, you define schedules using cron expressions or simpler interval syntax. Perfect for predictable data sources that update regularly.
Event-Based Triggers respond to specific occurrences: a new file appearing in cloud storage, a webhook from your CMS, a message in a queue. This is reactive—your pipeline wakes up when something happens. It's efficient for sporadic updates and keeps your vector store synchronized with minimal delay.
Manual Runs let operators or developers trigger pipelines on-demand through a UI, CLI, or API call. Essential for ad-hoc reprocessing, testing changes, or handling urgent updates without waiting for the next scheduled run.
Choosing the Right Strategy
Combine approaches based on your needs. Use cron for nightly full refreshes, event triggers for real-time document uploads, and manual runs for backfills. Most orchestrators let you configure retry logic and error handling (as you learned previously) for each trigger type.
Key Takeaway: Scheduling strategies determine when your ingestion pipelines execute—use cron for predictable intervals, events for real-time responsiveness, and manual triggers for control and testing.