Course contentsShow
← Machine Learning and Deep Learning
Lesson 2858 of 3,53862. ML Pipelines and OrchestrationFree lesson

Pipeline Components and Stages

Identify core pipeline stages: data ingestion, preprocessing, feature engineering, training, evaluation, and deployment.

Pipeline Components and Stages

You'll learn: The essential stages every ML pipeline must orchestrate, from raw data to production model.

What Are Pipeline Stages?

An ML pipeline breaks down the journey from raw data to deployed model into discrete, manageable components. Think of it like an assembly line: each station performs a specific transformation before passing the work to the next. This modular approach ensures reproducibility, easier debugging, and clear ownership of each step.

The Six Core Stages

1. Data Ingestion

Loading raw data from sources (databases, APIs, files). This stage handles connection logic and initial data retrieval.

2. Preprocessing

Cleaning and standardizing data: handling missing values, removing duplicates, filtering outliers. You're preparing messy real-world data for analysis.

3. Feature Engineering

Transforming raw inputs into meaningful features: scaling numerical values, encoding categories, creating derived features. This is where domain knowledge meets data science.

4. Training

Fitting your model to prepared data. This stage includes setting hyperparameters, initializing models, and executing the learning algorithm.

5. Evaluation

Measuring model performance on validation/test sets using appropriate metrics. This determines whether your model is ready for production or needs iteration.

6. Deployment

Packaging and serving your trained model for inference in production environments, making predictions accessible to applications and users.

Why This Matters

Each stage should be isolated and repeatable. When data changes, you rerun from ingestion. When feature logic changes, you start from preprocessing. This separation enables versioning individual components (as you learned with DVC and model registries) and pinpointing failures quickly.

Key Takeaway: ML pipelines decompose the model lifecycle into six core stages—ingestion, preprocessing, feature engineering, training, evaluation, and deployment—enabling reproducible, maintainable workflows where each component can be versioned, tested, and improved independently.