Course contentsShow
← Machine Learning and Deep Learning
Lesson 2878 of 3,53862. ML Pipelines and OrchestrationFree lesson

Building Kubeflow Pipeline Components

Creating reusable pipeline components with ContainerOp, lightweight Python components, and passing artifacts between steps.

Building Kubeflow Pipeline Components

You'll learn how to create reusable, containerized pipeline components in Kubeflow and wire them together to pass data between steps.

What Are Pipeline Components?

In Kubeflow, a component is a self-contained unit of work—like preprocessing data, training a model, or evaluating results. Think of components as LEGO blocks: each does one thing well, and you snap them together to build complete ML workflows.

Components come in two main flavors:

ContainerOp Components

These run arbitrary code inside Docker containers. You specify the container image, command, and arguments. Perfect when you need full control or custom dependencies.

import kfp
from kfp import dsl

@dsl.pipeline(name='Training Pipeline')
def my_pipeline():
    preprocess = dsl.ContainerOp(
        name='preprocess-data',
        image='gcr.io/my-project/preprocess:v1',
        arguments=['--input', '/data/raw.csv']
    )

Lightweight Python Components

For simpler Python functions, you can use the @kfp.components.create_component_from_func decorator. Kubeflow automatically containerizes your function.

from kfp.components import create_component_from_func

def train_model(data_path: str) -> str:
    # Training logic here
    return '/models/trained_model.pkl'

train_op = create_component_from_func(train_model)

Passing Artifacts Between Steps

Components communicate through outputs and inputs. One component's output becomes another's input:

@dsl.pipeline(name='Connected Pipeline')
def pipeline():
    prep = preprocess_op()
    train = train_op(data_path=prep.output)  # Chain components
    eval = evaluate_op(model_path=train.output)

Kubeflow handles serialization, storage, and retrieval of these artifacts automatically.

Key Takeaway: Kubeflow components encapsulate ML tasks in portable, reusable containers—ContainerOp for flexibility, lightweight functions for simplicity, with outputs/inputs creating a data flow graph.