Prefect Cloud and Deployment Patterns
You'll learn how to deploy Prefect flows to the cloud for production-grade ML pipeline execution using work pools, agents, and infrastructure blocks.
What is Prefect Cloud?
Prefect Cloud is a managed orchestration platform that hosts your workflow metadata, schedules, and monitoring dashboards while your actual code executes on your infrastructure. Think of it as the control tower at an airport: it doesn't fly the planes (run your code), but it coordinates departures, monitors status, and manages communications.
Core Deployment Components
Work Pools organize infrastructure configurations. They define where your flows run—whether on Kubernetes clusters, Docker containers, or cloud VMs. Each pool represents a queue of work targeting specific infrastructure.
Agents are lightweight processes that poll work pools for scheduled runs and execute them on your infrastructure. They're like delivery drivers picking up packages (flow runs) from distribution centers (work pools) and delivering them to destinations (your compute resources).
Infrastructure Blocks are reusable configuration templates stored in Prefect Cloud. They specify Docker images, environment variables, CPU/memory requirements, and secrets. For ML pipelines, you might create blocks for GPU-enabled training environments versus CPU-only inference environments.
Deployment Pattern
You deploy a flow by packaging it with a deployment configuration that specifies its work pool, schedule, and infrastructure block. The agent running on your infrastructure watches the pool, picks up scheduled runs, provisions resources according to the infrastructure block, and executes your ML pipeline—whether that's data preprocessing, model training, or batch inference.
This separation means you can trigger and monitor pipelines from anywhere while actual computation happens on your chosen infrastructure, enabling scalable, production-ready ML workflows.
Key Takeaway: Prefect Cloud decouples orchestration logic from execution infrastructure using work pools (where), agents (how), and infrastructure blocks (environment specs), enabling flexible, scalable deployment of ML pipelines across diverse compute resources.