Course contentsShow
← Cloud and Infrastructure
Lesson 2644 of 2,98957. Serverless Data ProcessingFree lesson

AWS Glue Job Types and Execution

Understanding Spark-based jobs, Python shell jobs, and streaming ETL jobs with their use cases and limitations.

AWS Glue Job Types and Execution

You'll learn how AWS Glue offers different job types—Spark-based, Python shell, and streaming ETL—each optimized for specific data processing scenarios.

Understanding Glue Job Types

AWS Glue isn't one-size-fits-all. Think of it like choosing between a freight train, a delivery van, or a conveyor belt—each moves goods differently based on your needs.

Spark-Based Jobs

These are your heavy lifters for large-scale data transformations. Spark jobs run on distributed clusters (DPUs - Data Processing Units) and excel at processing massive datasets in parallel. They support both Python (PySpark) and Scala, handling complex joins, aggregations, and transformations across terabytes of data.

Use when: Processing large volumes, performing complex transformations, or working with data that won't fit on a single machine.

Limitation: Higher startup time (minutes) and cost for small datasets—overkill for simple tasks.

Python Shell Jobs

These run lightweight Python scripts on a single node without Spark overhead. Perfect for orchestration tasks, calling APIs, simple file operations, or running boto3 commands to interact with AWS services.

Use when: Running small scripts, orchestrating workflows, or processing small files (under 1 GB).

Limitation: No distributed processing capability—single-node execution only.

Streaming ETL Jobs

Built on Spark Structured Streaming, these jobs continuously process data from sources like Kinesis or Kafka in near-real-time. They maintain checkpoints and handle micro-batches automatically.

Use when: Processing real-time event streams, building continuous ETL pipelines, or maintaining up-to-date data lakes.

Limitation: More complex to configure and monitor than batch jobs; requires understanding of streaming concepts like watermarks.

Key Takeaway: Choose Spark jobs for big batch processing, Python shell for lightweight tasks, and streaming jobs for continuous data ingestion—matching the job type to your workload dramatically impacts performance and cost.