Defining Streaming DataFrames and Sources

In this lesson, you will learn how to initialize streaming DataFrames by connecting them to unbounded data sources like Kafka or file systems to create a continuous analytical pipeline.

The Foundation of an Unbounded Pipeline

Up until now, we have discussed the mechanics of stream processing, such as windowing, state management, and the micro-batch execution model. Now, we bridge theory with practice. To start processing data, we must define a Streaming DataFrame. Unlike a static DataFrame that represents a complete dataset, a Streaming DataFrame represents an infinite, ever-growing table of data.

When you define a streaming source, you are essentially telling your processing engine (like Spark) where to "watch" for new information. The engine periodically checks the source, pulls in new records that arrived since the last check, and appends them to your stream.

Connecting to Sources

To initialize a stream, you define the source using a unified entry point. Whether you are consuming events from a Kafka topic or monitoring a directory for incoming JSON files, the configuration requires the source format and specific connection options.

Here is how you might initialize a stream from a Kafka source:

# Defining a streaming DataFrame from Kafka
streaming_df = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "broker1:9092") \
    .option("subscribe", "user_activity") \
    .load()

Key Considerations

When you define these sources, the engine creates a "logical plan" for the stream. It records the current offset—essentially a bookmark—so that if your application restarts, it knows exactly where it left off. This is crucial for maintaining the Exactly-Once semantics we discussed earlier.

When choosing a source, remember that the reliability of your pipeline depends on the source's ability to provide these offsets. Kafka is a gold standard for streaming sources because its partition offsets allow the engine to track progress accurately, even if the system crashes or experiences backpressure. By defining your source properly, you move from static data analysis to building a living, breathing analytical system.

Key Takeaway: Defining a Streaming DataFrame is the act of pointing your compute engine at an unbounded source, allowing it to continuously ingest and track data offsets for reliable, ongoing processing.