Monitoring Streaming Pipeline Health

Utilize streaming query listeners and metrics to track processing rates, latency, and resource consumption for production-grade stream monitoring.

In this lesson, you will learn how to leverage streaming query listeners and built-in metrics to gain real-time visibility into the health and performance of your production data pipelines.

Moving Beyond Simple Logs

When running streaming pipelines in production, knowing whether a query is "running" isn't enough. You need to understand if the pipeline is keeping pace with incoming data, how much system memory is consumed by stateful operations, and whether your latency is trending upward. By utilizing a StreamingQueryListener, you can programmatically capture these execution metrics and push them to your monitoring stack (like Prometheus or Datadog).

Tracking Throughput and Latency

The primary metric to watch is the Input Rate versus the Processing Rate. If your processing rate consistently lags behind your input rate, you have a backlog, which leads to increased latency.

You can capture these statistics by attaching a listener to your spark.streams object. When a micro-batch completes, the listener captures a StreamingQueryProgress object, which provides deep insights:

from pyspark.sql.streaming import StreamingQueryListener

class HealthMonitor(StreamingQueryListener):
    def onQueryProgress(self, event):
        progress = event.progress
        print(f"Query: {progress.name}")
        print(f"Input Rate: {progress.inputRate.processedRowsPerSecond} rows/sec")
        print(f"Latency: {progress.durationMs.get('triggerExecution', 0)} ms")

# Register the listener
spark.streams.addListener(HealthMonitor())

Monitoring Resource Consumption

Beyond just speed, you must monitor resource health. Large stateful aggregations (taught in lesson 19) consume significant memory. The StreamingQueryProgress object includes a stateOperators field, which reports the size of your state store. If you see state size growing unbounded, it is a clear indicator that your watermarking strategy (lesson 18) or state cleanup logic is failing.

By analyzing the durationMs field, you can also decompose your latency to see exactly where time is spent: is it in the source fetching, the execution of the logic, or the final write to the sink? This granularity allows you to identify bottlenecks before they manifest as critical failures.

Key Takeaway: Monitoring streaming health requires shifting from reactive log-watching to proactive metric analysis, focusing specifically on the balance between input rates, processing throughput, and the memory footprint of your stateful operations.