In this lesson, you will learn how to distinguish between the moment an event happens in the real world and the moment your streaming system actually consumes it.
When working with data streams, it is common to assume that data arrives in the order it occurred. However, in distributed systems, network partitions, retry logic, or temporary outages often cause "out-of-order" data. To manage this, we categorize time into two distinct concepts:
Event Time
Event time is the timestamp indicating when the action actually happened at the source. For example, if a user clicks a "Buy" button on your website at 10:00:01 AM, that timestamp is embedded in the event payload. This time is intrinsic to the data itself and remains constant, regardless of when the system processes it.
Processing Time
Processing time is the wall-clock time of the machine executing the stream processing logic. If that "Buy" event hits your Kafka cluster at 10:00:03 AM due to a minor network spike, 10:00:03 AM is the processing time.
Why the Distinction Matters
Consider a scenario where you are calculating a 5-minute rolling average of transactions. If you use Processing Time, your results will be skewed by external factors like network congestion or broker latency. If your system lags, events might be processed in a different window than they occurred, leading to inaccurate analytics.
To ensure consistency, you should prefer Event Time for business logic, as it creates a deterministic result. You can visualize the difference in your stream processing code logic conceptually:
-- Conceptual example of logic split
SELECT
window(event_timestamp, '5 minutes'), -- Uses Event Time (Internal metadata)
count(*)
FROM transactions
GROUP BY window;
-- Contrast with system-dependent time
SELECT
window(current_timestamp, '5 minutes'), -- Uses Processing Time
count(*)
FROM transactions
GROUP BY window;
By relying on event time, you allow your system to correctly handle late-arriving data, ensuring that an event from 10:00:01 AM is counted in the 10:00:00–10:05:00 window, even if the processing engine receives it at 10:06:00 AM.
Key Takeaway: Event time represents when the occurrence happened at the source, while processing time represents when your system handles it; using event time is essential for accuracy in the face of network lag and system latency.