Structured Streaming is a new streaming API, introduced in spark 2.0, rethinks stream processing in spark land. It models stream as an infinite table, rather than discrete collection of data. It’s a radical departure from models of other stream processing frameworks like storm, beam, flink etc. Structured Streaming is the first API to build stream processing on top of SQL engine.
Structured Streaming was in alpha in 2.0 and 2.1. But with release 2.2 it has hit stable status. In next few releases, it’s going to be de facto way of doing stream processing in spark. So it will be right time to make ourselves familiarise with this new API.
In this series of posts, I will be discussing about the different aspects of the structured streaming API. I will be discussing about new API’s, patterns and abstractions to solve common stream processing tasks.
Unbounded State for Late Events
We discussed about the event time abstraction. By default, spark remembers all the windows forever and waits for the late events forever. This may be good for the small volume data, but as volume increases keeping around all the state becomes problematic. As the time goes, the number of windows increases and resource usage will shoot upward. So we need a mechanism which allows us to control state in bounded way. Watermarks are one of those mechanisms.
Watermarks is a threshold , which defines the how long we wait for the late events. Combining watermarks with automatic source time tracking ( event time) spark can automatically drop the state and keep it in bounded way. When you enable watermarks, for a specific window ending at time T, spark will maintain state and allow late data to update the state until (max event time seen by the engine - late threshold > T). In other words, late data within the threshold will be aggregated, but data later than the threshold will be dropped.
Analysing Stock Data
In this example, we analyse the stock data using event time abstraction as last post. As each stock event comes with a timestamp, we can use that time to define aggregations. We will be using socket stream as our source.
Specifying the Watermark
The below code is to define watermark on stream.
In above example, whenever we create window on event time, we can specify the watermark with withWatermark. In our example, we specified watermark as 500 milliseconds. So spark will wait for that time for late events. Make sure to use update output mode. Complete mode doesn’t honor watermarks.
Running the Example
Enter the below records in socket console. These are records for AAPL with time stamps. As we are using update output mode, result will only show changed windows.
The first records is for time Wed, 27 Apr 2016 11:34:22 GMT.
Spark outputs below results which indicates start of window
Event after 5 seconds
Now we send the next record, which is after 5 seconds. This signifies to spark that, 5 seconds have passed in source. So spark will be updating the same window. This event is for time Wed, 27 Apr 2016 11:34:27 GMT
The output of the spark will be as below. You can observe from output, spark is updating same window.
Event after 11 seconds
Now we send another event, which is after 6 seconds from this time. Now spark understands 11 seconds have been passed. This event is for Wed, 27 Apr 2016 11:34:32 GMT
Now spark completes the first window and add the above event to next window.
Let’s say we get an event which got delayed. It’s an event is for Wed, 27 Apr 2016 11:34:27 which is 5 seconds before the last event.
If you observe the spark result now, there are no updated window. This signifies that late event is dropped.
In this post we understood how watermarks help us to define bounded state and handle late events efficiently.
Complete Structure Streaming Lessons
- Lesson-1: Data Frame Abstraction - Spark Structured Streaming
- Lesson-2: Source and Sinks In Spark Structured Stream
- Lesson-3: Stateful WordCount - Spark Structured Stream
- Lesson-4: Stateful Aggregations - Spark Structured Stream
- Lesson-5: File Stream - Spark Structured Stream
- Lesson-6: Stream Enrichment using Static Data Join - Spark Structured Stream
- Lesson-7: Checkpointing State - Spark Structured Stream
- Lesson-8: Time Abstraction - Spark Structured Stream
- Lesson-9: Processing Time Window - Spark Structured Stream
- Lesson-10: Ingestion Time - Spark Structured Stream
- Lesson-11: Event Time - Spark Structured Stream
- Lesson-12: Watermarks - Spark Structured Stream
- Lesson-13: Session Windows Using Custom State - Spark Structured Stream