How to know that a partition is ready when using Structured Streaming

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How to know that a partition is ready when using Structured Streaming

Wayne Guo
When using structured streaming, we use "partitionBy" api  to partition the
output data, and use the watermark based on event-time to handle delay
records, but how to tell downstream users  that a partition is ready? For
example, when to write an empty "hadoop.done" file in a paritition
directory?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]