The last successful batch before stop re-execute after restart the DStreams with checkpoint

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

The last successful batch before stop re-execute after restart the DStreams with checkpoint

Terry Hoo
Experts,

I see the last batch before stop (graceful shutdown) always re-execute after restart the DStream from a checkpoint, is this a expected behavior? 

I see a bug in JIRA: https://issues.apache.org/jira/browse/SPARK-20050, whic reports duplicates on Kafka, I also see this with HDFS file. 

Regards
- Terry