Application Upgrade - structured streaming

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Application Upgrade - structured streaming


When doing application upgrade for spark structured streaming, do we need to delete the checkpoint or does it start consuming offsets from the point we left?

kafka source we need to use the option "StartingOffsets" with a json string like

""" {"topicA":{"0":23,"1":-1},"topicB":{"0":-2}} """
Do we need to use as above when application is restarted by storing offsets at some place ? 


Sent from my iPhone