Upgrades of streaming jobs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgrades of streaming jobs

geoHeil
Hi

What is the state of spark structured streaming jobs and upgrades?

Can checkpoints of version 1 be read by version 2 of a job? Is downtime required to upgrade the job?

Thanks 
Reply | Threaded
Open this post in threaded view
|

Re: Upgrades of streaming jobs

Tathagata Das
Yes, all checkpoints are forward compatible. 

However, you do need to restart the query if you want to update the code of the query. This downtime can be in less than a second (if you just restart the query without stopping the application/Spark driver) or 10s of seconds  (if you have to stop the application and resubmit your application to the cluster).



On Thu, Mar 8, 2018 at 12:11 PM, Georg Heiler <[hidden email]> wrote:
Hi

What is the state of spark structured streaming jobs and upgrades?

Can checkpoints of version 1 be read by version 2 of a job? Is downtime required to upgrade the job?

Thanks