Restart Spark Streaming after deployment

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Restart Spark Streaming after deployment

khajaasmath786
Hi,

I am new in the usage of spark streaming. I have developed one spark streaming job which runs every 30 minutes with checkpointing directory. 

I have to implement minor change, shall I kill the spark streaming job once the batch is completed using yarn application -kill command and update the jar file?

Question I have is, if I follow the above approach will spark streaming picks up data from offset saved in checkpoint after restart?

is there any other better approaches you have. Thanks in advance for your suggestions.

Thanks,
Asmath
Reply | Threaded
Open this post in threaded view
|

Re: Restart Spark Streaming after deployment

Jacek Laskowski
Hi,

You're right...killing the spark streaming job is the way to go. If a batch was completed successfully, Spark Streaming will recover from the controlled failure and start where it left off. I don't think there's other way to do it.

Pozdrawiam,
Jacek Laskowski
----
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark

On Wed, Nov 15, 2017 at 5:18 PM, KhajaAsmath Mohammed <[hidden email]> wrote:
Hi,

I am new in the usage of spark streaming. I have developed one spark streaming job which runs every 30 minutes with checkpointing directory. 

I have to implement minor change, shall I kill the spark streaming job once the batch is completed using yarn application -kill command and update the jar file?

Question I have is, if I follow the above approach will spark streaming picks up data from offset saved in checkpoint after restart?

is there any other better approaches you have. Thanks in advance for your suggestions.

Thanks,
Asmath