Apache Spark - Structured Streaming graceful shutdown

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache Spark - Structured Streaming graceful shutdown

M Singh
Hi:
Are there any patterns/recommendations for gracefully stopping a structured streaming application ?
Thanks


Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark - Structured Streaming graceful shutdown

Diogo Munaro Vieira
Hi M Singh! Here I'm using query.stop()

Em 25 de dez de 2017 19:19, "M Singh" <[hidden email]> escreveu:
Hi:
Are there any patterns/recommendations for gracefully stopping a structured streaming application ?
Thanks


Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark - Structured Streaming graceful shutdown

M Singh
Thanks Diogo.  My question is how to gracefully call the stop method while the streaming application is running in a cluster.




On Monday, December 25, 2017 5:39 PM, Diogo Munaro Vieira <[hidden email]> wrote:


Hi M Singh! Here I'm using query.stop()

Em 25 de dez de 2017 19:19, "M Singh" <[hidden email]> escreveu:
Hi:
Are there any patterns/recommendations for gracefully stopping a structured streaming application ?
Thanks




Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark - Structured Streaming graceful shutdown

Eyal Zituny
Hi
if you're interested in stopping you're spark application externally, you will probably need a way to communicate with the spark driver  (which start and holds a ref to the spark context)
this can be done by adding some code to the driver app, for example:
  • you can expose a rest api that stop the query and the spark context
  • if running in client mode you can listen to stdin
  • you can also listen to an external system (like kafka)
Eyal

On Tue, Dec 26, 2017 at 10:37 PM, M Singh <[hidden email]> wrote:
Thanks Diogo.  My question is how to gracefully call the stop method while the streaming application is running in a cluster.




On Monday, December 25, 2017 5:39 PM, Diogo Munaro Vieira <[hidden email]> wrote:


Hi M Singh! Here I'm using query.stop()

Em 25 de dez de 2017 19:19, "M Singh" <[hidden email]> escreveu:
Hi:
Are there any patterns/recommendations for gracefully stopping a structured streaming application ?
Thanks





Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark - Structured Streaming graceful shutdown

M Singh
Thanks Eyal - it appears that these are the same patterns used for spark DStreams.


On Wednesday, December 27, 2017 1:15 AM, Eyal Zituny <[hidden email]> wrote:


Hi
if you're interested in stopping you're spark application externally, you will probably need a way to communicate with the spark driver  (which start and holds a ref to the spark context)
this can be done by adding some code to the driver app, for example:
  • you can expose a rest api that stop the query and the spark context
  • if running in client mode you can listen to stdin
  • you can also listen to an external system (like kafka)
Eyal

On Tue, Dec 26, 2017 at 10:37 PM, M Singh <[hidden email]> wrote:
Thanks Diogo.  My question is how to gracefully call the stop method while the streaming application is running in a cluster.




On Monday, December 25, 2017 5:39 PM, Diogo Munaro Vieira <[hidden email]> wrote:


Hi M Singh! Here I'm using query.stop()

Em 25 de dez de 2017 19:19, "M Singh" <[hidden email]> escreveu:
Hi:
Are there any patterns/recommendations for gracefully stopping a structured streaming application ?
Thanks