[Structured Streaming] Restarting streaming query on exception/termination

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Structured Streaming] Restarting streaming query on exception/termination

Priyank Shrivastava-2
What's the right way of programmatically restarting a structured streaming query which has terminated due to an exception? Example code or reference would be appreciated.

Could it be done from within the onQueryTerminated() event handler of StreamingQueryListener class?

Priyank

Reply | Threaded
Open this post in threaded view
|

Re: [Structured Streaming] Restarting streaming query on exception/termination

Priyank Shrivastava-2
Thanks for the reply formice.  I think that --supervise param helps to restart the whole spark application - what I want to be able to do is to only restart the structured streaming query which terminated due to error. Also, I am running my app in client mode.

Thanks,
Priyank

On Sun, Apr 22, 2018 at 8:52 PM, formice <[hidden email]> wrote:
standlone 
      add  config:(1)--deploy-mode cluster (2)--supervise
      example:  spark-submit  --master spark://master:7077 --deploy-mode cluster --supervise ......


------------------ 原始邮件 ------------------
发件人: "Priyank Shrivastava"<[hidden email]>;
发送时间: 2018年4月21日(星期六) 凌晨5:45
收件人: "user"<[hidden email]>;
主题: [Structured Streaming] Restarting streaming query on exception/termination

What's the right way of programmatically restarting a structured streaming query which has terminated due to an exception? Example code or reference would be appreciated.

Could it be done from within the onQueryTerminated() event handler of StreamingQueryListener class?

Priyank


Reply | Threaded
Open this post in threaded view
|

Re: [Structured Streaming] Restarting streaming query on exception/termination

Arun Mahadevan
I guess you can wait for the termination, catch exception and then restart the query in a loop. Something like…

while (true) {
  try {
    val query = df.writeStream().
                            …
                           .start()
    query.awaitTermination()
  } catch {
    case e: StreamingQueryException => // log it
  }
}

Thanks,
Arun

From: Priyank Shrivastava <[hidden email]>
Date: Monday, April 23, 2018 at 11:27 AM
To: formice <[hidden email]>, "[hidden email]" <[hidden email]>
Subject: Re: [Structured Streaming] Restarting streaming query on exception/termination

Thanks for the reply formice.  I think that --supervise param helps to restart the whole spark application - what I want to be able to do is to only restart the structured streaming query which terminated due to error. Also, I am running my app in client mode.

Thanks,
Priyank

On Sun, Apr 22, 2018 at 8:52 PM, formice <[hidden email]> wrote:
standlone 
      add  config:(1)--deploy-mode cluster (2)--supervise
      example:  spark-submit  --master spark://master:7077 --deploy-mode cluster --supervise ......


------------------ 原始邮件 ------------------
发件人: "Priyank Shrivastava"<[hidden email]>;
发送时间: 2018年4月21日(星期六) 凌晨5:45
收件人: "user"<[hidden email]>;
主题: [Structured Streaming] Restarting streaming query on exception/termination

What's the right way of programmatically restarting a structured streaming query which has terminated due to an exception? Example code or reference would be appreciated.

Could it be done from within the onQueryTerminated() event handler of StreamingQueryListener class?

Priyank