In spark streaming application how to distinguish between normal and abnormal termination of application?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

In spark streaming application how to distinguish between normal and abnormal termination of application?

imax
Hi All,

I'm looking for a way to distinguish between normal and abnormal termination of a spark streaming application with (checkpointing enabled). 

Adding application listener doesn't really help because onApplicationEnd event has no information regarding the cause of the termination.

ssc_.sc.addSparkListener(new SparkListener {
override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit = {
I need to manage an internal metadata if streaming application has been terminated and that termination is not recoverable I have to delete the metadata (state of stream in this particular application).



--
Sincerely,

Igor Makhlin
Reply | Threaded
Open this post in threaded view
|

Re: In spark streaming application how to distinguish between normal and abnormal termination of application?

imax
looks like nobody knows the answer on this question ;)

On Sat, Mar 31, 2018 at 1:59 PM, Igor Makhlin <[hidden email]> wrote:
Hi All,

I'm looking for a way to distinguish between normal and abnormal termination of a spark streaming application with (checkpointing enabled). 

Adding application listener doesn't really help because onApplicationEnd event has no information regarding the cause of the termination.

ssc_.sc.addSparkListener(new SparkListener {
override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit = {
I need to manage an internal metadata if streaming application has been terminated and that termination is not recoverable I have to delete the metadata (state of stream in this particular application).



--
Sincerely,

Igor Makhlin



--
Sincerely,

Igor Makhlin