[SparkLauncher] stateChanged event not received in standalone cluster mode

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[SparkLauncher] stateChanged event not received in standalone cluster mode

bsikander
I have a client application which launches multiple jobs in Spark Cluster using SparkLauncher. I am using Standalone cluster mode. Launching jobs works fine till now. I use launcher.startApplication() to launch. 

But now, I have a requirement to check the states of my Driver process. I added a Listener implementing the SparkAppHandle.Listener but I don't get any events. I am following the approach mentioned here

I tried the same code with client code and I receive all the events as expected.

So, I am guessing that something different needs to be done in cluster mode. Is there any example with cluster mode?

Regards,
Behroz
Reply | Threaded
Open this post in threaded view
|

Re: [SparkLauncher] stateChanged event not received in standalone cluster mode

bsikander
Any help would be appreciated.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [SparkLauncher] stateChanged event not received in standalone cluster mode

Marcelo Vanzin
In reply to this post by bsikander
That feature has not been implemented yet.
https://issues.apache.org/jira/browse/SPARK-11033

On Wed, Jun 6, 2018 at 5:18 AM, Behroz Sikander <[hidden email]> wrote:

> I have a client application which launches multiple jobs in Spark Cluster
> using SparkLauncher. I am using Standalone cluster mode. Launching jobs
> works fine till now. I use launcher.startApplication() to launch.
>
> But now, I have a requirement to check the states of my Driver process. I
> added a Listener implementing the SparkAppHandle.Listener but I don't get
> any events. I am following the approach mentioned here
> https://www.linkedin.com/pulse/spark-launcher-amol-kale
>
> I tried the same code with client code and I receive all the events as
> expected.
>
> So, I am guessing that something different needs to be done in cluster mode.
> Is there any example with cluster mode?
>
> Regards,
> Behroz



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [SparkLauncher] stateChanged event not received in standalone cluster mode

bsikander
Thanks.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]