dynamic executor scalling spark on kubernetes client mode

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

dynamic executor scalling spark on kubernetes client mode

Pradeepta Choudhury
Hiii ,

The dynamic executor scalling is working fine for spark on kubernetes (latest from spark master repository ) in cluster mode . is the dynamic executor scalling available for client mode ? if yes where can i find the usage doc for same .
If no is there any PR open for this ?

Thanks ,
Pradeepta
Reply | Threaded
Open this post in threaded view
|

Re: dynamic executor scalling spark on kubernetes client mode

stevenstetzler
Hi all,

I am interested in this as well. My use-case could benefit from dynamic executor scaling but we are restricted to using client mode since we are only using Spark shells.

Could anyone help me understand the barriers to getting dynamic executor scaling to work in client mode on Kubernetes?

Thanks,
Steven

On Sat, May 9, 2020 at 9:48 AM Pradeepta Choudhury <[hidden email]> wrote:
Hiii ,

The dynamic executor scalling is working fine for spark on kubernetes (latest from spark master repository ) in cluster mode . is the dynamic executor scalling available for client mode ? if yes where can i find the usage doc for same .
If no is there any PR open for this ?

Thanks ,
Pradeepta
Reply | Threaded
Open this post in threaded view
|

Re: dynamic executor scalling spark on kubernetes client mode

Roland Johann
Hi all,

don’t want to interrupt the conversation but are keen where I can find information regarding dynamic allocation on kubernetes. As far as I know the docs just point to future work.

Thanks a lot,
Roland



Am 12.05.2020 um 09:25 schrieb Steven Stetzler <[hidden email]>:

Hi all,

I am interested in this as well. My use-case could benefit from dynamic executor scaling but we are restricted to using client mode since we are only using Spark shells.

Could anyone help me understand the barriers to getting dynamic executor scaling to work in client mode on Kubernetes?

Thanks,
Steven

On Sat, May 9, 2020 at 9:48 AM Pradeepta Choudhury <[hidden email]> wrote:
Hiii ,

The dynamic executor scalling is working fine for spark on kubernetes (latest from spark master repository ) in cluster mode . is the dynamic executor scalling available for client mode ? if yes where can i find the usage doc for same .
If no is there any PR open for this ?

Thanks ,
Pradeepta

Reply | Threaded
Open this post in threaded view
|

Re: dynamic executor scalling spark on kubernetes client mode

Pradeepta Choudhury
Hey guys i was able to run dynamic scaling in both cluster and client mode . would document and send it over this weekend

On Tue 12 May, 2020, 1:26 PM Roland Johann, <[hidden email]> wrote:
Hi all,

don’t want to interrupt the conversation but are keen where I can find information regarding dynamic allocation on kubernetes. As far as I know the docs just point to future work.

Thanks a lot,
Roland



Am 12.05.2020 um 09:25 schrieb Steven Stetzler <[hidden email]>:

Hi all,

I am interested in this as well. My use-case could benefit from dynamic executor scaling but we are restricted to using client mode since we are only using Spark shells.

Could anyone help me understand the barriers to getting dynamic executor scaling to work in client mode on Kubernetes?

Thanks,
Steven

On Sat, May 9, 2020 at 9:48 AM Pradeepta Choudhury <[hidden email]> wrote:
Hiii ,

The dynamic executor scalling is working fine for spark on kubernetes (latest from spark master repository ) in cluster mode . is the dynamic executor scalling available for client mode ? if yes where can i find the usage doc for same .
If no is there any PR open for this ?

Thanks ,
Pradeepta

Reply | Threaded
Open this post in threaded view
|

Re: dynamic executor scalling spark on kubernetes client mode

stevenstetzler
Oh, thanks for mentioning that, it looks l dynamic allocation on Kubernetes works in client mode in Spark 3.0.0. I just had to set the following configurations:

spark.dynamicAllocation.enabled=true 
spark.dynamicAllocation.shuffleTracking.enabled=true 
 
to enable dynamic allocation and disable the need for the external shuffle service (which looks like it is experimental right now). My executor pods couldn't connect to the  external shuffle service when it was enabled. This seems to be working okay for me.

Thanks,
Steven
 

On Tue, May 12, 2020 at 4:42 AM Pradeepta Choudhury <[hidden email]> wrote:
Hey guys i was able to run dynamic scaling in both cluster and client mode . would document and send it over this weekend

On Tue 12 May, 2020, 1:26 PM Roland Johann, <[hidden email]> wrote:
Hi all,

don’t want to interrupt the conversation but are keen where I can find information regarding dynamic allocation on kubernetes. As far as I know the docs just point to future work.

Thanks a lot,
Roland



Am 12.05.2020 um 09:25 schrieb Steven Stetzler <[hidden email]>:

Hi all,

I am interested in this as well. My use-case could benefit from dynamic executor scaling but we are restricted to using client mode since we are only using Spark shells.

Could anyone help me understand the barriers to getting dynamic executor scaling to work in client mode on Kubernetes?

Thanks,
Steven

On Sat, May 9, 2020 at 9:48 AM Pradeepta Choudhury <[hidden email]> wrote:
Hiii ,

The dynamic executor scalling is working fine for spark on kubernetes (latest from spark master repository ) in cluster mode . is the dynamic executor scalling available for client mode ? if yes where can i find the usage doc for same .
If no is there any PR open for this ?

Thanks ,
Pradeepta