External shuffle service on K8S

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

External shuffle service on K8S

曹礼俊-2
Hi all:

Does Spark 2.3.2 supports external shuffle service on Kubernetes? 

I have looked up the documentation(https://spark.apache.org/docs/latest/running-on-kubernetes.html), but couldn't find related suggestions.

If suppports, how can I enable it?

Best Regards

Lijun Cao

 
Reply | Threaded
Open this post in threaded view
|

Re: External shuffle service on K8S

vincent gromakowski
No it's on the roadmap >2.4

Le ven. 26 oct. 2018 à 11:15, 曹礼俊 <[hidden email]> a écrit :
Hi all:

Does Spark 2.3.2 supports external shuffle service on Kubernetes? 

I have looked up the documentation(https://spark.apache.org/docs/latest/running-on-kubernetes.html), but couldn't find related suggestions.

If suppports, how can I enable it?

Best Regards

Lijun Cao

 
Reply | Threaded
Open this post in threaded view
|

Re: External shuffle service on K8S

Li Gao-3
There are existing 2.2 based ext shuffle on the fork:

You can modify it to suit your needs.

-Li


On Fri, Oct 26, 2018 at 3:22 AM vincent gromakowski <[hidden email]> wrote:
No it's on the roadmap >2.4

Le ven. 26 oct. 2018 à 11:15, 曹礼俊 <[hidden email]> a écrit :
Hi all:

Does Spark 2.3.2 supports external shuffle service on Kubernetes? 

I have looked up the documentation(https://spark.apache.org/docs/latest/running-on-kubernetes.html), but couldn't find related suggestions.

If suppports, how can I enable it?

Best Regards

Lijun Cao

 
Reply | Threaded
Open this post in threaded view
|

Re: External shuffle service on K8S

Matt Cheah

Hi there,

 

Please see https://issues.apache.org/jira/browse/SPARK-25299 for more discussion around this matter.

 

-Matt Cheah

 

From: Li Gao <[hidden email]>
Date: Friday, October 26, 2018 at 9:10 AM
To: "[hidden email]" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>, "[hidden email]" <[hidden email]>
Subject: Re: External shuffle service on K8S

 

There are existing 2.2 based ext shuffle on the fork:

 

You can modify it to suit your needs.

 

-Li

 

 

On Fri, Oct 26, 2018 at 3:22 AM vincent gromakowski <[hidden email]> wrote:

No it's on the roadmap >2.4

 

Le ven. 26 oct. 2018 à 11:15, 曹礼俊 <[hidden email]> a écrit :

Hi all:

 

Does Spark 2.3.2 supports external shuffle service on Kubernetes? 

 

I have looked up the documentation(https://spark.apache.org/docs/latest/running-on-kubernetes.html [spark.apache.org]), but couldn't find related suggestions.

 

If suppports, how can I enable it?

 

Best Regards

 

Lijun Cao

 

 


smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: External shuffle service on K8S

Garlapati, Suryanarayana (Nokia - IN/Bangalore)

Hi,

There is an unmerged PR which can be used against spark 2.4(if you are interested) or master branch(3.0). Spark 2.3 K8S lacks lot of features. I suggest you upgrade to 2.4 which will be released in few days from now.

 

https://github.com/apache/spark/pull/22722

 

Regards

Surya

 

From: Matt Cheah <[hidden email]>
Sent: Saturday, October 27, 2018 6:12 AM
To: Li Gao <[hidden email]>; [hidden email]
Cc: [hidden email]; [hidden email]
Subject: Re: External shuffle service on K8S

 

Hi there,

 

Please see https://issues.apache.org/jira/browse/SPARK-25299 for more discussion around this matter.

 

-Matt Cheah

 

From: Li Gao <[hidden email]>
Date: Friday, October 26, 2018 at 9:10 AM
To: "[hidden email]" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>, "[hidden email]" <[hidden email]>
Subject: Re: External shuffle service on K8S

 

There are existing 2.2 based ext shuffle on the fork:

 

You can modify it to suit your needs.

 

-Li

 

 

On Fri, Oct 26, 2018 at 3:22 AM vincent gromakowski <[hidden email]> wrote:

No it's on the roadmap >2.4

 

Le ven. 26 oct. 2018 à 11:15, 曹礼俊 <[hidden email]> a écrit :

Hi all:

 

Does Spark 2.3.2 supports external shuffle service on Kubernetes? 

 

I have looked up the documentation(https://spark.apache.org/docs/latest/running-on-kubernetes.html [spark.apache.org]), but couldn't find related suggestions.

 

If suppports, how can I enable it?

 

Best Regards

 

Lijun Cao