Spark on kubernetes : missing spark.kubernetes.driver.request.cores parameter ?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark on kubernetes : missing spark.kubernetes.driver.request.cores parameter ?

jcdauchy
Hello all,

I am surprised that it is not possible to define
"spark.kubernetes.driver.request.cores" when submitting a spark job on
kubernetes.

My understanding is that it would limit the CPU requests for the driver on
the k8s cluster and we could still define how many cores (threads) we use in
the driver with spark.driver.cores.

In the 2.4 branch, the request is then defined by "spark.driver.cores", so
we cannot throttle the driver.

Is my understanding wrong or I missed something ?

Thanks
Christophe



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark on kubernetes : missing spark.kubernetes.driver.request.cores parameter ?

jcdauchy
I am actually answering myself as I have check on master 3.x branch, and
there is this feature !

https://issues.apache.org/jira/browse/SPARK-27754

So my understanding was correct.




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]