HPA - Kubernetes for Spark

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

HPA - Kubernetes for Spark

Sachit Murarka
Hi All,

I have read about HPA  Horizontal Pod Autoscaling(for pod scaling).

I understand it can be achieved by setting the request and limit for resources in yaml:
kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 --max=10  // example command.

But does Kubernetes actually work with Spark for HPA? Since 1 pod is used to launch 1 unique executor. Here in spark ideally pod can be scaled by dynamical allocation of executors(which in turn is a pod) instead of HPA. But Dynamic allocation is not supported as shuffle service is not there till Spark 3 release.

Could any one suggest how can I proceed achieving pod scaling in Spark?

Please note : I am using Kubernetes with Spark operator.


Kind Regards,
Sachit Murarka
Reply | Threaded
Open this post in threaded view
|

Re: HPA - Kubernetes for Spark

Sandish Kumar HN
Sachit,

K8S based spark dynamic allocation is only available on Spark 3.0.X+ and that too without External Shuffling Service.

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#dynamic-allocation
http://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work

On Sun, 10 Jan 2021 at 13:23, Sachit Murarka <[hidden email]> wrote:
Hi All,

I have read about HPA  Horizontal Pod Autoscaling(for pod scaling).

I understand it can be achieved by setting the request and limit for resources in yaml:
kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 --max=10  // example command.

But does Kubernetes actually work with Spark for HPA? Since 1 pod is used to launch 1 unique executor. Here in spark ideally pod can be scaled by dynamical allocation of executors(which in turn is a pod) instead of HPA. But Dynamic allocation is not supported as shuffle service is not there till Spark 3 release.

Could any one suggest how can I proceed achieving pod scaling in Spark?

Please note : I am using Kubernetes with Spark operator.


Kind Regards,
Sachit Murarka


--

Thanks,
Regards,
SandishKumar HN
Reply | Threaded
Open this post in threaded view
|

Re: HPA - Kubernetes for Spark

Sachit Murarka
Hi ,

Yes I know  by setting shuffle tracking property enabled we can use DRA. But , it is marked as experimental. Is it advised to use ?

Also , regarding HPA. We do not have HPA differently as such for Spark. Right?

Kind Regards,
Sachit Murarka


On Mon, Jan 11, 2021 at 2:17 AM Sandish Kumar HN <[hidden email]> wrote:
Sachit,

K8S based spark dynamic allocation is only available on Spark 3.0.X+ and that too without External Shuffling Service.

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#dynamic-allocation
http://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work

On Sun, 10 Jan 2021 at 13:23, Sachit Murarka <[hidden email]> wrote:
Hi All,

I have read about HPA  Horizontal Pod Autoscaling(for pod scaling).

I understand it can be achieved by setting the request and limit for resources in yaml:
kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 --max=10  // example command.

But does Kubernetes actually work with Spark for HPA? Since 1 pod is used to launch 1 unique executor. Here in spark ideally pod can be scaled by dynamical allocation of executors(which in turn is a pod) instead of HPA. But Dynamic allocation is not supported as shuffle service is not there till Spark 3 release.

Could any one suggest how can I proceed achieving pod scaling in Spark?

Please note : I am using Kubernetes with Spark operator.


Kind Regards,
Sachit Murarka


--

Thanks,
Regards,
SandishKumar HN