HPA - Kubernetes for Spark

[Sachit Murarka]

Hi All,

I have read about HPA  Horizontal Pod Autoscaling(for pod scaling).

I understand it can be achieved by setting the request and limit for resources in yaml:

kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 --max=10  // example command.

But does Kubernetes actually work with Spark for HPA? Since 1 pod is used to launch 1 unique executor. Here in spark ideally pod can be scaled by dynamical allocation of executors(which in turn is a pod) instead of HPA. But Dynamic allocation is not supported as shuffle service is not there till Spark 3 release.

Could any one suggest how can I proceed achieving pod scaling in Spark?

Please note : I am using Kubernetes with Spark operator.

[Sandish Kumar HN]

Sachit,

K8S based spark dynamic allocation is only available on Spark 3.0.X+ and that too without External Shuffling Service.

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#dynamic-allocation
http://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work

On Sun, 10 Jan 2021 at 13:23, Sachit Murarka <[hidden email]> wrote:

Hi All,

I have read about HPA  Horizontal Pod Autoscaling(for pod scaling).

I understand it can be achieved by setting the request and limit for resources in yaml:

kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 --max=10  // example command.

But does Kubernetes actually work with Spark for HPA? Since 1 pod is used to launch 1 unique executor. Here in spark ideally pod can be scaled by dynamical allocation of executors(which in turn is a pod) instead of HPA. But Dynamic allocation is not supported as shuffle service is not there till Spark 3 release.

Could any one suggest how can I proceed achieving pod scaling in Spark?

Please note : I am using Kubernetes with Spark operator.

Kind Regards,
Sachit Murarka

--

[Sachit Murarka]

Hi ,

Yes I know  by setting shuffle tracking property enabled we can use DRA. But , it is marked as experimental. Is it advised to use ?

Also , regarding HPA. We do not have HPA differently as such for Spark. Right?

On Mon, Jan 11, 2021 at 2:17 AM Sandish Kumar HN <[hidden email]> wrote:

Sachit,

K8S based spark dynamic allocation is only available on Spark 3.0.X+ and that too without External Shuffling Service.

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#dynamic-allocation
http://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work

On Sun, 10 Jan 2021 at 13:23, Sachit Murarka <[hidden email]> wrote:

Hi All,

I have read about HPA  Horizontal Pod Autoscaling(for pod scaling).

I understand it can be achieved by setting the request and limit for resources in yaml:

kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 --max=10  // example command.

But does Kubernetes actually work with Spark for HPA? Since 1 pod is used to launch 1 unique executor. Here in spark ideally pod can be scaled by dynamical allocation of executors(which in turn is a pod) instead of HPA. But Dynamic allocation is not supported as shuffle service is not there till Spark 3 release.

Could any one suggest how can I proceed achieving pod scaling in Spark?

Please note : I am using Kubernetes with Spark operator.

Kind Regards,
Sachit Murarka

--

Thanks,

Regards,

SandishKumar HN