Hi All,
I have read about HPA Horizontal Pod Autoscaling(for pod scaling).
I understand it can be achieved by setting the request and limit for resources in yaml:
kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 --max=10 // example command.
But does Kubernetes actually work with Spark for HPA? Since 1 pod is used to launch 1 unique executor. Here in spark ideally pod can be scaled by dynamical allocation of executors(which in turn is a pod) instead of HPA. But Dynamic allocation is not supported as shuffle service is not there till Spark 3 release.
Could any one suggest how can I proceed achieving pod scaling in Spark?
Please note : I am using Kubernetes with Spark operator.
Kind Regards,
Sachit Murarka