[k8s] PersistentVolumeClaim support in 3.1.1 on minikube

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[k8s] PersistentVolumeClaim support in 3.1.1 on minikube

Jacek Laskowski
Hi,

I've been toying with persistent volumes using Spark 3.1.1 on minikube and am wondering whether it's a supported platform. I'd not be surprised if not given all the surprises I've been experiencing lately.

Can I use spark-shell or any Spark app in client mode with PVCs with the default 2 executors? Should the following work if I removed --num-executors 1?

./bin/spark-shell \
  --master k8s://$K8S_SERVER \
  --num-executors 1 \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.path=$MOUNT_PATH \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.claimName=$PVC_CLAIM_NAME \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.storageClass=$PVC_STORAGE_CLASS \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.sizeLimit=$PVC_SIZE_LIMIT \
  --conf spark.kubernetes.container.image=$IMAGE_NAME \
  --conf spark.kubernetes.context=minikube \
  --conf spark.kubernetes.namespace=spark-demo \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --verbose


Please help. Thank you!

Reply | Threaded
Open this post in threaded view
|

Re: [k8s] PersistentVolumeClaim support in 3.1.1 on minikube

Jacek Laskowski
Hi,

I think I found it. I should be using OnDemand claim name so it gets replaced to be unique per executor (?)

On Mon, Mar 15, 2021 at 8:36 PM Jacek Laskowski <[hidden email]> wrote:
Hi,

I've been toying with persistent volumes using Spark 3.1.1 on minikube and am wondering whether it's a supported platform. I'd not be surprised if not given all the surprises I've been experiencing lately.

Can I use spark-shell or any Spark app in client mode with PVCs with the default 2 executors? Should the following work if I removed --num-executors 1?

./bin/spark-shell \
  --master k8s://$K8S_SERVER \
  --num-executors 1 \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.path=$MOUNT_PATH \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.claimName=$PVC_CLAIM_NAME \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.storageClass=$PVC_STORAGE_CLASS \
  --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.sizeLimit=$PVC_SIZE_LIMIT \
  --conf spark.kubernetes.container.image=$IMAGE_NAME \
  --conf spark.kubernetes.context=minikube \
  --conf spark.kubernetes.namespace=spark-demo \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --verbose


Please help. Thank you!