Spark 3 pod template for the driver

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark 3 pod template for the driver

msumbul
Hi guys,

I try to use Spark 3 on top of Kubernetes and to specify a pod template for the driver.

Here is my pod manifest or the driver and when I do a spark-submit with the option:
--conf spark.kubernetes.driver.podTemplateFile=/data/k8s/podtemplate_driver3.yaml

I got the error message that I need to specify an image, but it's the manifest.
Does my manifest file is wrong, How should it look like?

Thanks for your help,
Michel

--------
The pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: mySpark3App
  labels:
    app: mySpark3App
    customlabel/app-id: "1"
spec:
  securityContext:
    runAsUser: 1000
  volumes:
    - name: "test-volume"
      emptyDir: {}
  containers:
    - name: spark3driver
      image: mydockerregistry.example.com/images/dev/spark3:latest
      instances: 1
      resources:
        requests:
          cpu: "1000m"
          memory: "512Mi"
        limits:
          cpu: "1000m"
          memory: "512Mi"
      volumeMounts:
       - name: "test-volume"
         mountPath: "/tmp"
Reply | Threaded
Open this post in threaded view
|

Re: Spark 3 pod template for the driver

Jorge Machado-2
Try to set spark.kubernetes.container.image

On 26. Jun 2020, at 14:58, Michel Sumbul <[hidden email]> wrote:

Hi guys,

I try to use Spark 3 on top of Kubernetes and to specify a pod template for the driver.

Here is my pod manifest or the driver and when I do a spark-submit with the option:
--conf spark.kubernetes.driver.podTemplateFile=/data/k8s/podtemplate_driver3.yaml

I got the error message that I need to specify an image, but it's the manifest.
Does my manifest file is wrong, How should it look like?

Thanks for your help,
Michel

--------
The pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: mySpark3App
  labels:
    app: mySpark3App
    customlabel/app-id: "1"
spec:
  securityContext:
    runAsUser: 1000
  volumes:
    - name: "test-volume"
      emptyDir: {}
  containers:
    - name: spark3driver
      image: mydockerregistry.example.com/images/dev/spark3:latest
      instances: 1
      resources:
        requests:
          cpu: "1000m"
          memory: "512Mi"
        limits:
          cpu: "1000m"
          memory: "512Mi"
      volumeMounts:
       - name: "test-volume"
         mountPath: "/tmp"

Reply | Threaded
Open this post in threaded view
|

Re: Spark 3 pod template for the driver

msumbul
Hi Jorge,
If I set that in the spark submit command it works but I want it only in the pod template file.

Best regards,
Michel

Le ven. 26 juin 2020 à 14:01, Jorge Machado <[hidden email]> a écrit :
Try to set spark.kubernetes.container.image

On 26. Jun 2020, at 14:58, Michel Sumbul <[hidden email]> wrote:

Hi guys,

I try to use Spark 3 on top of Kubernetes and to specify a pod template for the driver.

Here is my pod manifest or the driver and when I do a spark-submit with the option:
--conf spark.kubernetes.driver.podTemplateFile=/data/k8s/podtemplate_driver3.yaml

I got the error message that I need to specify an image, but it's the manifest.
Does my manifest file is wrong, How should it look like?

Thanks for your help,
Michel

--------
The pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: mySpark3App
  labels:
    app: mySpark3App
    customlabel/app-id: "1"
spec:
  securityContext:
    runAsUser: 1000
  volumes:
    - name: "test-volume"
      emptyDir: {}
  containers:
    - name: spark3driver
      image: mydockerregistry.example.com/images/dev/spark3:latest
      instances: 1
      resources:
        requests:
          cpu: "1000m"
          memory: "512Mi"
        limits:
          cpu: "1000m"
          memory: "512Mi"
      volumeMounts:
       - name: "test-volume"
         mountPath: "/tmp"