Error while running Spark on K8s

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Error while running Spark on K8s

Sachit Murarka
Hi All,
I am getting the below error when I am trying to run the spark job on Kubernetes, I am running it in cluster mode.

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-test] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) 


I saw there was an JIRA opened already. SPARK-31786
I tried the parameter mentioned in JIRA too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work.
Can anyone suggest what can be done? 

Kind Regards,
Sachit Murarka
Reply | Threaded
Open this post in threaded view
|

Re: Error while running Spark on K8s

Prashant Sharma
Hi Sachit,

Can you give more details on how did you run? i.e. spark submit command. My guess is, a service account with sufficient privilege is not provided. Please see: http://spark.apache.org/docs/latest/running-on-kubernetes.html#rbac

Thanks,

On Mon, Jan 4, 2021 at 5:27 PM Sachit Murarka <[hidden email]> wrote:
Hi All,
I am getting the below error when I am trying to run the spark job on Kubernetes, I am running it in cluster mode.

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-test] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) 


I saw there was an JIRA opened already. SPARK-31786
I tried the parameter mentioned in JIRA too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work.
Can anyone suggest what can be done? 

Kind Regards,
Sachit Murarka
Reply | Threaded
Open this post in threaded view
|

Re: Error while running Spark on K8s

Sachit Murarka
Hi Prashant

Thanks for the response!

I created the service account with the permissions  and following is the command:

spark-submit --deploy-mode cluster --master k8s://http://ip:port --name "sachit"   --conf spark.kubernetes.pyspark.pythonVersion=3  --conf spark.kubernetes.namespace=spark-test --conf spark.executor.instances=5 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa  --conf spark.kubernetes.container.image=sparkpy local:///opt/spark/da/main.py

Kind Regards,
Sachit Murarka


On Mon, Jan 4, 2021 at 5:46 PM Prashant Sharma <[hidden email]> wrote:
Hi Sachit,

Can you give more details on how did you run? i.e. spark submit command. My guess is, a service account with sufficient privilege is not provided. Please see: http://spark.apache.org/docs/latest/running-on-kubernetes.html#rbac

Thanks,

On Mon, Jan 4, 2021 at 5:27 PM Sachit Murarka <[hidden email]> wrote:
Hi All,
I am getting the below error when I am trying to run the spark job on Kubernetes, I am running it in cluster mode.

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-test] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) 


I saw there was an JIRA opened already. SPARK-31786
I tried the parameter mentioned in JIRA too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work.
Can anyone suggest what can be done? 

Kind Regards,
Sachit Murarka
Reply | Threaded
Open this post in threaded view
|

Re: Error while running Spark on K8s

Prashant Sharma
Can you please paste the full exception trace, and mention spark and k8s version?

On Mon, Jan 4, 2021 at 6:19 PM Sachit Murarka <[hidden email]> wrote:
Hi Prashant

Thanks for the response!

I created the service account with the permissions  and following is the command:

spark-submit --deploy-mode cluster --master k8s://http://ip:port --name "sachit"   --conf spark.kubernetes.pyspark.pythonVersion=3  --conf spark.kubernetes.namespace=spark-test --conf spark.executor.instances=5 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa  --conf spark.kubernetes.container.image=sparkpy local:///opt/spark/da/main.py

Kind Regards,
Sachit Murarka


On Mon, Jan 4, 2021 at 5:46 PM Prashant Sharma <[hidden email]> wrote:
Hi Sachit,

Can you give more details on how did you run? i.e. spark submit command. My guess is, a service account with sufficient privilege is not provided. Please see: http://spark.apache.org/docs/latest/running-on-kubernetes.html#rbac

Thanks,

On Mon, Jan 4, 2021 at 5:27 PM Sachit Murarka <[hidden email]> wrote:
Hi All,
I am getting the below error when I am trying to run the spark job on Kubernetes, I am running it in cluster mode.

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-test] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) 


I saw there was an JIRA opened already. SPARK-31786
I tried the parameter mentioned in JIRA too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work.
Can anyone suggest what can be done? 

Kind Regards,
Sachit Murarka