|
|
Hi All, I am getting the below error when I am trying to run the spark job on Kubernetes, I am running it in cluster mode. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-test] failed. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
I saw there was an JIRA opened already. SPARK-31786 I tried the parameter mentioned in JIRA too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work. Can anyone suggest what can be done? Kind Regards, Sachit Murarka
|
|
Hi All, I am getting the below error when I am trying to run the spark job on Kubernetes, I am running it in cluster mode. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-test] failed. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
I saw there was an JIRA opened already. SPARK-31786 I tried the parameter mentioned in JIRA too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work. Can anyone suggest what can be done? Kind Regards, Sachit Murarka
|
|
Hi Prashant
Thanks for the response!
I created the service account with the permissions and following is the command:
spark-submit --deploy-mode cluster --master k8s://http://ip:port --name "sachit" --conf spark.kubernetes.pyspark.pythonVersion=3 --conf spark.kubernetes.namespace=spark-test --conf spark.executor.instances=5 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa --conf spark.kubernetes.container.image=sparkpy local:///opt/spark/da/main.py
Kind Regards, Sachit Murarka On Mon, Jan 4, 2021 at 5:46 PM Prashant Sharma < [hidden email]> wrote:
Hi All, I am getting the below error when I am trying to run the spark job on Kubernetes, I am running it in cluster mode. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-test] failed. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
I saw there was an JIRA opened already. SPARK-31786 I tried the parameter mentioned in JIRA too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work. Can anyone suggest what can be done? Kind Regards, Sachit Murarka
|
|
Can you please paste the full exception trace, and mention spark and k8s version? Hi Prashant
Thanks for the response!
I created the service account with the permissions and following is the command:
spark-submit --deploy-mode cluster --master k8s://http://ip:port --name "sachit" --conf spark.kubernetes.pyspark.pythonVersion=3 --conf spark.kubernetes.namespace=spark-test --conf spark.executor.instances=5 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa --conf spark.kubernetes.container.image=sparkpy local:///opt/spark/da/main.py
Kind Regards, Sachit Murarka On Mon, Jan 4, 2021 at 5:46 PM Prashant Sharma < [hidden email]> wrote:
Hi All, I am getting the below error when I am trying to run the spark job on Kubernetes, I am running it in cluster mode. Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [spark-test] failed. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
I saw there was an JIRA opened already. SPARK-31786 I tried the parameter mentioned in JIRA too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work. Can anyone suggest what can be done? Kind Regards, Sachit Murarka
|
|