Problem running Spark on Kubernetes: Certificate error

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem running Spark on Kubernetes: Certificate error

stevenstetzler
Hello, 

I am following the tutorial here (https://spark.apache.org/docs/latest/running-on-kubernetes.html) to get spark running on a Kubernetes cluster. My Kubernetes cluster is hosted with Digital Ocean's kubernetes cluster manager. I have change the KUBECONFIG environment variable to point to my cluster access credentials, so both Spark and kubectl can speak with the nodes. 

I am running into an issue when trying to run the SparkPi example as described in the Spark on Kubernetes tutorials. The command I am running is: 

./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=1 --conf spark.kubernetes.container.image=$IMAGEURL --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar 

where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the URL of the Spark docker image I am using (https://hub.docker.com/r/stevenstetzler/spark/). This docker image was built and pushed with the script included in the Spark 2.4 distribution. I have created a service account for Spark to ensure that it has proper permissions to create pods etc., which I checked using 

kubectl auth can-i create pods --as=system:serviceaccount:default:spark 

When I try to run the SparkPi example using the above command, I get the following output: 

2018-12-12 06:26:15 WARN  Utils:66 - Your hostname, docker-test resolves to a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface eth0) 
2018-12-12 06:26:15 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: N/A 
         start time: N/A 
         container images: N/A 
         phase: Pending 
         status: [] 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: N/A 
         container images: N/A 
         phase: Pending 
         status: [] 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: docker.io/stevenstetzler/spark:v1 
         phase: Pending 
         status: [ContainerStatus(containerID=null, image=docker.io/stevenstetzler/spark:v1, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:19 INFO  Client:54 - Waiting for application spark-pi to finish... 
2018-12-12 06:26:21 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: stevenstetzler/spark:v1 
         phase: Running 
         status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: stevenstetzler/spark:v1 
         phase: Failed 
         status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z, additionalProperties={}), message=null, reason=Error, signal=null, startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - Container final statuses: 


         Container name: spark-kubernetes-driver 
         Container image: stevenstetzler/spark:v1 
         Container state: Terminated 
         Exit code: 1 
2018-12-12 06:26:24 INFO  Client:54 - Application spark-pi finished. 
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Shutdown hook called 
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd 

When I check the logs with 

kubectl logs spark-pi-1544595975520-driver 

I get the following: 

++ id -u 
+ myuid=0 
++ id -g 
+ mygid=0 
+ set +e 
++ getent passwd 0 
+ uidentry=root:x:0:0:root:/root:/bin/ash 
+ set -e 
+ '[' -z root:x:0:0:root:/root:/bin/ash ']' 
+ SPARK_K8S_CMD=driver 
+ case "$SPARK_K8S_CMD" in 
+ shift 1 
+ SPARK_CLASSPATH=':/opt/spark/jars/*' 
+ env 
+ grep SPARK_JAVA_OPT_ 
+ sed 's/[^=]*=\(.*\)/\1/g' 
+ sort -t_ -k4 -n 
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS 
+ '[' -n '' ']' 
+ '[' -n '' ']' 
+ PYSPARK_ARGS= 
+ '[' -n '' ']' 
+ R_ARGS= 
+ '[' -n '' ']' 
+ '[' '' == 2 ']' 
+ '[' '' == 3 ']' 
+ case "$SPARK_K8S_CMD" in 
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@") 
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi spark-internal 
2018-12-12 06:26:22 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
2018-12-12 06:26:22 INFO  SparkContext:54 - Running Spark version 2.4.0 
2018-12-12 06:26:22 INFO  SparkContext:54 - Submitted application: Spark Pi 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls to: root 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls to: root 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls groups to: 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls groups to: 
2018-12-12 06:26:22 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set() 
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 7078. 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering MapOutputTracker 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering BlockManagerMaster 
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up 
2018-12-12 06:26:23 INFO  DiskBlockManager:54 - Created local directory at /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf 
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering OutputCommitCoordinator 
2018-12-12 06:26:23 INFO  log:192 - Logging initialized @2059ms 
2018-12-12 06:26:23 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown 
2018-12-12 06:26:23 INFO  Server:419 - Started @2155ms 
2018-12-12 06:26:23 INFO  AbstractConnector:278 - Started ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040. 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6b410923{/jobs/job/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@203dd56b{/stages/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e8fadb0{/stages/stage,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1d3e6d34{/stages/stage/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6eafb10e{/stages/pool,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@26a94fa5{/stages/pool/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2873d672{/storage/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3bc735b3{/storage/rdd,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@577f9109{/storage/rdd/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4303b7f0{/environment,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@757529a4{/environment/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c41d037{/executors/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2234078{/executors/threadDump,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ec77191{/executors/threadDump/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@534ca02b{/jobs/job/kill,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29a23c3d{/stages/stage/kill,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://spark-pi-1544595975520-driver-svc.default.svc:4040
2018-12-12 06:26:23 INFO  SparkContext:54 - Added JAR file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar with timestamp 1544595983645 
2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing SparkContext. 
org.apache.spark.SparkException: External scheduler cannot be instantiated 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) 
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) 
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) 
        at scala.Option.getOrElse(Option.scala:121) 
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) 
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) 
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:498) 
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) 
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) 
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) 
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) 
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) 
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) 
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167) 
        at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) 
        at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64) 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788) 
        ... 20 more 
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) 
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) 
        ... 23 more 
Caused by: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) 
        ... 29 more 
2018-12-12 06:26:23 INFO  AbstractConnector:318 - Stopped Spark@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040
2018-12-12 06:26:23 INFO  SparkUI:54 - Stopped Spark web UI at http://spark-pi-1544595975520-driver-svc.default.svc:4040
2018-12-12 06:26:23 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped! 
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore cleared 
2018-12-12 06:26:23 INFO  BlockManager:54 - BlockManager stopped 
2018-12-12 06:26:23 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped 
2018-12-12 06:26:23 WARN  MetricsSystem:66 - Stopping a MetricsSystem that is not running 
2018-12-12 06:26:23 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped! 
2018-12-12 06:26:23 INFO  SparkContext:54 - Successfully stopped SparkContext 
Exception in thread "main" org.apache.spark.SparkException: External scheduler cannot be instantiated 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) 
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) 
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) 
        at scala.Option.getOrElse(Option.scala:121) 
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) 
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) 
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:498) 
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) 
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) 
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) 
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) 
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) 
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) 
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167) 
        at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) 
        at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64) 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788) 
        ... 20 more 
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) 
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) 
        ... 23 more 
Caused by: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) 
        ... 29 more 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Shutdown hook called 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa 

It looks to me like there is a communication / authentication issue during pod creation, and it looks like there might be a certificate missing that Spark needs to communicate with Kubernetes, which I infer from the "java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input" error message. 

Any ideas on what I can do to resolve this? Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Problem running Spark on Kubernetes: Certificate error

Matt Cheah

Hi Steven,

 

What I think is happening is that your machine has a CA certificate that is used for communicating with your API server, particularly because you’re using Digital Ocean’s cluster manager. However, it’s unclear if your pod has the same CA certificate or if the pod needs that certificate file. You can use configurations to have your pod use a particular CA certificate file to communicate with the APi server. If you set spark.kubernetes.authenticate.driver.caCertFile to the path of your CA certificate on your local disk, spark-submit will create a secret that contains that certificate file and use that certificate to configure TLS for the driver pod’s communication with the API server.

 

It's clear that your driver pod doesn’t have the right TLS certificate to communicate with the API server, so I would try to introspect the driver pod and see what certificate it’s using for that communication. If there’s a fix that needs to happen in Spark, feel free to indicate as such.

 

-Matt Cheah

 

From: Steven Stetzler <[hidden email]>
Date: Thursday, December 13, 2018 at 1:49 PM
To: "[hidden email]" <[hidden email]>
Subject: Problem running Spark on Kubernetes: Certificate error

 

Hello, 

I am following the tutorial here (
https://spark.apache.org/docs/latest/running-on-kubernetes.html [spark.apache.org]) to get spark running on a Kubernetes cluster. My Kubernetes cluster is hosted with Digital Ocean's kubernetes cluster manager. I have change the KUBECONFIG environment variable to point to my cluster access credentials, so both Spark and kubectl can speak with the nodes. 

I am running into an issue when trying to run the SparkPi example as described in the Spark on Kubernetes tutorials. The command I am running is: 

./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=1 --conf spark.kubernetes.container.image=$IMAGEURL --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar 

where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the URL of the Spark docker image I am using (
https://hub.docker.com/r/stevenstetzler/spark/ [hub.docker.com]). This docker image was built and pushed with the script included in the Spark 2.4 distribution. I have created a service account for Spark to ensure that it has proper permissions to create pods etc., which I checked using 

kubectl auth can-i create pods --as=system:serviceaccount:default:spark 

When I try to run the SparkPi example using the above command, I get the following output: 

2018-12-12 06:26:15 WARN  Utils:66 - Your hostname, docker-test resolves to a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface eth0) 
2018-12-12 06:26:15 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: N/A 
         start time: N/A 
         container images: N/A 
         phase: Pending 
         status: [] 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: N/A 
         container images: N/A 
         phase: Pending 
         status: [] 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: docker.io/stevenstetzler/spark:v1 [docker.io] 
         phase: Pending 
         status: [ContainerStatus(containerID=null, image=docker.io/stevenstetzler/spark:v1 [docker.io], imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:19 INFO  Client:54 - Waiting for application spark-pi to finish... 
2018-12-12 06:26:21 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: stevenstetzler/spark:v1 
         phase: Running 
         status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: stevenstetzler/spark:v1 
         phase: Failed 
         status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z, additionalProperties={}), message=null, reason=Error, signal=null, startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - Container final statuses: 


         Container name: spark-kubernetes-driver 
         Container image: stevenstetzler/spark:v1 
         Container state: Terminated 
         Exit code: 1 
2018-12-12 06:26:24 INFO  Client:54 - Application spark-pi finished. 
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Shutdown hook called 
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd 

When I check the logs with 

kubectl logs spark-pi-1544595975520-driver 

I get the following: 

++ id -u 
+ myuid=0 
++ id -g 
+ mygid=0 
+ set +e 
++ getent passwd 0 
+ uidentry=root:x:0:0:root:/root:/bin/ash 
+ set -e 
+ '[' -z root:x:0:0:root:/root:/bin/ash ']' 
+ SPARK_K8S_CMD=driver 
+ case "$SPARK_K8S_CMD" in 
+ shift 1 
+ SPARK_CLASSPATH=':/opt/spark/jars/*' 
+ env 
+ grep SPARK_JAVA_OPT_ 
+ sed 's/[^=]*=\(.*\)/\1/g' 
+ sort -t_ -k4 -n 
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS 
+ '[' -n '' ']' 
+ '[' -n '' ']' 
+ PYSPARK_ARGS= 
+ '[' -n '' ']' 
+ R_ARGS= 
+ '[' -n '' ']' 
+ '[' '' == 2 ']' 
+ '[' '' == 3 ']' 
+ case "$SPARK_K8S_CMD" in 
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@") 
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi spark-internal 
2018-12-12 06:26:22 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
2018-12-12 06:26:22 INFO  SparkContext:54 - Running Spark version 2.4.0 
2018-12-12 06:26:22 INFO  SparkContext:54 - Submitted application: Spark Pi 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls to: root 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls to: root 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls groups to: 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls groups to: 
2018-12-12 06:26:22 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set() 
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 7078. 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering MapOutputTracker 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering BlockManagerMaster 
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up 
2018-12-12 06:26:23 INFO  DiskBlockManager:54 - Created local directory at /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf 
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering OutputCommitCoordinator 
2018-12-12 06:26:23 INFO  log:192 - Logging initialized @2059ms 
2018-12-12 06:26:23 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown 
2018-12-12 06:26:23 INFO  Server:419 - Started @2155ms 
2018-12-12 06:26:23 INFO  AbstractConnector:278 - Started ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040. 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6b410923{/jobs/job/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@203dd56b{/stages/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e8fadb0{/stages/stage,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1d3e6d34{/stages/stage/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6eafb10e{/stages/pool,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@26a94fa5{/stages/pool/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2873d672{/storage/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3bc735b3{/storage/rdd,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@577f9109{/storage/rdd/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4303b7f0{/environment,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@757529a4{/environment/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c41d037{/executors/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2234078{/executors/threadDump,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ec77191{/executors/threadDump/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@534ca02b{/jobs/job/kill,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29a23c3d{/stages/stage/kill,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at 
http://spark-pi-1544595975520-driver-svc.default.svc:4040 [spark-pi-1544595975520-driver-svc.default.svc]
2018-12-12 06:26:23 INFO  SparkContext:54 - Added JAR file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar with timestamp 1544595983645 
2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing SparkContext. 
org.apache.spark.SparkException: External scheduler cannot be instantiated 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) 
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) 
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) 
        at scala.Option.getOrElse(Option.scala:121) 
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) 
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) 
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:498) 
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
        at org.apache.spark.deploy.SparkSubmit.org [org.apache.spark.deploy.sparksubmit.org]$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) 
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) 
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) 
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) 
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) 
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) 
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167) 
        at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) 
        at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64) 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788) 
        ... 20 more 
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) 
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) 
        ... 23 more 
Caused by: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) 
        ... 29 more 
2018-12-12 06:26:23 INFO  AbstractConnector:318 - Stopped Spark@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]
2018-12-12 06:26:23 INFO  SparkUI:54 - Stopped Spark web UI at 
http://spark-pi-1544595975520-driver-svc.default.svc:4040 [spark-pi-1544595975520-driver-svc.default.svc]
2018-12-12 06:26:23 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped! 
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore cleared 
2018-12-12 06:26:23 INFO  BlockManager:54 - BlockManager stopped 
2018-12-12 06:26:23 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped 
2018-12-12 06:26:23 WARN  MetricsSystem:66 - Stopping a MetricsSystem that is not running 
2018-12-12 06:26:23 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped! 
2018-12-12 06:26:23 INFO  SparkContext:54 - Successfully stopped SparkContext 
Exception in thread "main" org.apache.spark.SparkException: External scheduler cannot be instantiated 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) 
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) 
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) 
        at scala.Option.getOrElse(Option.scala:121) 
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) 
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) 
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:498) 
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
        at org.apache.spark.deploy.SparkSubmit.org [org.apache.spark.deploy.sparksubmit.org]$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) 
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) 
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) 
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) 
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) 
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) 
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167) 
        at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) 
        at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64) 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788) 
        ... 20 more 
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) 
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) 
        ... 23 more 
Caused by: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) 
        ... 29 more 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Shutdown hook called 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa 

It looks to me like there is a communication / authentication issue during pod creation, and it looks like there might be a certificate missing that Spark needs to communicate with Kubernetes, which I infer from the "java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input" error message. 

Any ideas on what I can do to resolve this? Thanks!


smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problem running Spark on Kubernetes: Certificate error

stevenstetzler
Hi Matt,

Thank your for the help. This worked for me. For posterity: convert the data in the certificate-authority-data field of your DigitalOcean Kubernetes configuration file, which is downloaded from their site, from base64 to PEM format and save it to a file ca.crt, then submit with --conf spark.kubernetes.authenticate.driver.caCertFile=ca.crt.

As an extension to this question, I am also interested in using PySpark on a JupyterHub deployment to interact with my cluster. The JupyterHub deployment provides a Python (with PySpark) environment in a pod on a node in the cluster, and I would like to create a SparkContext for computations that uses the same cluster. I have attempted to replicate the above solution there using:

from pyspark import *
conf = SparkConf()
conf.setMaster('k8s://<CLUSTERIP>')
conf.set('spark.kubernetes.container.image', '<IMAGEURL>')
conf.set('spark.kubernetes.authenticate.driver.caCertFile', 'ca.crt')
sc = SparkContext(conf=conf)

which returns an identical error:


On Thu, Dec 13, 2018 at 7:59 PM Matt Cheah <[hidden email]> wrote:

Hi Steven,

 

What I think is happening is that your machine has a CA certificate that is used for communicating with your API server, particularly because you’re using Digital Ocean’s cluster manager. However, it’s unclear if your pod has the same CA certificate or if the pod needs that certificate file. You can use configurations to have your pod use a particular CA certificate file to communicate with the APi server. If you set spark.kubernetes.authenticate.driver.caCertFile to the path of your CA certificate on your local disk, spark-submit will create a secret that contains that certificate file and use that certificate to configure TLS for the driver pod’s communication with the API server.

 

It's clear that your driver pod doesn’t have the right TLS certificate to communicate with the API server, so I would try to introspect the driver pod and see what certificate it’s using for that communication. If there’s a fix that needs to happen in Spark, feel free to indicate as such.

 

-Matt Cheah

 

From: Steven Stetzler <[hidden email]>
Date: Thursday, December 13, 2018 at 1:49 PM
To: "[hidden email]" <[hidden email]>
Subject: Problem running Spark on Kubernetes: Certificate error

 

Hello, 

I am following the tutorial here (
https://spark.apache.org/docs/latest/running-on-kubernetes.html [spark.apache.org]) to get spark running on a Kubernetes cluster. My Kubernetes cluster is hosted with Digital Ocean's kubernetes cluster manager. I have change the KUBECONFIG environment variable to point to my cluster access credentials, so both Spark and kubectl can speak with the nodes. 

I am running into an issue when trying to run the SparkPi example as described in the Spark on Kubernetes tutorials. The command I am running is: 

./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=1 --conf spark.kubernetes.container.image=$IMAGEURL --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar 

where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the URL of the Spark docker image I am using (
https://hub.docker.com/r/stevenstetzler/spark/ [hub.docker.com]). This docker image was built and pushed with the script included in the Spark 2.4 distribution. I have created a service account for Spark to ensure that it has proper permissions to create pods etc., which I checked using 

kubectl auth can-i create pods --as=system:serviceaccount:default:spark 

When I try to run the SparkPi example using the above command, I get the following output: 

2018-12-12 06:26:15 WARN  Utils:66 - Your hostname, docker-test resolves to a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface eth0) 
2018-12-12 06:26:15 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: N/A 
         start time: N/A 
         container images: N/A 
         phase: Pending 
         status: [] 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: N/A 
         container images: N/A 
         phase: Pending 
         status: [] 
2018-12-12 06:26:19 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: docker.io/stevenstetzler/spark:v1 [docker.io] 
         phase: Pending 
         status: [ContainerStatus(containerID=null, image=docker.io/stevenstetzler/spark:v1 [docker.io], imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:19 INFO  Client:54 - Waiting for application spark-pi to finish... 
2018-12-12 06:26:21 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: stevenstetzler/spark:v1 
         phase: Running 
         status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
         pod name: spark-pi-1544595975520-driver 
         namespace: default 
         labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver 
         pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 
         creation time: 2018-12-12T06:26:18Z 
         service account name: spark 
         volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt 
         node name: flamboyant-darwin-3rhc 
         start time: 2018-12-12T06:26:18Z 
         container images: stevenstetzler/spark:v1 
         phase: Failed 
         status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z, additionalProperties={}), message=null, reason=Error, signal=null, startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})] 
2018-12-12 06:26:24 INFO  LoggingPodStatusWatcherImpl:54 - Container final statuses: 


         Container name: spark-kubernetes-driver 
         Container image: stevenstetzler/spark:v1 
         Container state: Terminated 
         Exit code: 1 
2018-12-12 06:26:24 INFO  Client:54 - Application spark-pi finished. 
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Shutdown hook called 
2018-12-12 06:26:24 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd 

When I check the logs with 

kubectl logs spark-pi-1544595975520-driver 

I get the following: 

++ id -u 
+ myuid=0 
++ id -g 
+ mygid=0 
+ set +e 
++ getent passwd 0 
+ uidentry=root:x:0:0:root:/root:/bin/ash 
+ set -e 
+ '[' -z root:x:0:0:root:/root:/bin/ash ']' 
+ SPARK_K8S_CMD=driver 
+ case "$SPARK_K8S_CMD" in 
+ shift 1 
+ SPARK_CLASSPATH=':/opt/spark/jars/*' 
+ env 
+ grep SPARK_JAVA_OPT_ 
+ sed 's/[^=]*=\(.*\)/\1/g' 
+ sort -t_ -k4 -n 
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS 
+ '[' -n '' ']' 
+ '[' -n '' ']' 
+ PYSPARK_ARGS= 
+ '[' -n '' ']' 
+ R_ARGS= 
+ '[' -n '' ']' 
+ '[' '' == 2 ']' 
+ '[' '' == 3 ']' 
+ case "$SPARK_K8S_CMD" in 
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@") 
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi spark-internal 
2018-12-12 06:26:22 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
2018-12-12 06:26:22 INFO  SparkContext:54 - Running Spark version 2.4.0 
2018-12-12 06:26:22 INFO  SparkContext:54 - Submitted application: Spark Pi 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls to: root 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls to: root 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing view acls groups to: 
2018-12-12 06:26:22 INFO  SecurityManager:54 - Changing modify acls groups to: 
2018-12-12 06:26:22 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set() 
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 7078. 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering MapOutputTracker 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering BlockManagerMaster 
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 
2018-12-12 06:26:23 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up 
2018-12-12 06:26:23 INFO  DiskBlockManager:54 - Created local directory at /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf 
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore started with capacity 413.9 MB 
2018-12-12 06:26:23 INFO  SparkEnv:54 - Registering OutputCommitCoordinator 
2018-12-12 06:26:23 INFO  log:192 - Logging initialized @2059ms 
2018-12-12 06:26:23 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown 
2018-12-12 06:26:23 INFO  Server:419 - Started @2155ms 
2018-12-12 06:26:23 INFO  AbstractConnector:278 - Started ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]
2018-12-12 06:26:23 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040. 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6b410923{/jobs/job/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@203dd56b{/stages/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e8fadb0{/stages/stage,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1d3e6d34{/stages/stage/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6eafb10e{/stages/pool,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@26a94fa5{/stages/pool/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2873d672{/storage/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3bc735b3{/storage/rdd,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@577f9109{/storage/rdd/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4303b7f0{/environment,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@757529a4{/environment/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c41d037{/executors/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2234078{/executors/threadDump,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ec77191{/executors/threadDump/json,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@534ca02b{/jobs/job/kill,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29a23c3d{/stages/stage/kill,null,AVAILABLE,@Spark} 
2018-12-12 06:26:23 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at 
http://spark-pi-1544595975520-driver-svc.default.svc:4040 [spark-pi-1544595975520-driver-svc.default.svc]
2018-12-12 06:26:23 INFO  SparkContext:54 - Added JAR file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar with timestamp 1544595983645 
2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing SparkContext. 
org.apache.spark.SparkException: External scheduler cannot be instantiated 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) 
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) 
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) 
        at scala.Option.getOrElse(Option.scala:121) 
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) 
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) 
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:498) 
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
        at org.apache.spark.deploy.SparkSubmit.org [org.apache.spark.deploy.sparksubmit.org]$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) 
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) 
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) 
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) 
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) 
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) 
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167) 
        at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) 
        at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64) 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788) 
        ... 20 more 
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) 
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) 
        ... 23 more 
Caused by: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) 
        ... 29 more 
2018-12-12 06:26:23 INFO  AbstractConnector:318 - Stopped Spark@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040 [0.0.0.0]
2018-12-12 06:26:23 INFO  SparkUI:54 - Stopped Spark web UI at 
http://spark-pi-1544595975520-driver-svc.default.svc:4040 [spark-pi-1544595975520-driver-svc.default.svc]
2018-12-12 06:26:23 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped! 
2018-12-12 06:26:23 INFO  MemoryStore:54 - MemoryStore cleared 
2018-12-12 06:26:23 INFO  BlockManager:54 - BlockManager stopped 
2018-12-12 06:26:23 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped 
2018-12-12 06:26:23 WARN  MetricsSystem:66 - Stopping a MetricsSystem that is not running 
2018-12-12 06:26:23 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped! 
2018-12-12 06:26:23 INFO  SparkContext:54 - Successfully stopped SparkContext 
Exception in thread "main" org.apache.spark.SparkException: External scheduler cannot be instantiated 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) 
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) 
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) 
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) 
        at scala.Option.getOrElse(Option.scala:121) 
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) 
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) 
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:498) 
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
        at org.apache.spark.deploy.SparkSubmit.org [org.apache.spark.deploy.sparksubmit.org]$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) 
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) 
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) 
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) 
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) 
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) 
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) 
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167) 
        at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) 
        at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64) 
        at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788) 
        ... 20 more 
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) 
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93) 
        at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114) 
        at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93) 
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) 
        ... 23 more 
Caused by: java.io.IOException: Empty input 
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) 
        ... 29 more 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Shutdown hook called 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7 
2018-12-12 06:26:23 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa 

It looks to me like there is a communication / authentication issue during pod creation, and it looks like there might be a certificate missing that Spark needs to communicate with Kubernetes, which I infer from the "java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input" error message. 

Any ideas on what I can do to resolve this? Thanks!