[Spark on Google Kubernetes Engine] Properties File Error

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Spark on Google Kubernetes Engine] Properties File Error

Eric Wang
Hello all,

I've been trying to spark-submit a job to the Google Kubernetes Engine but I keep encountering a "Exception in thread "main" java.lang.IllegalArgumentException: Server properties file given at /opt/spark/work-dir/driver does not exist or is not a file."
error. I'm unsure of how to even begin debugging this so any help would be greatly appreciated. I've attached the logs and the full spark-submit command I'm running here: https://gist.github.com/erkkel/c04a0b5ca60ad755cf62e9ad18e5b7ed


Thanks,
Eric

Reply | Threaded
Open this post in threaded view
|

Re: [Spark on Google Kubernetes Engine] Properties File Error

Holden Karau
So, while its not perfect, I have a guide focused on running custom Spark on GKE https://cloud.google.com/blog/big-data/2018/03/testing-future-apache-spark-releases-and-changes-on-google-kubernetes-engine-and-cloud-dataproc and if you want to run pre-built Spark on GKE there is a solutions article - https://cloud.google.com/solutions/spark-on-kubernetes-engine which could be relevant.

On Mon, Apr 30, 2018 at 7:51 PM, Eric Wang <[hidden email]> wrote:
Hello all,

I've been trying to spark-submit a job to the Google Kubernetes Engine but I keep encountering a "Exception in thread "main" java.lang.IllegalArgumentException: Server properties file given at /opt/spark/work-dir/driver does not exist or is not a file."
error. I'm unsure of how to even begin debugging this so any help would be greatly appreciated. I've attached the logs and the full spark-submit command I'm running here: https://gist.github.com/erkkel/c04a0b5ca60ad755cf62e9ad18e5b7ed


Thanks,
Eric




--
Reply | Threaded
Open this post in threaded view
|

Re: [Spark on Google Kubernetes Engine] Properties File Error

Yinan Li
 Which version of Spark are you using to run spark-submit, and which version of Spark your container image is based off? This looks to be caused my mismatched versions of Spark used for spark-submit and for the driver/executor at runtime.

On Mon, Apr 30, 2018 at 12:00 PM, Holden Karau <[hidden email]> wrote:
So, while its not perfect, I have a guide focused on running custom Spark on GKE https://cloud.google.com/blog/big-data/2018/03/testing-future-apache-spark-releases-and-changes-on-google-kubernetes-engine-and-cloud-dataproc and if you want to run pre-built Spark on GKE there is a solutions article - https://cloud.google.com/solutions/spark-on-kubernetes-engine which could be relevant.

On Mon, Apr 30, 2018 at 7:51 PM, Eric Wang <[hidden email]> wrote:
Hello all,

I've been trying to spark-submit a job to the Google Kubernetes Engine but I keep encountering a "Exception in thread "main" java.lang.IllegalArgumentException: Server properties file given at /opt/spark/work-dir/driver does not exist or is not a file."
error. I'm unsure of how to even begin debugging this so any help would be greatly appreciated. I've attached the logs and the full spark-submit command I'm running here: https://gist.github.com/erkkel/c04a0b5ca60ad755cf62e9ad18e5b7ed


Thanks,
Eric




--

Reply | Threaded
Open this post in threaded view
|

Re: [Spark on Google Kubernetes Engine] Properties File Error

Eric Wang
Thanks so much! I'll take a look at the guide right now. The versions should all be 2.2 of spark. In my configuration, I'm using

  --conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
  --conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \

and I built the version of spark located here: https://github.com/apache-spark-on-k8s/spark which also seems to be 2.2


On Mon, Apr 30, 2018 at 12:09 PM, Yinan Li <[hidden email]> wrote:
 Which version of Spark are you using to run spark-submit, and which version of Spark your container image is based off? This looks to be caused my mismatched versions of Spark used for spark-submit and for the driver/executor at runtime.

On Mon, Apr 30, 2018 at 12:00 PM, Holden Karau <[hidden email]> wrote:
So, while its not perfect, I have a guide focused on running custom Spark on GKE https://cloud.google.com/blog/big-data/2018/03/testing-future-apache-spark-releases-and-changes-on-google-kubernetes-engine-and-cloud-dataproc and if you want to run pre-built Spark on GKE there is a solutions article - https://cloud.google.com/solutions/spark-on-kubernetes-engine which could be relevant.

On Mon, Apr 30, 2018 at 7:51 PM, Eric Wang <[hidden email]> wrote:
Hello all,

I've been trying to spark-submit a job to the Google Kubernetes Engine but I keep encountering a "Exception in thread "main" java.lang.IllegalArgumentException: Server properties file given at /opt/spark/work-dir/driver does not exist or is not a file."
error. I'm unsure of how to even begin debugging this so any help would be greatly appreciated. I've attached the logs and the full spark-submit command I'm running here: https://gist.github.com/erkkel/c04a0b5ca60ad755cf62e9ad18e5b7ed


Thanks,
Eric




--


Reply | Threaded
Open this post in threaded view
|

Re: [Spark on Google Kubernetes Engine] Properties File Error

Yinan Li
In reply to this post by Yinan Li
Also looks like you are mixing configuration properties from different versions of Spark on Kubernetes. "spark.kubernetes.{driver|executor}.docker.image" is only available in the apache-spark-on-k8s fork, whereas "spark.kubernetes.container.image" is new in Spark 2.3.0. Please make sure you use the same version of Spark to run spark-submit and to build your image. You also need to make sure that you use the right properties for the version of Spark you use. BTW: we suggest people to use the official Spark 2.3.0 release instead of the fork. The guide is at http://spark.apache.org/docs/latest/running-on-kubernetes.html

On Mon, Apr 30, 2018 at 12:09 PM, Yinan Li <[hidden email]> wrote:
 Which version of Spark are you using to run spark-submit, and which version of Spark your container image is based off? This looks to be caused my mismatched versions of Spark used for spark-submit and for the driver/executor at runtime.

On Mon, Apr 30, 2018 at 12:00 PM, Holden Karau <[hidden email]> wrote:
So, while its not perfect, I have a guide focused on running custom Spark on GKE https://cloud.google.com/blog/big-data/2018/03/testing-future-apache-spark-releases-and-changes-on-google-kubernetes-engine-and-cloud-dataproc and if you want to run pre-built Spark on GKE there is a solutions article - https://cloud.google.com/solutions/spark-on-kubernetes-engine which could be relevant.

On Mon, Apr 30, 2018 at 7:51 PM, Eric Wang <[hidden email]> wrote:
Hello all,

I've been trying to spark-submit a job to the Google Kubernetes Engine but I keep encountering a "Exception in thread "main" java.lang.IllegalArgumentException: Server properties file given at /opt/spark/work-dir/driver does not exist or is not a file."
error. I'm unsure of how to even begin debugging this so any help would be greatly appreciated. I've attached the logs and the full spark-submit command I'm running here: https://gist.github.com/erkkel/c04a0b5ca60ad755cf62e9ad18e5b7ed


Thanks,
Eric




--