JDK1.8 for spark workers

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

JDK1.8 for spark workers

khajaasmath786
Hi,

I am running cloudera version of spark2.1 and our cluster is on JDK1.7. For some of the libraries, I need JDK1.8, is there a way to set to run Spark worker in JDK1.8 without upgrading .

I was able run driver in JDK 1.8 by setting the path but not the workers. 

17/11/28 20:22:27 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1, brksvl267.brk.navistar.com, executor 1): java.lang.UnsupportedClassVersionError: org/wololo/geojson/GeoJSON : Unsupported major.minor version 52.0

Thanks,
Asmath
Reply | Threaded
Open this post in threaded view
|

Re: JDK1.8 for spark workers

Vadim Semenov
You can pass `JAVA_HOME` environment variable

`spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-1.8.0`

On Wed, Nov 29, 2017 at 10:54 AM, KhajaAsmath Mohammed <[hidden email]> wrote:
Hi,

I am running cloudera version of spark2.1 and our cluster is on JDK1.7. For some of the libraries, I need JDK1.8, is there a way to set to run Spark worker in JDK1.8 without upgrading .

I was able run driver in JDK 1.8 by setting the path but not the workers. 

17/11/28 20:22:27 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1, brksvl267.brk.navistar.com, executor 1): java.lang.UnsupportedClassVersionError: org/wololo/geojson/GeoJSON : Unsupported major.minor version 52.0

Thanks,
Asmath

Reply | Threaded
Open this post in threaded view
|

Re: JDK1.8 for spark workers

khajaasmath786
This didnt work. I tried it but no luck.

On Wed, Nov 29, 2017 at 7:49 PM, Vadim Semenov <[hidden email]> wrote:
You can pass `JAVA_HOME` environment variable

`spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-1.8.0`

On Wed, Nov 29, 2017 at 10:54 AM, KhajaAsmath Mohammed <[hidden email]> wrote:
Hi,

I am running cloudera version of spark2.1 and our cluster is on JDK1.7. For some of the libraries, I need JDK1.8, is there a way to set to run Spark worker in JDK1.8 without upgrading .

I was able run driver in JDK 1.8 by setting the path but not the workers. 

17/11/28 20:22:27 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1, brksvl267.brk.navistar.com, executor 1): java.lang.UnsupportedClassVersionError: org/wololo/geojson/GeoJSON : Unsupported major.minor version 52.0

Thanks,
Asmath