Run Spark on Java 10

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Run Spark on Java 10

*my user case*: We run Spark cluster on Mesos, since our Mesos cluster is
also hosting other frameworks such as Storm, Cassandra, we had incidents
where Spark job over-utilizes CPU which caused resource contention with
other frameworks.

*objective* : run un-modularized spark application (jar is compiled with
java 8 compatible sbt compiler) on Java 10 to leverage Java 10 container
support. Related link:

After reading some readings about Java 9, this is my imaginary *happy-path*:

1) Point JAVA_HOME to Java 10,
2) Run my spark job, resolve classNotFoundException, lookup the missing
modules in oracle documentation, for example (java.sql module is
missing, add “spark.executor.extraJavaOptions --add-modules” ->
3) Repeat step 2 until no more exceptions are thrown

however I found this,

Warning: Local jar ***/ does not exist, skipping.

/java.lang.ClassNotFoundException: com.**.spark.Main at
at java.base/java.lang.ClassLoader.loadClass(
at java.base/java.lang.ClassLoader.loadClass(
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(
at org.apache.spark.util.Utils$.classForName(Utils.scala:233)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)/

This is what I have observed after turning verbose mode


Using properties file:

Adding default property: spark.eventLog.enabled=true Adding default
property: spark.eventLog.dir=hdfs://*** Adding default property:

Parsed arguments:

master                 mesos://localhost:10017
deployMode             cluster
executorMemory         16G
executorCores           2
totalExecutorCores     50
propertiesFile         /tmp/spark-2.2.2-bin-hadoop2.6/conf/spark-
driverMemory           4G
driverCores             1
driverExtraClassPath   null
driverExtraLibraryPath null
driverExtraJavaOptions null
supervise               false
queue                   null
numExecutors           null
files                   null
pyFiles                 null
archives               null
mainClass               com.**.spark.Main
primaryResource         ***.jar
name                   ***
childArgs               [***]
jars                   null
packages               null
packagesExclusions     null
repositories           null
verbose                true

Spark properties used, including those specified through
--conf and those from the properties file

*The warning was printed here:*

After reading the source code. Seems to me that spark-submit does not
understand --add-modules option so it treat as a jar file rather
than a module. *And I coundn`t make it the way I want it to translate
--add-modules when launching executor JVM.

Has anyone done similar experiments running Spark on Java 9/10?*

Thanks in advance

Sent from:

To unsubscribe e-mail: [hidden email]