Spark 2.2.1 worker invocation

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark 2.2.1 worker invocation

Christopher Piggott
I need to set java.library.path to get access to some native code.  Following directions, I made a spark-env.sh:

    #!/usr/bin/env bash
    export LD_LIBRARY_PATH="/usr/local/lib/libcdfNativeLibrary.so:/usr/local/lib/libcdf.so:${LD_LIBRARY_PATH}"
    export SPARK_WORKER_OPTS=-Djava.library.path=/usr/local/lib
    export SPARK_WORKER_MEMORY=2g

to no avail.  (I tried both with and without exporting the environment).  Looking at how the worker actually starts up:

     /usr/lib/jvm/default/bin/java -cp /home/spark/conf/:/home/spark/jars/* -Xmx1024M -Dspark.driver.port=37219 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.1.1.1:37219 --executor-id 113 --hostname 10.2.2.1 --cores 8 --app-id app-20171225145607-0003 --worker-url spark://Worker@10.2.2.1:35449



It doesn't seem to take any options.  I put an 'echo' in just to confirm that spark-env.sh is getting invoked (and it is).

So, just out of curiosity, I tried to troubleshoot this:



    spark@node2-1:~$ grep -R SPARK_WORKER_OPTS *
    conf/spark-env.sh:export SPARK_WORKER_OPTS=-Djava.library.path=/usr/local/lib
    conf/spark-env.sh.template:# - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y")


The variable doesn't seem to get referenced anywhere in the spark distribution.  I checked a number of other options in spark-env.sh.template and they didn't seem to be referenced either.  I expected to find them in various startup scripts.

I can probably "fix" my problem by hacking the lower-level startup scripts, but first I'd like to inquire about what's going on here.  How and where are these variables actually used?


Reply | Threaded
Open this post in threaded view
|

Re: Spark 2.2.1 worker invocation

Felix Cheung
I think you are looking for spark.executor.extraJavaOptions

https://spark.apache.org/docs/latest/configuration.html#runtime-environment


From: Christopher Piggott <[hidden email]>
Sent: Tuesday, December 26, 2017 8:00:56 AM
To: [hidden email]
Subject: Spark 2.2.1 worker invocation
 
I need to set java.library.path to get access to some native code.  Following directions, I made a spark-env.sh:

    #!/usr/bin/env bash
    export LD_LIBRARY_PATH="/usr/local/lib/libcdfNativeLibrary.so:/usr/local/lib/libcdf.so:${LD_LIBRARY_PATH}"
    export SPARK_WORKER_OPTS=-Djava.library.path=/usr/local/lib
    export SPARK_WORKER_MEMORY=2g

to no avail.  (I tried both with and without exporting the environment).  Looking at how the worker actually starts up:

     /usr/lib/jvm/default/bin/java -cp /home/spark/conf/:/home/spark/jars/* -Xmx1024M -Dspark.driver.port=37219 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.1.1.1:37219 --executor-id 113 --hostname 10.2.2.1 --cores 8 --app-id app-20171225145607-0003 --worker-url spark://Worker@10.2.2.1:35449



It doesn't seem to take any options.  I put an 'echo' in just to confirm that spark-env.sh is getting invoked (and it is).

So, just out of curiosity, I tried to troubleshoot this:



    spark@node2-1:~$ grep -R SPARK_WORKER_OPTS *
    conf/spark-env.sh:export SPARK_WORKER_OPTS=-Djava.library.path=/usr/local/lib
    conf/spark-env.sh.template:# - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y")


The variable doesn't seem to get referenced anywhere in the spark distribution.  I checked a number of other options in spark-env.sh.template and they didn't seem to be referenced either.  I expected to find them in various startup scripts.

I can probably "fix" my problem by hacking the lower-level startup scripts, but first I'd like to inquire about what's going on here.  How and where are these variables actually used?