sample syntax in spark-env.sh for env.

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

sample syntax in spark-env.sh for env.

Zahid Rahman

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

I was chasing this warning when I found misinformation
from SPARK training companies such as eudreka who offer weird and wonderful suggestions

https://www.edureka.co/community/110/hadoop-unable-native-hadoop-library-your-platform-warning


Meanhwile another SPARK Training company cloudera has claimed resolution of issue by turning the warning off to an API which may be needed as warned by SPARK  developer.  

https://community.cloudera.com/t5/Support-Questions/Apache-Spark-Error-Unable-to-load-native-hadoop-library/td-p/160968

StackOverflow also has misleading information i.e. change /etc/profile,  ~/.bashrc.

Anyway I found the resolution is to insert the bottom two lines to include the hadoop libraries
not SPARK_LOCAL_IP.

SPARK_LOCAL_IP=192.168.0.38
HADOOP_HOME=/home/kub18/hadoop-3.2.1
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native


The reason for this post is that there was no sample syntax so this is was my first change.
SPARK_LOCAL_IP=192.168.0.38
export HADOOP_HOME=/home/kub18/hadoop-3.2.1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native
That was accepted by SPARK code.

My second change was :
SPARK_LOCAL_IP=192.168.0.38
HADOOP_HOME=/home/kub18/hadoop-3.2.1
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native
that was also accepted.


my third change was
SPARK_LOCAL_IP=192.168.0.38
doogie HADOOP_HOME=/home/kub18/hadoop-3.2.1
doogie LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native
This change produced a doogie command not found error.

So it is unclear by having the prefix "export" in spark-env.sh as illustrated
changes the behaviour of SPARK or NOT.
 
my suggestion would be to supply example syntax in the spark-env.sh and also give information whether
by prefixing "export" to the SPARK environment variable  makes a difference or not.


Best Regards
Z.


¯\_(ツ)_/¯
♡۶Java♡۶RMI ♡۶
Make Use Method {MUM}