[Spark Core] - Installation issue - "java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path"

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[Spark Core] - Installation issue - "java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path"

jelvis
This post was updated on .
Dear all, Sorry for the edit, but I unintentionally messed up the formatting. Looked good when I created it on the web. Not so when I looked at what was posted. I have setup two Spark standalone test clusters which both suffered from the same problem. I have a workaround but it's bad. I would appreciate some help and input. I'm too much of a beginner to conclude that it's a bug but I found someone else having the exact same issue on Stack Overflow (where I also posted my own question.) My own question on Stack Overflow: https://stackoverflow.com/questions/64029599/cant-run-example-pyspark-code-on-standalone-cluster Another person with the same question on Stack Overflow: https://stackoverflow.com/questions/63143795/spark-3-error-java-lang-unsatisfiedlinkerror-no-zstd-jni-in-java-library-path Error: When I run the following python test example from any of the two clusters, I get the same error. --- command line--- /usr/spark/bin/spark-submit --master "spark://:7077" /usr/spark/examples/src/main/python/wordcount.py '/usr/spark/LICENSE' ---command line--- --- error --- Exception in thread "map-output-dispatcher-0" java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.pathUnsupported OS/arch, cannot find /linux/amd64/libzstd-jni.so or load zstd-jni from system libraries. Please try building from source the jar or providing libzstd-jni in your system." --- error --- If I only run the example on one node Local[x] then it works without an issue. I assume that doesn't trigger the usage of libzstd-jni. Cluster setup: First cluster: - Three RHEL7 VMs. - Spark 3.0.0 downloaded from the apache site, prebuilt for hadoop 3.2. - JDK8. - JAVA_HOME configured. - Cluster set up in stand-alone mode, one master and two slaves. Second cluster: - Three RHEL8 VMs. - Spark 3.0.1 downloaded from the apache site, prebuilt for hadoop 3.2. - OpenJDK 11. - JAVA_HOME configured. - Cluster set up in stand-alone mode, one master and two slaves. - Hadoop 3.2.1 downloaded and installed. - LD_LIBRARY_PATH configured to point to the hadoop native library directory. - SPARK_HOME configured. - CLASSPATH set to point to $SPARK_HOME/jars Workaround: Looking at the issue I determined that the appropriate libzstd-jni.so was available in $SPARK_HOME/jars/zstd-jni-1.4.4-3.jar and it was possible to extract it from there. So I extracted the .so-file and put it into /usr/lib64 and then the example ran without an issue. But that is not a good way to go, and I'm wondering if I have made a mistake somewhere. I would expect that the java code would pick up the correct so-file from the jar automatically based on the environment it executes on? -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/