Add snappy support for spark in Windows

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Add snappy support for spark in Windows

Junfeng Chen

I am working on importing snappy compressed json file into spark rdd or dataset. However I meet this error: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

I have set the following configuration:

SparkConf conf = new SparkConf()
            .setAppName("normal spark")
            .setMaster("local")
            .set("spark.io.compression.codec", "org.apache.spark.io.SnappyCompressionCodec")
            .set("spark.driver.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.driver.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.executor.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.executor.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            ;

Where D:\Downloads\spark-2.2.0-bin-hadoop2.7 is my spark unpacked path, and I can find the snappy jar file snappy-0.2.jar and snappy-java-1.1.2.6.jar in

D:\Downloads\spark-2.2.0-bin-hadoop2.7\spark-2.2.0-bin-hadoop2.7\jars\

However nothing works and even the error message not change.

How can I fix it?


ref of stackoverflow: https://stackoverflow.com/questions/47626012/config-snappy-support-for-spark-in-windows  



Regard,
Junfeng Chen
Reply | Threaded
Open this post in threaded view
|

Re: Add snappy support for spark in Windows

Qiao, Richard
It seems a common mistake that the path is not accessible by workers/executors.

Best regards 
Richard 

Sent from my iPhone

On Dec 3, 2017, at 22:32, Junfeng Chen <[hidden email]> wrote:

I am working on importing snappy compressed json file into spark rdd or dataset. However I meet this error: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

I have set the following configuration:

SparkConf conf = new SparkConf()
            .setAppName("normal spark")
            .setMaster("local")
            .set("spark.io.compression.codec", "org.apache.spark.io.SnappyCompressionCodec")
            .set("spark.driver.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.driver.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.executor.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.executor.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            ;

Where D:\Downloads\spark-2.2.0-bin-hadoop2.7 is my spark unpacked path, and I can find the snappy jar file snappy-0.2.jar and snappy-java-1.1.2.6.jar in

D:\Downloads\spark-2.2.0-bin-hadoop2.7\spark-2.2.0-bin-hadoop2.7\jars\

However nothing works and even the error message not change.

How can I fix it?


ref of stackoverflow: https://stackoverflow.com/questions/47626012/config-snappy-support-for-spark-in-windows  



Regard,
Junfeng Chen


The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

Reply | Threaded
Open this post in threaded view
|

Re: Add snappy support for spark in Windows

Junfeng Chen
But I am working on my local development machine, so it should have no relative to workers/executers.

I find some documents about enable snappy on hadoop. If I want to use snappy with spark, do I need to config spark as hadoop or have some easy way to access it? 


Regard,
Junfeng Chen

On Mon, Dec 4, 2017 at 4:12 PM, Qiao, Richard <[hidden email]> wrote:
It seems a common mistake that the path is not accessible by workers/executors.

Best regards 
Richard 

Sent from my iPhone

On Dec 3, 2017, at 22:32, Junfeng Chen <[hidden email]> wrote:

I am working on importing snappy compressed json file into spark rdd or dataset. However I meet this error: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

I have set the following configuration:

SparkConf conf = new SparkConf()
            .setAppName("normal spark")
            .setMaster("local")
            .set("spark.io.compression.codec", "org.apache.spark.io.SnappyCompressionCodec")
            .set("spark.driver.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.driver.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.executor.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.executor.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            ;

Where D:\Downloads\spark-2.2.0-bin-hadoop2.7 is my spark unpacked path, and I can find the snappy jar file snappy-0.2.jar and snappy-java-1.1.2.6.jar in

D:\Downloads\spark-2.2.0-bin-hadoop2.7\spark-2.2.0-bin-hadoop2.7\jars\

However nothing works and even the error message not change.

How can I fix it?


ref of stackoverflow: https://stackoverflow.com/questions/47626012/config-snappy-support-for-spark-in-windows  



Regard,
Junfeng Chen


The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.


Reply | Threaded
Open this post in threaded view
|

Re: Add snappy support for spark in Windows

Qiao, Richard

Junjeng, it worth a try to start your spark local with hadoop.dll/winutils.exe etc hadoop windows support package in HADOOP_HOME, if you didn’t do that yet.

 

Best Regards

Richard

 

 

From: Junfeng Chen <[hidden email]>
Date: Monday, December 4, 2017 at 3:53 AM
To: "Qiao, Richard" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Add snappy support for spark in Windows

 

But I am working on my local development machine, so it should have no relative to workers/executers.

 

I find some documents about enable snappy on hadoop. If I want to use snappy with spark, do I need to config spark as hadoop or have some easy way to access it? 


 

Regard,
Junfeng Chen

 

On Mon, Dec 4, 2017 at 4:12 PM, Qiao, Richard <[hidden email]> wrote:

It seems a common mistake that the path is not accessible by workers/executors.

 

Best regards 

Richard 

Sent from my iPhone


On Dec 3, 2017, at 22:32, Junfeng Chen <[hidden email]> wrote:

I am working on importing snappy compressed json file into spark rdd or dataset. However I meet this error: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

I have set the following configuration:

SparkConf conf = new SparkConf()
            .setAppName("normal spark")
            .setMaster("local")
            .set("spark.io.compression.codec", "org.apache.spark.io.SnappyCompressionCodec")
            .set("spark.driver.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.driver.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.executor.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.executor.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            ;

Where D:\Downloads\spark-2.2.0-bin-hadoop2.7 is my spark unpacked path, and I can find the snappy jar file snappy-0.2.jar and snappy-java-1.1.2.6.jar in

D:\Downloads\spark-2.2.0-bin-hadoop2.7\spark-2.2.0-bin-hadoop2.7\jars\

However nothing works and even the error message not change.

How can I fix it?

 

ref of stackoverflow: https://stackoverflow.com/questions/47626012/config-snappy-support-for-spark-in-windows  

 

 

Regard,
Junfeng Chen

 


The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

 



The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

Reply | Threaded
Open this post in threaded view
|

Re: Add snappy support for spark in Windows

Junfeng Chen
I have put winutils and hadoop.dll within HADOOP_HOME, and spark works well with it, but snappy decompress function throw the above exception. 


Regard,
Junfeng Chen

On Mon, Dec 4, 2017 at 7:07 PM, Qiao, Richard <[hidden email]> wrote:

Junjeng, it worth a try to start your spark local with hadoop.dll/winutils.exe etc hadoop windows support package in HADOOP_HOME, if you didn’t do that yet.

 

Best Regards

Richard

 

 

From: Junfeng Chen <[hidden email]>
Date: Monday, December 4, 2017 at 3:53 AM
To: "Qiao, Richard" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Add snappy support for spark in Windows

 

But I am working on my local development machine, so it should have no relative to workers/executers.

 

I find some documents about enable snappy on hadoop. If I want to use snappy with spark, do I need to config spark as hadoop or have some easy way to access it? 


 

Regard,
Junfeng Chen

 

On Mon, Dec 4, 2017 at 4:12 PM, Qiao, Richard <[hidden email]> wrote:

It seems a common mistake that the path is not accessible by workers/executors.

 

Best regards 

Richard 

Sent from my iPhone


On Dec 3, 2017, at 22:32, Junfeng Chen <[hidden email]> wrote:

I am working on importing snappy compressed json file into spark rdd or dataset. However I meet this error: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

I have set the following configuration:

SparkConf conf = new SparkConf()
            .setAppName("normal spark")
            .setMaster("local")
            .set("spark.io.compression.codec", "org.apache.spark.io.SnappyCompressionCodec")
            .set("spark.driver.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.driver.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.executor.extraLibraryPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            .set("spark.executor.extraClassPath","D:\\Downloads\\spark-2.2.0-bin-hadoop2.7\\spark-2.2.0-bin-hadoop2.7\\jars")
            ;

Where D:\Downloads\spark-2.2.0-bin-hadoop2.7 is my spark unpacked path, and I can find the snappy jar file snappy-0.2.jar and snappy-java-1.1.2.6.jar in

D:\Downloads\spark-2.2.0-bin-hadoop2.7\spark-2.2.0-bin-hadoop2.7\jars\

However nothing works and even the error message not change.

How can I fix it?

 

ref of stackoverflow: https://stackoverflow.com/questions/47626012/config-snappy-support-for-spark-in-windows  

 

 

Regard,
Junfeng Chen

 


The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

 



The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.