Why can't Spark find the classes in this Jar?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Why can't Spark find the classes in this Jar?

akh2103
This post has NOT been accepted by the mailing list yet.
I am new to spark. I am trying to compile and run a spark application that requires classes from an (external) jar file on my local machine. If I open the jar (on ~/Desktop) I can see the missing class in the local jar but when I run spark I get

NoClassDefFoundError: edu/stanford/nlp/ie/AbstractSequenceClassifier

I add the jar to the spark context like this

String[] jars = {"/home/pathto/Desktop/stanford-corenlp-3.5.0.jar"};
SparkConf conf = new SparkConf().setAppName("Simple Application").setJars(jars);
Then I try to run a submit script like this

/home/me/Downloads/spark-1.2.0-bin-hadoop2.4/bin/spark-submit \
  --class "SimpleApp" \
  --master local[4] \
  target/simple-project-1.0.jar \
  --jars local[4] /home/abe/Desktop/stanford-corenlp-3.5.0.jar
and hit the NoClassDefFoundError.

I get that this means that the worker threads can't find the class from the jar. But I am not sure what I am doing wrong. I have tried different syntaxes for the last line (below) but none works.

  --addJars local[4] /home/abe/Desktop/stanford-corenlp-3.5.0.jar
  --addJars local:/home/abe/Desktop/stanford-corenlp-3.5.0.jar
  --addJars local:/home/abe/Desktop/stanford-corenlp-3.5.0.jar

How can I fix this error?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why can't Spark find the classes in this Jar?

Aplysia
This post has NOT been accepted by the mailing list yet.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why can't Spark find the classes in this Jar?

maziyar
This post has NOT been accepted by the mailing list yet.
Hi,

For the future readers, --jars in spark-shell must be from local file systems but in spark-submit it should be accessible by all the workers (executors). This means you need something like HDFS path for your jars.

It works on Spark 1.6 +
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why can't Spark find the classes in this Jar?

dori waldman
This post has NOT been accepted by the mailing list yet.
I dont think put jars in hdfs is correct ,  I am not using it and its working
I am using

--jars "../lib/myJar.jar"


and its working well also on the workers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why can't Spark find the classes in this Jar?

maziyar
This post has NOT been accepted by the mailing list yet.
Hi,

It's weird, cause I can't use HDFS path when I am using spark-shell (only accepts local files) and I can't use local path when I am using spark-submit over YARN cluster (Cloudera).

What is the cluster setup? (standalone, YARN, etc.) Maybe that helps finding why this is the case in my environment.

Thanks
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why can't Spark find the classes in this Jar?

dori waldman
This post has NOT been accepted by the mailing list yet.
I used local path both in yarn and standalone 
can you write the command you are running 

On Mon, Jul 17, 2017 at 7:17 PM, maziyar [via Apache Spark User List] <[hidden email]> wrote:
Hi,

It's weird, cause I can't use HDFS path when I am using spark-shell (only accepts local files) and I can't use local path when I am using spark-submit over YARN cluster (Cloudera).

What is the cluster setup? (standalone, YARN, etc.) Maybe that helps finding why this is the case in my environment.

Thanks


If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Why-can-t-Spark-find-the-classes-in-this-Jar-tp21584p28871.html
To unsubscribe from Why can't Spark find the classes in this Jar?, click here.
NAML



--

 

Dori Waldman
Big Data Lead

 

mobile. +972-52-8927444
website. www.inner-active.com

 

 

Loading...