Apache Spark User List

Errors with spark-0.8.1 hadoop-yarn 2.2.0

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
Izhar ul Hassan
Reply | Threaded
Open this post in threaded view
♦
♦
|

Errors with spark-0.8.1 hadoop-yarn 2.2.0

Izhar ul Hassan
Hi,

I have a 3 node installation of hadoop 2.2.0 with yarn. I have installed spark-0.8.1 with support for spark enabled. I get the following errors when trying to run the examples:

SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar \
    ./spark-class org.apache.spark.deploy.yarn.Client \
      --jar examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar \
      --class org.apache.spark.examples.SparkPi \
      --args yarn-standalone \
      --num-workers 3 \
      --master-memory 4g \
      --worker-memory 2g \
      --worker-cores 1


Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/Client
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.Client
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.deploy.yarn.Client. Program will exit.

spark-0.8.0 with hadooop 2.0.5-alpha works fine.
--
/Izhar 
Liu, Raymond
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: Errors with spark-0.8.1 hadoop-yarn 2.2.0

Liu, Raymond
Hi Izhar

        Is that the exact command you are running? Say with 0.8.0 instead of 0.8.1 in the cmd?

Raymond Liu

From: Izhar ul Hassan [mailto:[hidden email]]
Sent: Friday, December 27, 2013 9:40 PM
To: [hidden email]
Subject: Errors with spark-0.8.1 hadoop-yarn 2.2.0

Hi,
I have a 3 node installation of hadoop 2.2.0 with yarn. I have installed spark-0.8.1 with support for spark enabled. I get the following errors when trying to run the examples:
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar \
    ./spark-class org.apache.spark.deploy.yarn.Client \
      --jar examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar \
      --class org.apache.spark.examples.SparkPi \
      --args yarn-standalone \
      --num-workers 3 \
      --master-memory 4g \
      --worker-memory 2g \
      --worker-cores 1

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/Client
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.Client
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.deploy.yarn.Client. Program will exit.
spark-0.8.0 with hadooop 2.0.5-alpha works fine.
--
/Izhar 
Azuryy Yu
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Errors with spark-0.8.1 hadoop-yarn 2.2.0

Azuryy Yu
Maybe you don't enabled Yarn when build Spark.


On Mon, Dec 30, 2013 at 8:36 AM, Liu, Raymond <[hidden email]> wrote:
Hi Izhar

        Is that the exact command you are running? Say with 0.8.0 instead of 0.8.1 in the cmd?

Raymond Liu

From: Izhar ul Hassan [mailto:[hidden email]]
Sent: Friday, December 27, 2013 9:40 PM
To: [hidden email]
Subject: Errors with spark-0.8.1 hadoop-yarn 2.2.0

Hi,
I have a 3 node installation of hadoop 2.2.0 with yarn. I have installed spark-0.8.1 with support for spark enabled. I get the following errors when trying to run the examples:
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar \
    ./spark-class org.apache.spark.deploy.yarn.Client \
      --jar examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar \
      --class org.apache.spark.examples.SparkPi \
      --args yarn-standalone \
      --num-workers 3 \
      --master-memory 4g \
      --worker-memory 2g \
      --worker-cores 1

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/Client
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.Client
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.deploy.yarn.Client. Program will exit.
spark-0.8.0 with hadooop 2.0.5-alpha works fine.
--
/Izhar 
leosandylh@gmail.com
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: RE: Errors with spark-0.8.1 hadoop-yarn 2.2.0

leosandylh@gmail.com
In reply to this post by Liu, Raymond

What is your classpath ?
Had you builded your spark when you changed the new version and with yarn?
Have you find your jar under the $SPARK_HOME/assembly/target/scala-2.9.3  ? or there is not just only one ?
 
 
Date: 2013-12-30 08:36
Subject: RE: Errors with spark-0.8.1 hadoop-yarn 2.2.0
Hi Izhar
 
Is that the exact command you are running? Say with 0.8.0 instead of 0.8.1 in the cmd?
 
Raymond Liu
 
From: Izhar ul Hassan [mailto:[hidden email]]
Sent: Friday, December 27, 2013 9:40 PM
Subject: Errors with spark-0.8.1 hadoop-yarn 2.2.0
 
Hi,
I have a 3 node installation of hadoop 2.2.0 with yarn. I have installed spark-0.8.1 with support for spark enabled. I get the following errors when trying to run the examples:
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar \
    ./spark-class org.apache.spark.deploy.yarn.Client \
      --jar examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar \
      --class org.apache.spark.examples.SparkPi \
      --args yarn-standalone \
      --num-workers 3 \
      --master-memory 4g \
      --worker-memory 2g \
      --worker-cores 1
 
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/Client
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.Client
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.deploy.yarn.Client. Program will exit.
spark-0.8.0 with hadooop 2.0.5-alpha works fine.
--
/Izhar 
Free forum by Nabble Edit this page