(no subject)

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

(no subject)

Gino Mathews

Hi,

I am new to spark.
I would like to run jobs  in Spark stand alone cluster mode.

No cluser managers other than psark is used.
I have tried wordcount from spark shell and stand alone scala app.

The code reads input from HDFS and writes the results to HDFS. uses 2 worker nodes.

In shell the wordcount is successful, how ever my tries to run stand alone programmes are in vain.


My environement

Ubuntu 12.04  - 32 bit

I have installed spark @ $HOME/Downloads/spark-0.9.0-incubating
installed hadoop 2.2.0 as separate hduser and given permission to other users.
installed scala 2.10.3
installed sbt 0.13.1

Spark master act as HDFS master
I have one master and 2 worker nodes and HDFS is accessible in all nodes.
I downloaded example project and modified to use my spark cluster.
I started the sparkcluster at spark://192.168.0.138:7077

When I run the project as SPARK_HADOOP_VERSION=2.2.0 sbt run, I get following error

gino@master:~/Test/spark-example-project$ SPARK_HADOOP_VERSION=2.2.0 sbt run
[info] Loading project definition from /home/gino/Test/spark-example-project/project
[info] Set current project to spark-example-project (in build file:/home/gino/Test/spark-example-project/)
[info] Running com.Thinkpalm.spark.WordCountHDFS
[error] (run-main-0) java.lang.NoClassDefFoundError: org/apache/spark/SparkContext
java.lang.NoClassDefFoundError: org/apache/spark/SparkContext
    at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12)
    at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkContext
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at com.Thinkpalm.spark.WordCountHDFS$.main(WordCountHDFS.scala:12)
    at com.Thinkpalm.spark.WordCountHDFS.main(WordCountHDFS.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
    at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 0 s, completed Mar 11, 2014 2:54:54 PM

Could anyone give some pointers ... I have attached the project for reference.


Thanks and regards

Gino Mathews


spark-example-project.tgz (9K) Download Attachment