Problem running example GroupByTest from scala command line

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem running example GroupByTest from scala command line

Shing Hing Man
Hi,

  I am using the development version of Spark from
git://github.com/apache/incubator-spark.git
with Scala 2.10.3.


The example GroupByTest runs successfully using :
matmsh@gauss:~/Downloads/spark/github/incubator-spark> bin/run-example org.apache.spark.examples.GroupByTest local


The script  bin/run-example  essentially does the following.
1) set up SPARK_HOME
2) set up classpath

3) execute "java -cp $CLASSPATH org.apache.spark.examples.GroupByTest local"


I would have thought if (3) above is replaced by
      scala -cp $CLASSPATH org.apache.spark.examples.GroupByTest local
it should still works. But it does not. Please see below.
 
matmsh@gauss:~/Downloads/spark/github/incubator-spark> echo $SPARK_HOME
/home/matmsh/Downloads/spark/github/incubator-spark
matmsh@gauss:~/Downloads/spark/github/incubator-spark>

matmsh@gauss:~/Downloads/spark/github/incubator-spark> export CLASSPATH=/home/matmsh/Downloads/spark/github/incubator-spark/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar:/home/matmsh/Downloads/spark/github/incubator-spark/conf:/home/matmsh/Downloads/spark/github/incubator-spark/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar

matmsh@gauss:~/Downloads/spark/github/incubator-spark> scala  -cp $CLASSPATH    org.apache.spark.examples.GroupByTest local
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/matmsh/Downloads/spark/github/incubator-spark/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/matmsh/Downloads/spark/github/incubator-spark/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/01/12 12:01:25 INFO Utils: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/01/12 12:01:25 WARN Utils: Your hostname, gauss.site resolves to a loopback address: 127.0.0.2; using 192.168.0.10 instead (on interface eth0)
14/01/12 12:01:25 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
java.lang.NoSuchMethodException: akka.remote.RemoteActorRefProvider.<init>(java.lang.String, akka.actor.ActorSystem$Settings, akka.event.EventStream, akka.actor.Scheduler, akka.actor.DynamicAccess)
at java.lang.Class.getConstructor0(Class.java:2721)
at java.lang.Class.getDeclaredConstructor(Class.java:2002)
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:77)
at scala.util.Try$.apply(Try.scala:161)
at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:74)
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:85)
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:85)
at scala.util.Success.flatMap(Try.scala:200)
at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:85)
at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:546)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:96)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:100)
at org.apache.spark.examples.GroupByTest$.main(GroupByTest.scala:36)
at org.apache.spark.examples.GroupByTest.main(GroupByTest.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:71)
at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:139)
at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:71)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:139)
at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:28)
at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:45)
at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:35)
at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:45)
at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:74)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:96)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:105)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
matmsh@gauss:~/Downloads/spark/github/incubator-spark>

What is wrong with the following ?

 scala -cp $CLASSPATH org.apache.spark.examples.GroupByTest local


Thanks in advance for any assistance !

Shing
Reply | Threaded
Open this post in threaded view
|

Re: Problem running example GroupByTest from scala command line

Patrick Wendell
You should launch with "java" and not "scala" to launch. The "scala"
command in newer versions manually adds a specific version of akka to
the classpath which conflicts with the version spark is using. This
causes the error you are seeing. It's discussed in this thread on the
dev list:

http://apache-spark-developers-list.1001551.n3.nabble.com/Akka-problem-when-using-scala-command-to-launch-Spark-applications-in-the-current-0-9-0-SNAPSHOT-tp2.html

- Patrick

On Sun, Jan 12, 2014 at 4:21 AM, Shing Hing Man <[hidden email]> wrote:

> Hi,
>
>   I am using the development version of Spark from
> git://github.com/apache/incubator-spark.git
> with Scala 2.10.3.
>
>
> The example GroupByTest runs successfully using :
> matmsh@gauss:~/Downloads/spark/github/incubator-spark> bin/run-example org.apache.spark.examples.GroupByTest local
>
>
> The script  bin/run-example  essentially does the following.
> 1) set up SPARK_HOME
> 2) set up classpath
>
> 3) execute "java -cp $CLASSPATH org.apache.spark.examples.GroupByTest local"
>
>
> I would have thought if (3) above is replaced by
>       scala -cp $CLASSPATH org.apache.spark.examples.GroupByTest local
> it should still works. But it does not. Please see below.
>
> matmsh@gauss:~/Downloads/spark/github/incubator-spark> echo $SPARK_HOME
> /home/matmsh/Downloads/spark/github/incubator-spark
> matmsh@gauss:~/Downloads/spark/github/incubator-spark>
>
> matmsh@gauss:~/Downloads/spark/github/incubator-spark> export CLASSPATH=/home/matmsh/Downloads/spark/github/incubator-spark/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar:/home/matmsh/Downloads/spark/github/incubator-spark/conf:/home/matmsh/Downloads/spark/github/incubator-spark/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar
>
> matmsh@gauss:~/Downloads/spark/github/incubator-spark> scala  -cp $CLASSPATH    org.apache.spark.examples.GroupByTest local
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/home/matmsh/Downloads/spark/github/incubator-spark/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/home/matmsh/Downloads/spark/github/incubator-spark/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 14/01/12 12:01:25 INFO Utils: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> 14/01/12 12:01:25 WARN Utils: Your hostname, gauss.site resolves to a loopback address: 127.0.0.2; using 192.168.0.10 instead (on interface eth0)
> 14/01/12 12:01:25 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
> java.lang.NoSuchMethodException: akka.remote.RemoteActorRefProvider.<init>(java.lang.String, akka.actor.ActorSystem$Settings, akka.event.EventStream, akka.actor.Scheduler, akka.actor.DynamicAccess)
> at java.lang.Class.getConstructor0(Class.java:2721)
> at java.lang.Class.getDeclaredConstructor(Class.java:2002)
> at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:77)
> at scala.util.Try$.apply(Try.scala:161)
> at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:74)
> at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:85)
> at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:85)
> at scala.util.Success.flatMap(Try.scala:200)
> at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:85)
> at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:546)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
> at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:96)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:100)
> at org.apache.spark.examples.GroupByTest$.main(GroupByTest.scala:36)
> at org.apache.spark.examples.GroupByTest.main(GroupByTest.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:71)
> at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
> at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:139)
> at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:71)
> at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:139)
> at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:28)
> at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:45)
> at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:35)
> at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:45)
> at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:74)
> at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:96)
> at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:105)
> at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
> matmsh@gauss:~/Downloads/spark/github/incubator-spark>
>
> What is wrong with the following ?
>
>  scala -cp $CLASSPATH org.apache.spark.examples.GroupByTest local
>
>
> Thanks in advance for any assistance !
>
> Shing
Reply | Threaded
Open this post in threaded view
|

Re: Problem running example GroupByTest from scala command line

Shing Hing Man
Thanks for the explanation !

Shing




On Monday, January 13, 2014 1:28 AM, Patrick Wendell <[hidden email]> wrote:
You should launch with "java" and not "scala" to launch. The "scala"
command in newer versions manually adds a specific version of akka to
the classpath which conflicts with the version spark is using. This
causes the error you are seeing. It's discussed in this thread on the
dev list:

http://apache-spark-developers-list.1001551.n3.nabble.com/Akka-problem-when-using-scala-command-to-launch-Spark-applications-in-the-current-0-9-0-SNAPSHOT-tp2.html

- Patrick


On Sun, Jan 12, 2014 at 4:21 AM, Shing Hing Man <[hidden email]> wrote:

> Hi,
>
>   I am using the development version of Spark from
> git://github.com/apache/incubator-spark.git
> with Scala 2.10.3.
>
>
> The example GroupByTest runs successfully using :
> matmsh@gauss:~/Downloads/spark/github/incubator-spark> bin/run-example org.apache.spark.examples.GroupByTest local
>
>
> The script  bin/run-example  essentially does the following.
> 1) set up SPARK_HOME
> 2) set up classpath
>
> 3) execute "java -cp $CLASSPATH org.apache.spark.examples.GroupByTest local"
>
>
> I would have thought if (3) above is replaced by
>       scala -cp $CLASSPATH org.apache.spark.examples.GroupByTest local
> it should still works. But it does not. Please see below.
>
> matmsh@gauss:~/Downloads/spark/github/incubator-spark> echo $SPARK_HOME
> /home/matmsh/Downloads/spark/github/incubator-spark
> matmsh@gauss:~/Downloads/spark/github/incubator-spark>
>
> matmsh@gauss:~/Downloads/spark/github/incubator-spark> export CLASSPATH=/home/matmsh/Downloads/spark/github/incubator-spark/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar:/home/matmsh/Downloads/spark/github/incubator-spark/conf:/home/matmsh/Downloads/spark/github/incubator-spark/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar
>
> matmsh@gauss:~/Downloads/spark/github/incubator-spark> scala  -cp $CLASSPATH    org.apache.spark.examples.GroupByTest local
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/home/matmsh/Downloads/spark/github/incubator-spark/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/home/matmsh/Downloads/spark/github/incubator-spark/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 14/01/12 12:01:25 INFO Utils: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> 14/01/12 12:01:25 WARN Utils: Your hostname, gauss.site resolves to a loopback address: 127.0.0.2; using 192.168.0.10 instead (on interface eth0)
> 14/01/12 12:01:25 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
> java.lang.NoSuchMethodException: akka.remote.RemoteActorRefProvider.<init>(java.lang.String, akka.actor.ActorSystem$Settings, akka.event.EventStream, akka.actor.Scheduler, akka.actor.DynamicAccess)
> at java.lang.Class.getConstructor0(Class.java:2721)
> at java.lang.Class.getDeclaredConstructor(Class.java:2002)
> at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:77)
> at scala.util.Try$.apply(Try.scala:161)
> at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:74)
> at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:85)
> at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:85)
> at scala.util.Success.flatMap(Try.scala:200)
> at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:85)
> at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:546)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
> at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:96)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:100)
> at org.apache.spark.examples.GroupByTest$.main(GroupByTest.scala:36)
> at org.apache.spark.examples.GroupByTest.main(GroupByTest.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:71)
> at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
> at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:139)
> at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:71)
> at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:139)
> at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:28)
> at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:45)
> at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:35)
> at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:45)
> at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:74)
> at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:96)
> at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:105)
> at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
> matmsh@gauss:~/Downloads/spark/github/incubator-spark>
>
> What is wrong with the following ?
>
>  scala -cp $CLASSPATH org.apache.spark.examples.GroupByTest local
>
>
> Thanks in advance for any assistance !
>
> Shing