Please help running a standalone app on a Spark cluster

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Please help running a standalone app on a Spark cluster

ssimanta
Hi, 

I've been trying to resolve this issue for the past couple of weeks without any success. I had given up running a standalone app because I was able to use the REPL for most of my work but now I really need to run an app. This is becoming a potential showstopper for me in using Spark for my use case. 

My application is very simple. It reads a file from HDFS and does a count on the RDD. 

I believe my cluster is setup correctly because I can connect to it from my REPL and I can see all the worker nodes on the Spark UI. Is there any other test or way of figuring out if there is an issue my cluster setup. I'll try to provide as much detail as possible before. I've asked this question on the list before but I'm asking it again with more relevant details (all in one place). 


Here is the exception. 

14/02/16 21:45:59 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
14/02/16 21:46:13 ERROR client.AppClient$ClientActor: All masters are unresponsive! Giving up.
14/02/16 21:46:13 ERROR cluster.SparkDeploySchedulerBackend: Spark cluster looks dead, giving up.
14/02/16 21:46:13 INFO scheduler.TaskSchedulerImpl: Remove TaskSet 0.0 from pool 
14/02/16 21:46:13 INFO scheduler.DAGScheduler: Failed to run count at SimpleApp.scala:24
[error] (run-main) org.apache.spark.SparkException: Job aborted: Spark cluster looks down
org.apache.spark.SparkException: Job aborted: Spark cluster looks down
at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1026)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:619)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)


My directory structure

$find .
.
./sbt
./simple.sbt
./sbt-launch-0.12.4.jar
./src
./src/main
./src/main/scala
./src/main/scala/SimpleApp.scala

Simple.sbt 

name := "Simple Project"
version := "1.0"
scalaVersion := "2.10.0"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "0.9.0-incubating_SNAPSHOT" //This lib is build on my local version of Spark and taken from my ~/.m2/repository 
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "0.20.2-cdh3u6"
resolvers ++= Seq("Akka Repository" at "http://repo.akka.io/releases/","Spray Repository" at "http://repo.spray.cc/",  "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository")

Simple.scala

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp{

        def main(args: Array[String]){

                val conf = new SparkConf()
                                .setMaster("spark://aa.bb.xxx.yy:7077")
                                .setAppName("Simple Analysis")
                                .set("spark.executor.memory", "4g")

                val sc = new SparkContext(conf)

                println(" -- Starting analysis --")
                val textFile = sc.textFile("hdfs://xxx.xxx.xxx.xxx:54310/tmp/data/error.log")

                println(" -- Done reading file from HDFS --")
                textFile.cache
                println(textFile.count)


        }
}

Please note that Spark master URL above - spark://aa.bb.xxx.yy:7077 is exactly the same that shows up on my Spark Web UI. 
Also, I use the following to connect to using the Spark shell and it works fine. I can also run the same code above on the REPL without any exceptions. 

MASTER="spark://aa.bb.xxx.yy:7077" SPARK_MEM="24g" ./spark-shell
 

Any help to resolve this would be greatly appreciated and very helpful. 


Thanks

-Soumya 



Reply | Threaded
Open this post in threaded view
|

Re: Please help running a standalone app on a Spark cluster

David Thomas
I see that you have set spark.executor.memory as 4g. What is the memory available on each node? Check the last column under 'Workers' table in masterip:8080.


On Sun, Feb 16, 2014 at 8:18 PM, Soumya Simanta <[hidden email]> wrote:
Hi, 

I've been trying to resolve this issue for the past couple of weeks without any success. I had given up running a standalone app because I was able to use the REPL for most of my work but now I really need to run an app. This is becoming a potential showstopper for me in using Spark for my use case. 

My application is very simple. It reads a file from HDFS and does a count on the RDD. 

I believe my cluster is setup correctly because I can connect to it from my REPL and I can see all the worker nodes on the Spark UI. Is there any other test or way of figuring out if there is an issue my cluster setup. I'll try to provide as much detail as possible before. I've asked this question on the list before but I'm asking it again with more relevant details (all in one place). 


Here is the exception. 

14/02/16 21:45:59 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
14/02/16 21:46:13 ERROR client.AppClient$ClientActor: All masters are unresponsive! Giving up.
14/02/16 21:46:13 ERROR cluster.SparkDeploySchedulerBackend: Spark cluster looks dead, giving up.
14/02/16 21:46:13 INFO scheduler.TaskSchedulerImpl: Remove TaskSet 0.0 from pool 
14/02/16 21:46:13 INFO scheduler.DAGScheduler: Failed to run count at SimpleApp.scala:24
[error] (run-main) org.apache.spark.SparkException: Job aborted: Spark cluster looks down
org.apache.spark.SparkException: Job aborted: Spark cluster looks down
at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1026)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:619)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)


My directory structure

$find .
.
./sbt
./simple.sbt
./sbt-launch-0.12.4.jar
./src
./src/main
./src/main/scala
./src/main/scala/SimpleApp.scala

Simple.sbt 

name := "Simple Project"
version := "1.0"
scalaVersion := "2.10.0"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "0.9.0-incubating_SNAPSHOT" //This lib is build on my local version of Spark and taken from my ~/.m2/repository 
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "0.20.2-cdh3u6"
resolvers ++= Seq("Akka Repository" at "http://repo.akka.io/releases/","Spray Repository" at "http://repo.spray.cc/",  "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository")

Simple.scala

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp{

        def main(args: Array[String]){

                val conf = new SparkConf()
                                .setMaster("spark://aa.bb.xxx.yy:7077")
                                .setAppName("Simple Analysis")
                                .set("spark.executor.memory", "4g")

                val sc = new SparkContext(conf)

                println(" -- Starting analysis --")
                val textFile = sc.textFile("hdfs://xxx.xxx.xxx.xxx:54310/tmp/data/error.log")

                println(" -- Done reading file from HDFS --")
                textFile.cache
                println(textFile.count)


        }
}

Please note that Spark master URL above - spark://aa.bb.xxx.yy:7077 is exactly the same that shows up on my Spark Web UI. 
Also, I use the following to connect to using the Spark shell and it works fine. I can also run the same code above on the REPL without any exceptions. 

MASTER="spark://aa.bb.xxx.yy:7077" SPARK_MEM="24g" ./spark-shell
 

Any help to resolve this would be greatly appreciated and very helpful. 


Thanks

-Soumya 




Reply | Threaded
Open this post in threaded view
|

Re: Please help running a standalone app on a Spark cluster

ssimanta
David, 

Please see my answers below. 


On Sun, Feb 16, 2014 at 10:23 PM, David Thomas <[hidden email]> wrote:
I see that you have set spark.executor.memory as 4g.

I added 4g to ensure that I was not getting the error because of insufficient memory as the error message in the log states. 
But I don't think that's the case unless my understanding is fundamentally flawed. 
 
What is the memory available on each node? Check the last column under 'Workers' table in masterip:8080.


188.2 GB (0.0 B Used)
93.5 GB (0.0 B Used)
93.5 GB (0.0 B Used)




Reply | Threaded
Open this post in threaded view
|

Re: Please help running a standalone app on a Spark cluster

ssimanta
One additional detail. 

My started my cluster using the $SPARK_INSTALL/sbin/start-all.sh script. 
All my machines have multiple NICs but I've specified the IPs (e.g., 198.168.1.1 etc) of my slaves in conf/slaves file. 

Also, is there a fundamental difference between connecting from the spark-shell vs connecting from a standalone app. 




On Sun, Feb 16, 2014 at 10:28 PM, Soumya Simanta <[hidden email]> wrote:
David, 

Please see my answers below. 


On Sun, Feb 16, 2014 at 10:23 PM, David Thomas <[hidden email]> wrote:
I see that you have set spark.executor.memory as 4g.

I added 4g to ensure that I was not getting the error because of insufficient memory as the error message in the log states. 
But I don't think that's the case unless my understanding is fundamentally flawed. 
 
What is the memory available on each node? Check the last column under 'Workers' table in masterip:8080.


188.2 GB (0.0 B Used)
93.5 GB (0.0 B Used)
93.5 GB (0.0 B Used)





Reply | Threaded
Open this post in threaded view
|

Re: Please help running a standalone app on a Spark cluster

Francesco Bongiovanni
In reply to this post by ssimanta
I think you are missing sending over your jar file containing your app...not sure though but try it...

add this

  val conf = new SparkConf()
                                .setMaster("spark://aa.bb.xxx.yy:7077")
                                .setAppName("Simple Analysis")
                                .set("spark.executor.memory", "4g")
                                .setJars(Seq("target/scala-2.10/simpleapp_2.10-1.0.jar"))  <= put the correct name of that file

and to have that file, just do a `sbt package`. Then try `sbt run` and that should do it.

Btw...you should stop also your SC at the end of your app with sc.stop()

Reply | Threaded
Open this post in threaded view
|

Re: Please help running a standalone app on a Spark cluster

ssimanta
I added setJars with the correct and complete path to the jar file created by sbt built but I'm still getting the same error. 

Any other ideas about how I should go about debugging this issue? 

Thanks.




On Mon, Feb 17, 2014 at 3:44 AM, Francesco Bongiovanni <[hidden email]> wrote:
I think you are missing sending over your jar file containing your app...not
sure though but try it...

add this

  val conf = new SparkConf()
                                .setMaster("spark://aa.bb.xxx.yy:7077")
                                .setAppName("Simple Analysis")
                                .set("spark.executor.memory", "4g")

.*setJars(Seq("target/scala-2.10/simpleapp_2.10-1.0.jar")) * <= put the
correct name of that file

and to have that file, just do a `sbt package`. Then try `sbt run` and that
should do it.

Btw...you should stop also your SC at the end of your app with sc.stop()





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Please-help-running-a-standalone-app-on-a-Spark-cluster-tp1596p1606.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Please help running a standalone app on a Spark cluster

ssimanta
Again trying my best to figure out what's the root cause for this and looks like it may have something to do with Akka configuration. 
Any help will be greatly appreciated. 

I added the following to my SimpleApp.scala program. 

                val conf = new SparkConf()

                                .setMaster("spark://n001.xxx.yyy.zzz.kkk:7077")

                                .setAppName("Simple Analysis")

                                .set("spark.executor.memory", "4g")

                                .setJars(Seq("/complete/path/target/scala-2.10/simple-project_2.10-1.0.jar"))

                                .set("spark.driver.host", "n001.xxx.yyy.zzz.kkk")

                                .set("spark.driver.port", "10029")


and i get the following exceptions in my $SPARK_HOME/logs/spark-myuserid-org.apache.spark.deploy.master.Master-1-n001.xxx.yyy.zzz.kkk.out




14/02/17 14:27:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:27:43 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-16/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-0#-152177223] was not delivered. [27] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:27:43 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.xx.xxx.xx%3A45148-29#-420458324] was not delivered. [28] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:03 ERROR Remoting: 
java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:58)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:55)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:55)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:73)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:764)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
14/02/17 14:28:03 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:03 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-17/endpointReader-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-0#2062518114] was not delivered. [29] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:03 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.xx.xxx.xx%3A45150-31#-790914039] was not delivered. [30] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:23 ERROR Remoting: 
java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:58)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:55)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:55)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:73)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:764)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
14/02/17 14:28:23 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:23 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-18/endpointReader-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-0#975106594] was not delivered. [31] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:23 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.xx.xxx.xx%3A45152-32#-426826615] was not delivered. [32] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.xx.xxx.xx%3A45153-33#-2132092495] was not delivered. [33] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:43 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-30#1356488343] was not delivered. [34] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 ERROR EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:10029]: Error [Association failed with [akka.tcp://[hidden email]:10029]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:10029]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: n001.xxx.yyy.zzz.kkk/10.xx.xxx.xx:10029
]
14/02/17 14:28:43 ERROR EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:10029]: Error [Association failed with [akka.tcp://[hidden email]:10029]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:10029]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: n001.xxx.yyy.zzz.kkk/10.xx.xxx.xx:10029
]
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 ERROR EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:10029]: Error [Association failed with [akka.tcp://[hidden email]:10029]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:10029]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: n001.xxx.yyy.zzz.kkk/10.xx.xxx.xx:10029






On Mon, Feb 17, 2014 at 12:05 PM, Soumya Simanta <[hidden email]> wrote:
I added setJars with the correct and complete path to the jar file created by sbt built but I'm still getting the same error. 

Any other ideas about how I should go about debugging this issue? 

Thanks.




On Mon, Feb 17, 2014 at 3:44 AM, Francesco Bongiovanni <[hidden email]> wrote:
I think you are missing sending over your jar file containing your app...not
sure though but try it...

add this

  val conf = new SparkConf()
                                .setMaster("spark://aa.bb.xxx.yy:7077")
                                .setAppName("Simple Analysis")
                                .set("spark.executor.memory", "4g")

.*setJars(Seq("target/scala-2.10/simpleapp_2.10-1.0.jar")) * <= put the
correct name of that file

and to have that file, just do a `sbt package`. Then try `sbt run` and that
should do it.

Btw...you should stop also your SC at the end of your app with sc.stop()





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Please-help-running-a-standalone-app-on-a-Spark-cluster-tp1596p1606.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|

Re: Please help running a standalone app on a Spark cluster

ssimanta
I'm happy to report that I was finally able to figure this out and resolve the issue. I'm summarizing it here in case someone in the future has a similar issue. 


Before getting to the cause let me provide a little more context. 

1. I build Spark for my version of a Hadoop which meant that I had to use that in my Driver (Standalone Scala - SimpleApp.scala ) program. 
2. I could see the workers correctly registered both in the Spark web UI as well as the logs file ($SPARK_HOME/logs/xxx.master.log) 
3. I was able to successfully run the same code from the spark-shell. This meant the workers were deployed correctly and the basic networking was working fine. 

Given the above what was puzzling was were the following two errors in the logs. 

1. java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)

2. akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:10029]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: n001.xxx.yyy.zzz.kkk/10.xx.xxx.xx:10029


Since my master and worker machines have multiple NICs (and IPs) I was not sure if the names were not resolving to the correct IPs by Akka. So I explicitly used IP addresses and ports to ensure domain names resolving to different IP were not the issue. 

After that I did some digging around for the OptionalDataException and in one of the search it was something to do with incorrect Akka version and indeed that was the root cause. 

The scala version in my simple.sbt was 

scalaVersion := "2.10.0". However, the scala version in the $SPARK_HOME/pom.xml is  <scala.version>2.10.3</scala.version>. 

So once I changed the scalaVersion in the simple.sbt to 2.10.3 everything worked. 

scalaVersion := "2.10.3"

This was a really simple fix but unfortunately took a quite a bit of my time. On the positive side now I know a lot more about Spark configuration :) 

-Soumya









On Mon, Feb 17, 2014 at 2:44 PM, Soumya Simanta <[hidden email]> wrote:
Again trying my best to figure out what's the root cause for this and looks like it may have something to do with Akka configuration. 
Any help will be greatly appreciated. 

I added the following to my SimpleApp.scala program. 

                val conf = new SparkConf()

                                .setMaster("spark://n001.xxx.yyy.zzz.kkk:7077")

                                .setAppName("Simple Analysis")

                                .set("spark.executor.memory", "4g")

                                .setJars(Seq("/complete/path/target/scala-2.10/simple-project_2.10-1.0.jar"))

                                .set("spark.driver.host", "n001.xxx.yyy.zzz.kkk")

                                .set("spark.driver.port", "10029")


and i get the following exceptions in my $SPARK_HOME/logs/spark-myuserid-org.apache.spark.deploy.master.Master-1-n001.xxx.yyy.zzz.kkk.out




14/02/17 14:27:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:27:43 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-16/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-0#-152177223] was not delivered. [27] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:27:43 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.xx.xxx.xx%3A45148-29#-420458324] was not delivered. [28] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:03 ERROR Remoting: 
java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:58)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:55)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:55)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:73)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:764)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
14/02/17 14:28:03 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:03 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-17/endpointReader-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-0#2062518114] was not delivered. [29] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:03 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.xx.xxx.xx%3A45150-31#-790914039] was not delivered. [30] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:23 ERROR Remoting: 
java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1369)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1001)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1892)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:58)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:55)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:55)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:73)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:764)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
14/02/17 14:28:23 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:23 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-18/endpointReader-akka.tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-0#975106594] was not delivered. [31] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:23 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.xx.xxx.xx%3A45152-32#-426826615] was not delivered. [32] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.xx.xxx.xx%3A45153-33#-2132092495] was not delivered. [33] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:43 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fspark%40n001.xxx.yyy.zzz.kkk%3A10029-30#1356488343] was not delivered. [34] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 ERROR EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:10029]: Error [Association failed with [akka.tcp://[hidden email]:10029]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:10029]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: n001.xxx.yyy.zzz.kkk/10.xx.xxx.xx:10029
]
14/02/17 14:28:43 ERROR EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:10029]: Error [Association failed with [akka.tcp://[hidden email]:10029]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:10029]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: n001.xxx.yyy.zzz.kkk/10.xx.xxx.xx:10029
]
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 INFO Master: akka.tcp://[hidden email]:10029 got disassociated, removing it.
14/02/17 14:28:43 ERROR EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:10029]: Error [Association failed with [akka.tcp://[hidden email]:10029]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:10029]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: n001.xxx.yyy.zzz.kkk/10.xx.xxx.xx:10029






On Mon, Feb 17, 2014 at 12:05 PM, Soumya Simanta <[hidden email]> wrote:
I added setJars with the correct and complete path to the jar file created by sbt built but I'm still getting the same error. 

Any other ideas about how I should go about debugging this issue? 

Thanks.




On Mon, Feb 17, 2014 at 3:44 AM, Francesco Bongiovanni <[hidden email]> wrote:
I think you are missing sending over your jar file containing your app...not
sure though but try it...

add this

  val conf = new SparkConf()
                                .setMaster("spark://aa.bb.xxx.yy:7077")
                                .setAppName("Simple Analysis")
                                .set("spark.executor.memory", "4g")

.*setJars(Seq("target/scala-2.10/simpleapp_2.10-1.0.jar")) * <= put the
correct name of that file

and to have that file, just do a `sbt package`. Then try `sbt run` and that
should do it.

Btw...you should stop also your SC at the end of your app with sc.stop()





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Please-help-running-a-standalone-app-on-a-Spark-cluster-tp1596p1606.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.