Running spark on cluster

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Running spark on cluster

Mohit Singh
Hi,
    I am trying to run spark on a standalone cluster mode..
But I am not sure whether it is executing on cluster at all?
So basically, I have 8 node cluster
master
node01
node02
....
node07
Spark is installed in master @ ~/spark
I have copied ~/spark across all the nodes..
And then I am trying to use pyspark shell

I load a 10 GB file..present on my master disk (not hdfs etc)..
and try to count the number of lines:
I see this:
14/02/20 12:54:29 INFO TaskSetManager: Finished TID 420 in 233 ms on localhost (progress: 420/922)
14/02/20 12:54:29 INFO DAGScheduler: Completed ResultTask(0, 420)
14/02/20 12:54:29 INFO BlockManager: Found block broadcast_0 locally
14/02/20 12:54:29 INFO HadoopRDD: Input split: file:/home/hadoop/data/backup/data/domain/domainz0:14126415872+33554432
.....


I am starting pyspark shell as MASTER=spark://master:7070 ./pyspark

Though when I start pyspark.. I do see this error:
14/02/20 12:59:45 INFO HttpBroadcast: Broadcast server started at http://10.2.1.18:53329
14/02/20 12:59:45 INFO SparkEnv: Registering MapOutputTracker
14/02/20 12:59:45 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c0b1aebb-1995-4849-873f-eb1a534b4af2
14/02/20 12:59:45 INFO HttpServer: Starting HTTP Server
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@37a8d4d2: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)

02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)
14/02/20 12:59:45 INFO SparkUI: Started Spark Web UI at http://master:4041
14/02/20 12:59:45 INFO AppClient$ClientActor: Connecting to master spark://master:7070...
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]



And after all this thing.. pyspark shell starts..
And if kill 4040 port,, then things work just fine.
Can someone help me with this
Thanks


--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates
Reply | Threaded
Open this post in threaded view
|

Re: Running spark on cluster

Mayur Rustagi
netstat -an |grep 4040 check ports
In all likelyhood you are running spark shell or something in the background.




On Thu, Feb 20, 2014 at 1:43 PM, Mohit Singh <[hidden email]> wrote:
Hi,
    I am trying to run spark on a standalone cluster mode..
But I am not sure whether it is executing on cluster at all?
So basically, I have 8 node cluster
master
node01
node02
....
node07
Spark is installed in master @ ~/spark
I have copied ~/spark across all the nodes..
And then I am trying to use pyspark shell

I load a 10 GB file..present on my master disk (not hdfs etc)..
and try to count the number of lines:
I see this:
14/02/20 12:54:29 INFO TaskSetManager: Finished TID 420 in 233 ms on localhost (progress: 420/922)
14/02/20 12:54:29 INFO DAGScheduler: Completed ResultTask(0, 420)
14/02/20 12:54:29 INFO BlockManager: Found block broadcast_0 locally
14/02/20 12:54:29 INFO HadoopRDD: Input split: file:/home/hadoop/data/backup/data/domain/domainz0:14126415872+33554432
.....


I am starting pyspark shell as MASTER=spark://master:7070 ./pyspark

Though when I start pyspark.. I do see this error:
14/02/20 12:59:45 INFO HttpBroadcast: Broadcast server started at http://10.2.1.18:53329
14/02/20 12:59:45 INFO SparkEnv: Registering MapOutputTracker
14/02/20 12:59:45 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c0b1aebb-1995-4849-873f-eb1a534b4af2
14/02/20 12:59:45 INFO HttpServer: Starting HTTP Server
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@37a8d4d2: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)

02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)
14/02/20 12:59:45 INFO SparkUI: Started Spark Web UI at http://master:4041
14/02/20 12:59:45 INFO AppClient$ClientActor: Connecting to master spark://master:7070...
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]



And after all this thing.. pyspark shell starts..
And if kill 4040 port,, then things work just fine.
Can someone help me with this
Thanks


--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates

Reply | Threaded
Open this post in threaded view
|

Re: Running spark on cluster

Mohit Singh
Cool. That checked out..
But I still strongly feel that my simple line code job is running locally and not on cluster?
Any suggestions.
Thanks


On Thu, Feb 20, 2014 at 1:45 PM, Mayur Rustagi <[hidden email]> wrote:
netstat -an |grep 4040 check ports
In all likelyhood you are running spark shell or something in the background.


Mayur Rustagi
Ph: <a href="tel:%2B919632149971" value="+919632149971" target="_blank">+919632149971


On Thu, Feb 20, 2014 at 1:43 PM, Mohit Singh <[hidden email]> wrote:
Hi,
    I am trying to run spark on a standalone cluster mode..
But I am not sure whether it is executing on cluster at all?
So basically, I have 8 node cluster
master
node01
node02
....
node07
Spark is installed in master @ ~/spark
I have copied ~/spark across all the nodes..
And then I am trying to use pyspark shell

I load a 10 GB file..present on my master disk (not hdfs etc)..
and try to count the number of lines:
I see this:
14/02/20 12:54:29 INFO TaskSetManager: Finished TID 420 in 233 ms on localhost (progress: 420/922)
14/02/20 12:54:29 INFO DAGScheduler: Completed ResultTask(0, 420)
14/02/20 12:54:29 INFO BlockManager: Found block broadcast_0 locally
14/02/20 12:54:29 INFO HadoopRDD: Input split: file:/home/hadoop/data/backup/data/domain/domainz0:<a href="tel:14126415872" value="+14126415872" target="_blank">14126415872+33554432
.....


I am starting pyspark shell as MASTER=spark://master:7070 ./pyspark

Though when I start pyspark.. I do see this error:
14/02/20 12:59:45 INFO HttpBroadcast: Broadcast server started at http://10.2.1.18:53329
14/02/20 12:59:45 INFO SparkEnv: Registering MapOutputTracker
14/02/20 12:59:45 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c0b1aebb-1995-4849-873f-eb1a534b4af2
14/02/20 12:59:45 INFO HttpServer: Starting HTTP Server
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@37a8d4d2: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)

02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)
14/02/20 12:59:45 INFO SparkUI: Started Spark Web UI at http://master:4041
14/02/20 12:59:45 INFO AppClient$ClientActor: Connecting to master spark://master:7070...
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]



And after all this thing.. pyspark shell starts..
And if kill 4040 port,, then things work just fine.
Can someone help me with this
Thanks


--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates




--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates
Reply | Threaded
Open this post in threaded view
|

Re: Running spark on cluster

Mayur Rustagi
check the spark UI, you'll see in the application area if its working there else its running locally. 



On Thu, Feb 20, 2014 at 2:39 PM, Mohit Singh <[hidden email]> wrote:
Cool. That checked out..
But I still strongly feel that my simple line code job is running locally and not on cluster?
Any suggestions.
Thanks


On Thu, Feb 20, 2014 at 1:45 PM, Mayur Rustagi <[hidden email]> wrote:
netstat -an |grep 4040 check ports
In all likelyhood you are running spark shell or something in the background.


Mayur Rustagi
Ph: <a href="tel:%2B919632149971" value="+919632149971" target="_blank">+919632149971


On Thu, Feb 20, 2014 at 1:43 PM, Mohit Singh <[hidden email]> wrote:
Hi,
    I am trying to run spark on a standalone cluster mode..
But I am not sure whether it is executing on cluster at all?
So basically, I have 8 node cluster
master
node01
node02
....
node07
Spark is installed in master @ ~/spark
I have copied ~/spark across all the nodes..
And then I am trying to use pyspark shell

I load a 10 GB file..present on my master disk (not hdfs etc)..
and try to count the number of lines:
I see this:
14/02/20 12:54:29 INFO TaskSetManager: Finished TID 420 in 233 ms on localhost (progress: 420/922)
14/02/20 12:54:29 INFO DAGScheduler: Completed ResultTask(0, 420)
14/02/20 12:54:29 INFO BlockManager: Found block broadcast_0 locally
14/02/20 12:54:29 INFO HadoopRDD: Input split: file:/home/hadoop/data/backup/data/domain/domainz0:<a href="tel:14126415872" value="+14126415872" target="_blank">14126415872+33554432
.....


I am starting pyspark shell as MASTER=spark://master:7070 ./pyspark

Though when I start pyspark.. I do see this error:
14/02/20 12:59:45 INFO HttpBroadcast: Broadcast server started at http://10.2.1.18:53329
14/02/20 12:59:45 INFO SparkEnv: Registering MapOutputTracker
14/02/20 12:59:45 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c0b1aebb-1995-4849-873f-eb1a534b4af2
14/02/20 12:59:45 INFO HttpServer: Starting HTTP Server
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@37a8d4d2: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)

02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)
14/02/20 12:59:45 INFO SparkUI: Started Spark Web UI at http://master:4041
14/02/20 12:59:45 INFO AppClient$ClientActor: Connecting to master spark://master:7070...
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]



And after all this thing.. pyspark shell starts..
And if kill 4040 port,, then things work just fine.
Can someone help me with this
Thanks


--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates




--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates

Reply | Threaded
Open this post in threaded view
|

Re: Running spark on cluster

Mohit Singh
Ok.. That issue is fixed. I didnt set the master and port on spark-context...
Now after setting things look alright..but now it is stuck at:
14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 918)
14/02/20 14:47:15 INFO TaskSetManager: Finished TID 919 in 563 ms on master (progress: 690/922)
14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 919)
14/02/20 14:47:15 INFO TaskSetManager: Finished TID 920 in 558 ms on master (progress: 691/922)
14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 920)
14/02/20 14:47:15 INFO TaskSetManager: Finished TID 916 in 770 ms on master (progress: 692/922)
14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 916)


And I cant access the web ui because of corporate fire wall etc..
Any more ideas.
Thanks


On Thu, Feb 20, 2014 at 2:41 PM, Mayur Rustagi <[hidden email]> wrote:
check the spark UI, you'll see in the application area if its working there else its running locally. 

Mayur Rustagi
Ph: <a href="tel:%2B919632149971" value="+919632149971" target="_blank">+919632149971


On Thu, Feb 20, 2014 at 2:39 PM, Mohit Singh <[hidden email]> wrote:
Cool. That checked out..
But I still strongly feel that my simple line code job is running locally and not on cluster?
Any suggestions.
Thanks


On Thu, Feb 20, 2014 at 1:45 PM, Mayur Rustagi <[hidden email]> wrote:
netstat -an |grep 4040 check ports
In all likelyhood you are running spark shell or something in the background.


Mayur Rustagi
Ph: <a href="tel:%2B919632149971" value="+919632149971" target="_blank">+919632149971


On Thu, Feb 20, 2014 at 1:43 PM, Mohit Singh <[hidden email]> wrote:
Hi,
    I am trying to run spark on a standalone cluster mode..
But I am not sure whether it is executing on cluster at all?
So basically, I have 8 node cluster
master
node01
node02
....
node07
Spark is installed in master @ ~/spark
I have copied ~/spark across all the nodes..
And then I am trying to use pyspark shell

I load a 10 GB file..present on my master disk (not hdfs etc)..
and try to count the number of lines:
I see this:
14/02/20 12:54:29 INFO TaskSetManager: Finished TID 420 in 233 ms on localhost (progress: 420/922)
14/02/20 12:54:29 INFO DAGScheduler: Completed ResultTask(0, 420)
14/02/20 12:54:29 INFO BlockManager: Found block broadcast_0 locally
14/02/20 12:54:29 INFO HadoopRDD: Input split: file:/home/hadoop/data/backup/data/domain/domainz0:<a href="tel:14126415872" value="+14126415872" target="_blank">14126415872+33554432
.....


I am starting pyspark shell as MASTER=spark://master:7070 ./pyspark

Though when I start pyspark.. I do see this error:
14/02/20 12:59:45 INFO HttpBroadcast: Broadcast server started at http://10.2.1.18:53329
14/02/20 12:59:45 INFO SparkEnv: Registering MapOutputTracker
14/02/20 12:59:45 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c0b1aebb-1995-4849-873f-eb1a534b4af2
14/02/20 12:59:45 INFO HttpServer: Starting HTTP Server
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@37a8d4d2: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)

02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)
14/02/20 12:59:45 INFO SparkUI: Started Spark Web UI at http://master:4041
14/02/20 12:59:45 INFO AppClient$ClientActor: Connecting to master spark://master:7070...
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]



And after all this thing.. pyspark shell starts..
And if kill 4040 port,, then things work just fine.
Can someone help me with this
Thanks


--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates




--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates




--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates
Reply | Threaded
Open this post in threaded view
|

Re: Running spark on cluster

Mayur Rustagi
lynx for accessing webui on console :) 



On Thu, Feb 20, 2014 at 2:52 PM, Mohit Singh <[hidden email]> wrote:
Ok.. That issue is fixed. I didnt set the master and port on spark-context...
Now after setting things look alright..but now it is stuck at:
14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 918)
14/02/20 14:47:15 INFO TaskSetManager: Finished TID 919 in 563 ms on master (progress: 690/922)
14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 919)
14/02/20 14:47:15 INFO TaskSetManager: Finished TID 920 in 558 ms on master (progress: 691/922)
14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 920)
14/02/20 14:47:15 INFO TaskSetManager: Finished TID 916 in 770 ms on master (progress: 692/922)
14/02/20 14:47:15 INFO DAGScheduler: Completed ResultTask(0, 916)


And I cant access the web ui because of corporate fire wall etc..
Any more ideas.
Thanks


On Thu, Feb 20, 2014 at 2:41 PM, Mayur Rustagi <[hidden email]> wrote:
check the spark UI, you'll see in the application area if its working there else its running locally. 

Mayur Rustagi
Ph: <a href="tel:%2B919632149971" value="+919632149971" target="_blank">+919632149971


On Thu, Feb 20, 2014 at 2:39 PM, Mohit Singh <[hidden email]> wrote:
Cool. That checked out..
But I still strongly feel that my simple line code job is running locally and not on cluster?
Any suggestions.
Thanks


On Thu, Feb 20, 2014 at 1:45 PM, Mayur Rustagi <[hidden email]> wrote:
netstat -an |grep 4040 check ports
In all likelyhood you are running spark shell or something in the background.


Mayur Rustagi
Ph: <a href="tel:%2B919632149971" value="+919632149971" target="_blank">+919632149971


On Thu, Feb 20, 2014 at 1:43 PM, Mohit Singh <[hidden email]> wrote:
Hi,
    I am trying to run spark on a standalone cluster mode..
But I am not sure whether it is executing on cluster at all?
So basically, I have 8 node cluster
master
node01
node02
....
node07
Spark is installed in master @ ~/spark
I have copied ~/spark across all the nodes..
And then I am trying to use pyspark shell

I load a 10 GB file..present on my master disk (not hdfs etc)..
and try to count the number of lines:
I see this:
14/02/20 12:54:29 INFO TaskSetManager: Finished TID 420 in 233 ms on localhost (progress: 420/922)
14/02/20 12:54:29 INFO DAGScheduler: Completed ResultTask(0, 420)
14/02/20 12:54:29 INFO BlockManager: Found block broadcast_0 locally
14/02/20 12:54:29 INFO HadoopRDD: Input split: file:/home/hadoop/data/backup/data/domain/domainz0:<a href="tel:14126415872" value="+14126415872" target="_blank">14126415872+33554432
.....


I am starting pyspark shell as MASTER=spark://master:7070 ./pyspark

Though when I start pyspark.. I do see this error:
14/02/20 12:59:45 INFO HttpBroadcast: Broadcast server started at http://10.2.1.18:53329
14/02/20 12:59:45 INFO SparkEnv: Registering MapOutputTracker
14/02/20 12:59:45 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c0b1aebb-1995-4849-873f-eb1a534b4af2
14/02/20 12:59:45 INFO HttpServer: Starting HTTP Server
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 WARN AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@37a8d4d2: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:286)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
    at org.apache.spark.ui.SparkUI.bind(SparkUI.scala:57)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:159)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:47)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:214)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
14/02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)

02/20 12:59:45 INFO JettyUtils: Failed to create UI at port, 4040. Trying again.
14/02/20 12:59:45 INFO JettyUtils: Error was: Failure(java.net.BindException: Address already in use)
14/02/20 12:59:45 INFO SparkUI: Started Spark Web UI at http://master:4041
14/02/20 12:59:45 INFO AppClient$ClientActor: Connecting to master spark://master:7070...
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]
14/02/20 12:59:46 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@master:7070: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@master:7070]



And after all this thing.. pyspark shell starts..
And if kill 4040 port,, then things work just fine.
Can someone help me with this
Thanks


--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates




--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates




--
Mohit

"When you want success as badly as you want the air, then you will get it. There is no other secret of success."
-Socrates