Trouble with EC2

classic Classic list List threaded Threaded
6 messages Options
PJ$
Reply | Threaded
Open this post in threaded view
|

Trouble with EC2

PJ$
Hey Folks, 

I'm really having quite a bit of trouble getting spark running on ec2. I'm not using scripts the https://github.com/apache/spark/tree/master/ec2 because I'd like to know how everything works. But I'm going a little crazy. I think that something about the networking configuration must be messed up, but I'm at a loss. Shortly after starting the cluster, I get a lot of this: 

14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.100.75.70%3A36725-25#847210246] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
Reply | Threaded
Open this post in threaded view
|

Re: Trouble with EC2

Matei Zaharia
Administrator
What instance types did you launch on?

Sometimes you also get a bad individual machine from EC2. It might help to remove the node it’s complaining about from the conf/slaves file.

Matei

On May 30, 2014, at 11:18 AM, PJ$ <[hidden email]> wrote:

Hey Folks, 

I'm really having quite a bit of trouble getting spark running on ec2. I'm not using scripts the https://github.com/apache/spark/tree/master/ec2 because I'd like to know how everything works. But I'm going a little crazy. I think that something about the networking configuration must be messed up, but I'm at a loss. Shortly after starting the cluster, I get a lot of this: 

14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:05:54 INFO master.Master: <a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485">akka.tcp://spark@...:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[<a href="akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.100.75.70%3A36725-25#847210246">akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.100.75.70%3A36725-25#847210246] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/05/30 18:05:54 INFO master.Master: <a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485">akka.tcp://spark@...:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: <a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485">akka.tcp://spark@...:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [<a href="akka.tcp://sparkMaster@ip-10-100-184-45.ec2.internal:7077] -">akka.tcp://sparkMaster@...:7077] -> [<a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485]: Error [Association failed with [akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485]">akka.tcp://spark@...:38485]: Error [Association failed with [akka.tcp://spark@...:38485]] [
akka.remote.EndpointAssociationException: Association failed with [<a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485">akka.tcp://spark@...:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [<a href="akka.tcp://sparkMaster@ip-10-100-184-45.ec2.internal:7077] -">akka.tcp://sparkMaster@...:7077] -> [<a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485]: Error [Association failed with [akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485]">akka.tcp://spark@...:38485]: Error [Association failed with [akka.tcp://spark@...:38485]] [
akka.remote.EndpointAssociationException: Association failed with [<a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485">akka.tcp://spark@...:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 INFO master.Master: <a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485">akka.tcp://spark@...:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: <a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485">akka.tcp://spark@...:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [<a href="akka.tcp://sparkMaster@ip-10-100-184-45.ec2.internal:7077] -">akka.tcp://sparkMaster@...:7077] -> [<a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485]: Error [Association failed with [akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485]">akka.tcp://spark@...:38485]: Error [Association failed with [akka.tcp://spark@...:38485]] [
akka.remote.EndpointAssociationException: Association failed with [<a href="akka.tcp://spark@ip-10-100-75-70.ec2.internal:38485">akka.tcp://spark@...:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485

PJ$
Reply | Threaded
Open this post in threaded view
|

Re: Trouble with EC2

PJ$
Running on a few m3.larges with the ami-848a6eec image (debian 7). Haven't gotten any further. No clue what's wrong. I'd really appreciate any guidance y'all could offer. 

Best, 
PJ$


On Sat, May 31, 2014 at 1:40 PM, Matei Zaharia <[hidden email]> wrote:
What instance types did you launch on?

Sometimes you also get a bad individual machine from EC2. It might help to remove the node it’s complaining about from the conf/slaves file.

Matei

On May 30, 2014, at 11:18 AM, PJ$ <[hidden email]> wrote:

Hey Folks, 

I'm really having quite a bit of trouble getting spark running on ec2. I'm not using scripts the https://github.com/apache/spark/tree/master/ec2 because I'd like to know how everything works. But I'm going a little crazy. I think that something about the networking configuration must be messed up, but I'm at a loss. Shortly after starting the cluster, I get a lot of this: 

14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.100.75.70%3A36725-25#847210246] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485


Reply | Threaded
Open this post in threaded view
|

Re: Trouble with EC2

Matei Zaharia
Administrator
So to run spark-ec2, you should use the default AMI that it launches with if you don’t pass -a. Those are based on Amazon Linux, not Debian. Passing your own AMI is an advanced option but people need to install some stuff on their AMI in advance for it to work with our scripts.

Matei


On Jun 1, 2014, at 3:11 PM, PJ$ <[hidden email]> wrote:

Running on a few m3.larges with the ami-848a6eec image (debian 7). Haven't gotten any further. No clue what's wrong. I'd really appreciate any guidance y'all could offer. 

Best, 
PJ$


On Sat, May 31, 2014 at 1:40 PM, Matei Zaharia <[hidden email]> wrote:
What instance types did you launch on?

Sometimes you also get a bad individual machine from EC2. It might help to remove the node it’s complaining about from the conf/slaves file.

Matei

On May 30, 2014, at 11:18 AM, PJ$ <[hidden email]> wrote:

Hey Folks, 

I'm really having quite a bit of trouble getting spark running on ec2. I'm not using scripts the https://github.com/apache/spark/tree/master/ec2 because I'd like to know how everything works. But I'm going a little crazy. I think that something about the networking configuration must be messed up, but I'm at a loss. Shortly after starting the cluster, I get a lot of this: 

14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.100.75.70%3A36725-25#847210246] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485



Reply | Threaded
Open this post in threaded view
|

Re: Trouble with EC2

Jeremy Lee
In reply to this post by PJ$
Ha yes,,, I just went through this.

(a) You have to use the ;'default' spark AMI, ( ami-7a320f3f at the moment ) and not any of the other linux distros. They don't work.
(b) Start with m1.large instances.. I tried going for r3.large at first, and had no end of self-caused trouble. m1.large works.
(c) It's possible for the script to choose the wrong AMI, especially if one has been messing with it to allow other instance types. (ahem)

But it will work in the end.. just start simple. (yeah, I know m1.large doesn't look that large anymore. :-)


On Mon, Jun 2, 2014 at 8:11 AM, PJ$ <[hidden email]> wrote:
Running on a few m3.larges with the ami-848a6eec image (debian 7). Haven't gotten any further. No clue what's wrong. I'd really appreciate any guidance y'all could offer. 

Best, 
PJ$


On Sat, May 31, 2014 at 1:40 PM, Matei Zaharia <[hidden email]> wrote:
What instance types did you launch on?

Sometimes you also get a bad individual machine from EC2. It might help to remove the node it’s complaining about from the conf/slaves file.

Matei

On May 30, 2014, at 11:18 AM, PJ$ <[hidden email]> wrote:

Hey Folks, 

I'm really having quite a bit of trouble getting spark running on ec2. I'm not using scripts the https://github.com/apache/spark/tree/master/ec2 because I'd like to know how everything works. But I'm going a little crazy. I think that something about the networking configuration must be messed up, but I'm at a loss. Shortly after starting the cluster, I get a lot of this: 

14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.100.75.70%3A36725-25#847210246] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485





--
Jeremy Lee  BCompSci(Hons)
  The Unorthodox Engineers
Reply | Threaded
Open this post in threaded view
|

Re: Trouble with EC2

Stefan van Wouw
In reply to this post by PJ$
Dear PJ$,

If you are familiar with Puppet, you could try using the puppet module I wrote (currently for Spark 0.9.0, I custom compiled it since no Debian package was available at the time I started with a project I required it for).


---
Kind regards,

Stefan van Wouw

On 02 Jun 2014, at 00:11, PJ$ <[hidden email]> wrote:

Running on a few m3.larges with the ami-848a6eec image (debian 7). Haven't gotten any further. No clue what's wrong. I'd really appreciate any guidance y'all could offer. 

Best, 
PJ$


On Sat, May 31, 2014 at 1:40 PM, Matei Zaharia <[hidden email]> wrote:
What instance types did you launch on?

Sometimes you also get a bad individual machine from EC2. It might help to remove the node it’s complaining about from the conf/slaves file.

Matei

On May 30, 2014, at 11:18 AM, PJ$ <[hidden email]> wrote:

Hey Folks, 

I'm really having quite a bit of trouble getting spark running on ec2. I'm not using scripts the https://github.com/apache/spark/tree/master/ec2 because I'd like to know how everything works. But I'm going a little crazy. I think that something about the networking configuration must be messed up, but I'm at a loss. Shortly after starting the cluster, I get a lot of this: 

14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:22 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:03:23 INFO master.Master: Registering worker ip-10-100-184-45.ec2.internal:7078 with 2 cores, 6.3 GB RAM
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.100.75.70%3A36725-25#847210246] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485
]
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 INFO master.Master: akka.tcp://[hidden email]:38485 got disassociated, removing it.
14/05/30 18:05:54 ERROR remote.EndpointWriter: AssociationError [akka.tcp://[hidden email]:7077] -> [akka.tcp://[hidden email]:38485]: Error [Association failed with [akka.tcp://[hidden email]:38485]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[hidden email]:38485]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-100-75-70.ec2.internal/10.100.75.70:38485