Reg: Add/Remove slave nodes spark-ec2

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Reg: Add/Remove slave nodes spark-ec2

sirisha_devineni

Hi All,

 

I have created a spark cluster on EC2 using spark-ec2 script. Whenever more data is there to be processed I would like to add new slaves to existing cluster and would like to remove slave node when the data to be processed is low.

 

It seems currently spark-ec2 doesn’t have option to add/remove slaves to existing cluster. Could you please suggest how can we achieve this?

 

One liner problem statement: How to add/remove slaves to/from Spark cluster on EC2 using spark-ec2?

 

Suggestions on when to add/remove slaves is much appreciated.

 

Thanks & Regards,

Sirisha Devineni

DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

Reply | Threaded
Open this post in threaded view
|

Re: Reg: Add/Remove slave nodes spark-ec2

Mayur Rustagi
You'll have to restart the cluster.. create copy of your existing slave.. add it to slave files in master & restart the cluster

Mayur Rustagi
Ph: +1 (760) 203 3257


On Tue, Jun 3, 2014 at 4:30 PM, Sirisha Devineni <[hidden email]> wrote:

Hi All,

 

I have created a spark cluster on EC2 using spark-ec2 script. Whenever more data is there to be processed I would like to add new slaves to existing cluster and would like to remove slave node when the data to be processed is low.

 

It seems currently spark-ec2 doesn’t have option to add/remove slaves to existing cluster. Could you please suggest how can we achieve this?

 

One liner problem statement: How to add/remove slaves to/from Spark cluster on EC2 using spark-ec2?

 

Suggestions on when to add/remove slaves is much appreciated.

 

Thanks & Regards,

Sirisha Devineni

DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.