Suggestion for ec2 script

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Suggestion for ec2 script

Mingyu Kim
Hi all,

I found it confusing that "./spark-ec2 start” actually reinstalls the cluster, which ends up wiping out all the configurations. How about renaming “start” to “install” and add a real light-weight “start” for frequently starting and stopping ec2 instances for mostly cost reasons? The light-weight “start” can simply bring up the instances and invoke “bin/start-all.sh”. Thoughts?

Mingyu

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion for ec2 script

Matei Zaharia
Administrator
The issue is that “start” is designed to restart the cluster after pausing the instances with “stop”, and because some instances’ DNS names changed, we need to modify the master URL and the list of slaves in the various config files. If you make other changes in these config files, they will indeed be lost. One solution might be to create your own setup script to run on the instances after “start”.

Matei 

On Jan 24, 2014, at 2:19 PM, Mingyu Kim <[hidden email]> wrote:

Hi all,

I found it confusing that "./spark-ec2 start” actually reinstalls the cluster, which ends up wiping out all the configurations. How about renaming “start” to “install” and add a real light-weight “start” for frequently starting and stopping ec2 instances for mostly cost reasons? The light-weight “start” can simply bring up the instances and invoke “bin/start-all.sh”. Thoughts?

Mingyu

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion for ec2 script

Mingyu Kim
I see. I didn’t catch that. Thanks for the clarification!

Mingyu

From: Matei Zaharia <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, January 24, 2014 at 2:37 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: Suggestion for ec2 script

The issue is that “start” is designed to restart the cluster after pausing the instances with “stop”, and because some instances’ DNS names changed, we need to modify the master URL and the list of slaves in the various config files. If you make other changes in these config files, they will indeed be lost. One solution might be to create your own setup script to run on the instances after “start”.

Matei 

On Jan 24, 2014, at 2:19 PM, Mingyu Kim <[hidden email]> wrote:

Hi all,

I found it confusing that "./spark-ec2 start” actually reinstalls the cluster, which ends up wiping out all the configurations. How about renaming “start” to “install” and add a real light-weight “start” for frequently starting and stopping ec2 instances for mostly cost reasons? The light-weight “start” can simply bring up the instances and invoke “bin/start-all.sh”. Thoughts?

Mingyu


smime.p7s (7K) Download Attachment