EC2 scripts documentations lacks how to actually run applications

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

EC2 scripts documentations lacks how to actually run applications

Aureliano Buendia
Hi,

The EC2 documents has a section called 'Running Applications', but it actually lacks the step which should describe how to run the application.

The spark_ec2 script seems to set up a standalone cluster, although I'm not sure why AMI_PREFIX point to mesos ami list.

Assuming that the cluster type is standalone, we could run the app by spark-class script. Is this the missing step in the documentations?

spark-class script does not launch a daemon, is it suppose to be used with nohup for long running applications?

Finally, is the standalone cluster type used for real world applications, or do people use spark on yarn and mesos when it comes to production?
Reply | Threaded
Open this post in threaded view
|

Re: EC2 scripts documentations lacks how to actually run applications

Aureliano Buendia
Here is a refactored version of the question:

How to run spark-class for long running applications? Why is that spark-class doesn't launch a daemon?


On Wed, Jan 8, 2014 at 3:21 AM, Aureliano Buendia <[hidden email]> wrote:
Hi,

The EC2 documents has a section called 'Running Applications', but it actually lacks the step which should describe how to run the application.

The spark_ec2 script seems to set up a standalone cluster, although I'm not sure why AMI_PREFIX point to mesos ami list.

Assuming that the cluster type is standalone, we could run the app by spark-class script. Is this the missing step in the documentations?

spark-class script does not launch a daemon, is it suppose to be used with nohup for long running applications?

Finally, is the standalone cluster type used for real world applications, or do people use spark on yarn and mesos when it comes to production?

Reply | Threaded
Open this post in threaded view
|

Re: EC2 scripts documentations lacks how to actually run applications

Mark Hamstra


On Wed, Jan 8, 2014 at 10:12 AM, Aureliano Buendia <[hidden email]> wrote:
Here is a refactored version of the question:

How to run spark-class for long running applications? Why is that spark-class doesn't launch a daemon?


On Wed, Jan 8, 2014 at 3:21 AM, Aureliano Buendia <[hidden email]> wrote:
Hi,

The EC2 documents has a section called 'Running Applications', but it actually lacks the step which should describe how to run the application.

The spark_ec2 script seems to set up a standalone cluster, although I'm not sure why AMI_PREFIX point to mesos ami list.

Assuming that the cluster type is standalone, we could run the app by spark-class script. Is this the missing step in the documentations?

spark-class script does not launch a daemon, is it suppose to be used with nohup for long running applications?

Finally, is the standalone cluster type used for real world applications, or do people use spark on yarn and mesos when it comes to production?


Reply | Threaded
Open this post in threaded view
|

Re: EC2 scripts documentations lacks how to actually run applications

Patrick Wendell
Hey Aureliano,

Yes, people run long running applications with standalone mode and run
it in production. spark-class is a utility function for convenience.
If you want to run a long running application you would write a spark
application, bundle it, and submit it to the cluster. You can then
launch your own application with no-hup or however else you want to
daemonize it.

Here is an example of a standalone application.

http://spark.incubator.apache.org/docs/latest/quick-start.html

The pull request Mark referred to adds some support for submitting
your driver program to the cluster... but it's just an extra feature.
Launching packaged applications is the way you want to go for your use
case.

- Patrick

On Wed, Jan 8, 2014 at 10:31 AM, Mark Hamstra <[hidden email]> wrote:

> https://github.com/apache/incubator-spark/pull/293
>
>
> On Wed, Jan 8, 2014 at 10:12 AM, Aureliano Buendia <[hidden email]>
> wrote:
>>
>> Here is a refactored version of the question:
>>
>> How to run spark-class for long running applications? Why is that
>> spark-class doesn't launch a daemon?
>>
>>
>> On Wed, Jan 8, 2014 at 3:21 AM, Aureliano Buendia <[hidden email]>
>> wrote:
>>>
>>> Hi,
>>>
>>> The EC2 documents has a section called 'Running Applications', but it
>>> actually lacks the step which should describe how to run the application.
>>>
>>> The spark_ec2 script seems to set up a standalone cluster, although I'm
>>> not sure why AMI_PREFIX point to mesos ami list.
>>>
>>> Assuming that the cluster type is standalone, we could run the app by
>>> spark-class script. Is this the missing step in the documentations?
>>>
>>> spark-class script does not launch a daemon, is it suppose to be used
>>> with nohup for long running applications?
>>>
>>> Finally, is the standalone cluster type used for real world applications,
>>> or do people use spark on yarn and mesos when it comes to production?
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: EC2 scripts documentations lacks how to actually run applications

Aureliano Buendia
Thanks Patrick. I take it that spark does not come with a daemonizer, and the user is responsible for that.

Having said that, it feels odd that spark doesn't come with an application daemonizer out of the box. I know spark is fast, but it's not _that_ fast to not need a daemonizer :)


On Wed, Jan 8, 2014 at 7:57 PM, Patrick Wendell <[hidden email]> wrote:
Hey Aureliano,

Yes, people run long running applications with standalone mode and run
it in production. spark-class is a utility function for convenience.
If you want to run a long running application you would write a spark
application, bundle it, and submit it to the cluster. You can then
launch your own application with no-hup or however else you want to
daemonize it.

Here is an example of a standalone application.

http://spark.incubator.apache.org/docs/latest/quick-start.html

The pull request Mark referred to adds some support for submitting
your driver program to the cluster... but it's just an extra feature.
Launching packaged applications is the way you want to go for your use
case.

- Patrick

On Wed, Jan 8, 2014 at 10:31 AM, Mark Hamstra <[hidden email]> wrote:
> https://github.com/apache/incubator-spark/pull/293
>
>
> On Wed, Jan 8, 2014 at 10:12 AM, Aureliano Buendia <[hidden email]>
> wrote:
>>
>> Here is a refactored version of the question:
>>
>> How to run spark-class for long running applications? Why is that
>> spark-class doesn't launch a daemon?
>>
>>
>> On Wed, Jan 8, 2014 at 3:21 AM, Aureliano Buendia <[hidden email]>
>> wrote:
>>>
>>> Hi,
>>>
>>> The EC2 documents has a section called 'Running Applications', but it
>>> actually lacks the step which should describe how to run the application.
>>>
>>> The spark_ec2 script seems to set up a standalone cluster, although I'm
>>> not sure why AMI_PREFIX point to mesos ami list.
>>>
>>> Assuming that the cluster type is standalone, we could run the app by
>>> spark-class script. Is this the missing step in the documentations?
>>>
>>> spark-class script does not launch a daemon, is it suppose to be used
>>> with nohup for long running applications?
>>>
>>> Finally, is the standalone cluster type used for real world applications,
>>> or do people use spark on yarn and mesos when it comes to production?
>>
>>
>