Connecting App to cluster VS Launching app within cluster

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Connecting App to cluster VS Launching app within cluster

robin_up
Hi

Could someone kindly explain what the pros/cons are between "Connecting an Application to a cluster" and "Launching Applications inside a cluster" in a standalone cluster?

I'm referring to the sections in the below doc. I.e. in which situation one maybe more preferred than the other, or does it matter?

http://spark.incubator.apache.org/docs/latest/spark-standalone.html

thanks
Robin
-- Robin Li
Reply | Threaded
Open this post in threaded view
|

Re: Connecting App to cluster VS Launching app within cluster

Tathagata Das
Launching your application in a cluster may be useful in a number of scenarios. 

1) In a number of settings in companies, user who want to run jobs do not have ssh access to any of the cluster nodes. So they have to run the Spark driver program on their "local machine" and connect to the Spark standalone cluster's master for launching jobs. That can be pretty slow and can go down if the connection between "local machine" and cluster is severed. Instead, launching applciation inside the cluster allows you to launch the job and forget it. 

2) Another additional functionality that launching in the cluster provides is automatic restart of the Spark driver if it fails. This is useful for long running Spark jobs (e.g., Spark Streaming job) where you want to minimize downtime. 

TD


On Mon, Feb 10, 2014 at 6:10 PM, robin_up <[hidden email]> wrote:
Hi

Could someone kindly explain what the pros/cons are between "Connecting an
Application to a cluster" and "Launching Applications inside a cluster" in a
standalone cluster?

I'm referring to the sections in the below doc. I.e. in which situation one
maybe more preferred than the other, or does it matter?

http://spark.incubator.apache.org/docs/latest/spark-standalone.html

thanks
Robin




-----
-- Robin Li
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Connecting-App-to-cluster-VS-Launching-app-within-cluster-tp1375.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Connecting App to cluster VS Launching app within cluster

robin_up
Thanks TD. This is very useful information.


On Thu, Feb 13, 2014 at 12:30 PM, Tathagata Das [via Apache Spark User List] <[hidden email]> wrote:
Launching your application in a cluster may be useful in a number of scenarios. 

1) In a number of settings in companies, user who want to run jobs do not have ssh access to any of the cluster nodes. So they have to run the Spark driver program on their "local machine" and connect to the Spark standalone cluster's master for launching jobs. That can be pretty slow and can go down if the connection between "local machine" and cluster is severed. Instead, launching applciation inside the cluster allows you to launch the job and forget it. 

2) Another additional functionality that launching in the cluster provides is automatic restart of the Spark driver if it fails. This is useful for long running Spark jobs (e.g., Spark Streaming job) where you want to minimize downtime. 

TD


On Mon, Feb 10, 2014 at 6:10 PM, robin_up <[hidden email]> wrote:
Hi

Could someone kindly explain what the pros/cons are between "Connecting an
Application to a cluster" and "Launching Applications inside a cluster" in a
standalone cluster?

I'm referring to the sections in the below doc. I.e. in which situation one
maybe more preferred than the other, or does it matter?

http://spark.incubator.apache.org/docs/latest/spark-standalone.html

thanks
Robin




-----
-- Robin Li
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Connecting-App-to-cluster-VS-Launching-app-within-cluster-tp1375.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.




To unsubscribe from Connecting App to cluster VS Launching app within cluster, click here.
NAML



--
Robin
-- Robin Li