Basic Spark terms - Slaves, workers, executors

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Basic Spark terms - Slaves, workers, executors

ssimanta
Can someone explain what each of these terms means in terms of Spark? I've a confusion between what's the difference between Slaves, workers, executors. 

My understanding is the slaves and workers are interchangeable ?  

Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Basic Spark terms - Slaves, workers, executors

dachuan
I think so, slave = worker. one worker = multiple executors.

One spark cluster consists of one master and multiple workers.

One worker can have configured cores and memory.

One worker's memory size is specified in conf/spark-env.sh, locally visible.

One app = multiple executors

One executor manages many cores (calculated in Master.schedule() function)

One executor's memory size is specified in app's SparkConf.


On Sun, Feb 16, 2014 at 5:33 PM, Soumya Simanta <[hidden email]> wrote:
Can someone explain what each of these terms means in terms of Spark? I've a confusion between what's the difference between Slaves, workers, executors. 

My understanding is the slaves and workers are interchangeable ?  

Thanks.



--
Dachuan Huang
Cellphone: 614-390-7234
2015 Neil Avenue
Ohio State University
Columbus, Ohio
U.S.A.
43210
Reply | Threaded
Open this post in threaded view
|

Re: Basic Spark terms - Slaves, workers, executors

Matei Zaharia
Administrator
Hi Soumya,

You can find an official description of the terms here: http://spark.incubator.apache.org/docs/latest/cluster-overview.html. “Slave” is sometimes used interchangeably for a worker, but most of the code and log messages uses the terms in that doc.

Matei

On Feb 16, 2014, at 2:44 PM, dachuan <[hidden email]> wrote:

I think so, slave = worker. one worker = multiple executors.

One spark cluster consists of one master and multiple workers.

One worker can have configured cores and memory.

One worker's memory size is specified in conf/spark-env.sh, locally visible.

One app = multiple executors

One executor manages many cores (calculated in Master.schedule() function)

One executor's memory size is specified in app's SparkConf.


On Sun, Feb 16, 2014 at 5:33 PM, Soumya Simanta <[hidden email]> wrote:
Can someone explain what each of these terms means in terms of Spark? I've a confusion between what's the difference between Slaves, workers, executors. 

My understanding is the slaves and workers are interchangeable ?  

Thanks.



--
Dachuan Huang
Cellphone: 614-390-7234
2015 Neil Avenue
Ohio State University
Columbus, Ohio
U.S.A.
43210

Reply | Threaded
Open this post in threaded view
|

Re: Basic Spark terms - Slaves, workers, executors

ssimanta
Thanks Matei and Dachuan. 




On Sun, Feb 16, 2014 at 6:03 PM, Matei Zaharia <[hidden email]> wrote:
Hi Soumya,

You can find an official description of the terms here: http://spark.incubator.apache.org/docs/latest/cluster-overview.html. “Slave” is sometimes used interchangeably for a worker, but most of the code and log messages uses the terms in that doc.

Matei

On Feb 16, 2014, at 2:44 PM, dachuan <[hidden email]> wrote:

I think so, slave = worker. one worker = multiple executors.

One spark cluster consists of one master and multiple workers.

One worker can have configured cores and memory.

One worker's memory size is specified in conf/spark-env.sh, locally visible.

One app = multiple executors

One executor manages many cores (calculated in Master.schedule() function)

One executor's memory size is specified in app's SparkConf.


On Sun, Feb 16, 2014 at 5:33 PM, Soumya Simanta <[hidden email]> wrote:
Can someone explain what each of these terms means in terms of Spark? I've a confusion between what's the difference between Slaves, workers, executors. 

My understanding is the slaves and workers are interchangeable ?  

Thanks.



--
Dachuan Huang
Cellphone: <a href="tel:614-390-7234" value="+16143907234" target="_blank">614-390-7234
2015 Neil Avenue
Ohio State University
Columbus, Ohio
U.S.A.
43210