Question on web UI

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Question on web UI

David Thomas
On the spark web UI, there are two tables - Running Applications and Running Drivers. What is the difference between these two?

When I submit an application to run on the cluster, I can see the driver table getting populated, but why is only one worker listed under Worker column? What is the role of this single worker? Shouldn't the driver program be running on the master node?
Reply | Threaded
Open this post in threaded view
|

Re: Question on web UI

Nan Zhu
the driver is running on the machine where you run command like ./spark-shell


Best,

-- 
Nan Zhu

On Tuesday, February 18, 2014 at 10:06 PM, David Thomas wrote:

On the spark web UI, there are two tables - Running Applications and Running Drivers. What is the difference between these two?

When I submit an application to run on the cluster, I can see the driver table getting populated, but why is only one worker listed under Worker column? What is the role of this single worker? Shouldn't the driver program be running on the master node?

Reply | Threaded
Open this post in threaded view
|

Re: Question on web UI

David Thomas
I was using that exact command line (./bin/spark-class org.apache.spark.deploy.Client launch) to launch the app within the cluster. Now I was running this from the master node, but the Worker column shows just one worker in the cluster. So what is the significance of that column? Shouldn't the driver be running on the master node?


On Tue, Feb 18, 2014 at 8:31 PM, Nan Zhu <[hidden email]> wrote:
the driver is running on the machine where you run command like ./spark-shell


Best,

-- 
Nan Zhu

On Tuesday, February 18, 2014 at 10:06 PM, David Thomas wrote:

On the spark web UI, there are two tables - Running Applications and Running Drivers. What is the difference between these two?

When I submit an application to run on the cluster, I can see the driver table getting populated, but why is only one worker listed under Worker column? What is the role of this single worker? Shouldn't the driver program be running on the master node?


Reply | Threaded
Open this post in threaded view
|

Re: Question on web UI

Nan Zhu

the driver is more like a per-application task scheduler whereas the master is a resource manager to allocate resources to applications and maintain workers

with the command you use, the driver will run on the worker nodes 

Best,

-- 
Nan Zhu

On Tuesday, February 18, 2014 at 10:32 PM, David Thomas wrote:

I was using that exact command line (./bin/spark-class org.apache.spark.deploy.Client launch) to launch the app within the cluster. Now I was running this from the master node, but the Worker column shows just one worker in the cluster. So what is the significance of that column? Shouldn't the driver be running on the master node?


On Tue, Feb 18, 2014 at 8:31 PM, Nan Zhu <[hidden email]> wrote:
the driver is running on the machine where you run command like ./spark-shell


Best,

-- 
Nan Zhu

On Tuesday, February 18, 2014 at 10:06 PM, David Thomas wrote:

On the spark web UI, there are two tables - Running Applications and Running Drivers. What is the difference between these two?

When I submit an application to run on the cluster, I can see the driver table getting populated, but why is only one worker listed under Worker column? What is the role of this single worker? Shouldn't the driver program be running on the master node?