Query on Spark Driver CPU and Memory utilization

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Query on Spark Driver CPU and Memory utilization

Aakash Basu-2
Hi,


I observed the Spark Driver shows 0 on all the metric blocks in the Spark UI even if it actually works.

Below is the Spark UI -

Below is the observation of CPU utilization of Spark Driver node on My-NetData.



Why does this happen?

Thanks,
Aakash.
Reply | Threaded
Open this post in threaded view
|

Re: Query on Spark Driver CPU and Memory utilization

srinath
Hi Akash,

I don't see a problem with this. RDD blocks and tasks run on executors.

Regards,
Srinath.



On Tue, Jun 12, 2018 at 5:12 PM Aakash Basu <[hidden email]> wrote:
Hi,


I observed the Spark Driver shows 0 on all the metric blocks in the Spark UI even if it actually works.

Below is the Spark UI -

Below is the observation of CPU utilization of Spark Driver node on My-NetData.



Why does this happen?

Thanks,
Aakash.
Reply | Threaded
Open this post in threaded view
|

Re: Query on Spark Driver CPU and Memory utilization

Aakash Basu-2
Yes Srinath,

The Executors are all in worker, right? But, according to the second SS my driver CPU is taking huge tasks. But that doesn't reflect in the Spark Executor UI from the first SS (Driver/MasterNode), why so?

On Wed, Jun 13, 2018 at 7:21 AM, Srinath C <[hidden email]> wrote:
Hi Akash,

I don't see a problem with this. RDD blocks and tasks run on executors.

Regards,
Srinath.



On Tue, Jun 12, 2018 at 5:12 PM Aakash Basu <[hidden email]> wrote:
Hi,


I observed the Spark Driver shows 0 on all the metric blocks in the Spark UI even if it actually works.

Below is the Spark UI -

Below is the observation of CPU utilization of Spark Driver node on My-NetData.



Why does this happen?

Thanks,
Aakash.