Re: How to address seemingly low core utilization on a spark workload?
That is precisely my question- what kind of leads can I look at to get a hint of where the inefficiencies lay?
On Thu, Nov 15, 2018 at 4:56 PM David Markovitz <[hidden email]> wrote:
It seems it is almost fully utilized – when it is active.
What happens in the gaps, where there is no spark activity?
Technology Solutions Professional, Data Platform
From: Vitaliy Pisarev <[hidden email]> Sent: Thursday, November 15, 2018 4:51 PM To: user <[hidden email]> Cc: David Markovitz <[hidden email]> Subject: How to address seemingly low core utilization on a spark workload?
I have a workload that runs on a cluster of 300 cores.
Below is a plot of the amount of active tasks over time during the execution of this workload:
What I deduce is that there are substantial intervals where the cores are heavily under-utilised.
What actions can I take to:
Increase the efficiency (== core utilisation) of the cluster?
Understand the root causes behind the drops in core utilisation?