Απάντηση: Re: Question about Spark cluster memory usage monitoring

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Απάντηση: Re: Question about Spark cluster memory usage monitoring

kolokasis
Hello,
a good tool to manage the cluster usage for each mode is using Grafana which use Prometheus in the background to collect metrics. 

-Iacovos



Εστάλη από τη συσκευή μου Samsung


-------- Αρχικό μήνυμα --------
Από: "Jan Brabec (janbrabe)" <[hidden email]>
Ημερομηνία: 24/9/18 08:57 (GMT+01:00)
Προς: Muhib Khan <[hidden email]>, "Liu, Jialin" <[hidden email]>
Κοιν.: [hidden email]
Θέμα: Re: Question about Spark cluster memory usage monitoring

Hello,

 

Maybe Ganglia http://ganglia.info/ might be useful for you? I have only shallow experience with it, but it might be what you are looking for.

 

Best,

Jan

 

From: Muhib Khan <[hidden email]>
Date: Thursday 20 September 2018 at 23:39
To: "Liu, Jialin" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Question about Spark cluster memory usage monitoring

 

Hello,

 

As far as I know, there is no API provided for tracking the execution memory of a Spark Worker node. For tracking the execution memory you will probably need to access the MemoryManager's onHeapExecutionMemoryPool and offHeapExecutionMemoryPool objects that track the memory allocated to tasks for execution memory and write to a log the amount for parsing it later or send it to the driver in a periodic interval to collect the memory usage in a central location. Either way, I think tracking execution memory may require code changes in Apache Spark. Hope this helps.

 

Regards,

Muhib

Ph.D. Student, FSU

 

 

 

On Thu, Sep 20, 2018 at 4:46 PM Liu, Jialin <[hidden email]> wrote:

Hi there,

I am currently using Spark cluster to run jobs but I really need to collect the history of actually memory usage(that’s execution memory + storage memory) of the job in the whole cluster. I know we can get the storage memory usage through either Spark UI Executor page or SparkContext.getMemoryExecutorStatus() API, but I could not get the real time execution memory usage.
Is there anyway I can collect total memory usage? Thank you so much!

Best,
Jialin Liu
---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]