Spark Monitoring using Jolokia

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Monitoring using Jolokia

Irtiza Ali
Hello everyone,

I am building a monitoring tool for the spark, for that I needs sparks metrics. I am using jolokia to get the metrics. 

I have a question that: 

Can I get all the metrics provided by the spark rest api using the Jolokia?

How the spark rest api get the metrics internally?


Thanks 
Reply | Threaded
Open this post in threaded view
|

Re: Spark Monitoring using Jolokia

Maximiliano Felice
Hi!

I don't know very much about them, but I'm currently working in posting custom metrics into Graphite. I found useful the internals described in this library: https://github.com/groupon/spark-metrics

Hope this at least can give you a hint.

Best of luck!

2018-01-08 10:55 GMT-03:00 Irtiza Ali <[hidden email]>:
Hello everyone,

I am building a monitoring tool for the spark, for that I needs sparks metrics. I am using jolokia to get the metrics. 

I have a question that: 

Can I get all the metrics provided by the spark rest api using the Jolokia?

How the spark rest api get the metrics internally?


Thanks 

Reply | Threaded
Open this post in threaded view
|

Re: Spark Monitoring using Jolokia

Thakrar, Jayesh

And here's some more info on Spark Metrics

 

https://www.slideshare.net/JayeshThakrar/apache-bigdata2017sparkprofiling

 

 

From: Maximiliano Felice <[hidden email]>
Date: Monday, January 8, 2018 at 8:14 AM
To: Irtiza Ali <[hidden email]>
Cc: <[hidden email]>
Subject: Re: Spark Monitoring using Jolokia

 

Hi!

 

I don't know very much about them, but I'm currently working in posting custom metrics into Graphite. I found useful the internals described in this library: https://github.com/groupon/spark-metrics

 

Hope this at least can give you a hint.

 

Best of luck!

 

2018-01-08 10:55 GMT-03:00 Irtiza Ali <[hidden email]>:

Hello everyone,

 

I am building a monitoring tool for the spark, for that I needs sparks metrics. I am using jolokia to get the metrics. 

 

I have a question that: 

 

Can I get all the metrics provided by the spark rest api using the Jolokia?

 

How the spark rest api get the metrics internally?

 

 

Thanks 

 

Reply | Threaded
Open this post in threaded view
|

Re: Spark Monitoring using Jolokia

Gourav Sengupta
In reply to this post by Irtiza Ali
Hi,

I am totally confused here, may be because I do not exactly understand this, but why is this required? I have always used SPARK UI and found that more than sufficient. And if you know a bit about how SPARK session works then your performance does have a certain degree of predictability as well.

Sorry about this, but the input may really help.

I may be wrong, but I find such pursuits just a sheer exercise in creating products which in reality are not needed.


Regards,
Gourav Sengupta

On Mon, Jan 8, 2018 at 1:55 PM, Irtiza Ali <[hidden email]> wrote:
Hello everyone,

I am building a monitoring tool for the spark, for that I needs sparks metrics. I am using jolokia to get the metrics. 

I have a question that: 

Can I get all the metrics provided by the spark rest api using the Jolokia?

How the spark rest api get the metrics internally?


Thanks