Exporting spark custom metrics via prometheus jmx exporter

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Exporting spark custom metrics via prometheus jmx exporter

I'm new to spark.
Trying to export spark (2.4.0) custom metrics in prometheus format.
I found this guide https://argus-sec.com/monitoring-spark-prometheus/ 
Running on my local pc (windows) I am able to see my metrics in visualVM and
in spark's json http://localhost:4040/metrics/json/

My issue is that the JMX exporter
(https://github.com/prometheus/jmx_exporter/blob/master/README.md) does not
show any signs that it's active at all.
I added the following to spark conf :

val conf = new SparkConf()                                                  
val metrics  =  Map("spark.metrics.conf" ->
val prometheus = Map("spark.driver.extraJavaOptions" ->
And my prometheus-config.yaml file has no rules to present everything just
to see that the data is extracted.

lowercaseOutputName: true
attrNameSnakeCase: true
I do not see any logs from the JMX exporter and
http://localhost:8095/metrics can’t be reached.
Please advise :)

Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

To unsubscribe e-mail: [hidden email]