Emit Custom metrics in Spark Structured Streaming job

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Emit Custom metrics in Spark Structured Streaming job

subramgr
In our Spark Structured Streaming job we listen to Kafka and we filter out
some messages which we feel are malformed.

Currently we log that information using the LOGGER.

Is there a way to emit some kind of metrics for each time such a malformed
message is seen in Structured Streaming ?

Thanks



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Emit Custom metrics in Spark Structured Streaming job

subramgr
I am planning to send these metrics to our KairosDB. Let me know if there are
any examples that I can take a look



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Emit Custom metrics in Spark Structured Streaming job

chandan prakash
Hi Subramanian,
Did you find any solution for this ?
I am looking for something similar too.

Regards,
Chandan

On Wed, Jun 27, 2018 at 9:47 AM subramgr <[hidden email]> wrote:
I am planning to send these metrics to our KairosDB. Let me know if there are
any examples that I can take a look



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Chandan Prakash

Reply | Threaded
Open this post in threaded view
|

Re: Emit Custom metrics in Spark Structured Streaming job

subramgr
Hi

I am working on implementing my idea but here is how it goes:

1. Use this library https://github.com/groupon/spark-metrics
2. Have a cron job which periodically curl /metrics/json endpoint at driver
and all other nodes
3. Parse the response and send the data through a telegraf agent installed
on the node.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]