Re: Emit Custom metrics in Spark Structured Streaming job
I am working on implementing my idea but here is how it goes:
1. Use this library https://github.com/groupon/spark-metrics 2. Have a cron job which periodically curl /metrics/json endpoint at driver
and all other nodes
3. Parse the response and send the data through a telegraf agent installed
on the node.