How to monitor the throughput and latency of the combineByKey transformation in Spark 3?
I built a simple count and sum spark application which uses the
combineByKey transformation  and I would like to monitor the
throughput in/out of this transformation and the latency that the
combineByKey spends to pre-aggregate tuples. Ideally, the latency I
would like to take the average of the last 30 seconds using a
histogram and the 99th percentile.
I was imagining to add a dropwizard metrics  on the combiner
function that I pass to the combineByKey. But It is confused because
there are 2 more functions that I must pass to the combineByKey.
How would you suggest me to implement this monitoring strategy?