'ExecutorTaskSummary' alternative in Spark 2.3 onwards

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

'ExecutorTaskSummary' alternative in Spark 2.3 onwards

Ninja Coder
I have a spark streaming application (currently spark 2.2) which is using
`org.apache.spark.ui.exec.ExecutorTaskSummary` to grab executor details like
duration, tasks failed, tasks completed, GC time, etc after each batch is
completed. These metrics are then loaded to prometheus and pagerduty alerts
are set on it.

We are planning to upgrade spark and looks like`ExecutorTaskSummary` is no
longer available from spark 2.3

I would like to know what are the other alternatives which I can use.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: 'ExecutorTaskSummary' alternative in Spark 2.3 onwards

Ninja Coder

Any advise/help here is much appreciated.

On Mon, Dec 30, 2019 at 4:16 PM Ninja Coder <[hidden email]> wrote:
I have a spark streaming application (currently spark 2.2) which is using
`org.apache.spark.ui.exec.ExecutorTaskSummary` to grab executor details like
duration, tasks failed, tasks completed, GC time, etc after each batch is
completed. These metrics are then loaded to prometheus and pagerduty alerts
are set on it.

We are planning to upgrade spark and looks like`ExecutorTaskSummary` is no
longer available from spark 2.3

I would like to know what are the other alternatives which I can use.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]