Adding metrics to spark datasource

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Adding metrics to spark datasource

This post has NOT been accepted by the mailing list yet.


I am building a data source so I can convert a custom source to dataframe.

I have been going over examples such as JDBC and noticed that JDBC does the following:


val inputMetrics = context.taskMetrics().inputMetrics


and whenever a new record is added:




however, incRecordsRead is spark private.

I was wondering: how can I get standard metrics through a data source?