Adding metrics to spark datasource

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Adding metrics to spark datasource

AssafMendelson
This post has NOT been accepted by the mailing list yet.

Hi,

I am building a data source so I can convert a custom source to dataframe.

I have been going over examples such as JDBC and noticed that JDBC does the following:

 

val inputMetrics = context.taskMetrics().inputMetrics

 

and whenever a new record is added:

 

inputMetrics.incRecordsRead(1)

 

however, incRecordsRead is spark private.

I was wondering: how can I get standard metrics through a data source?

 

Thanks,

                Assaf.

Loading...