Pre query execution hook for custom datasources

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Pre query execution hook for custom datasources

Shubham Chaurasia

In our custom datasource implementation, we want to inject some query level information.

For example - 

scala> val df = spark.sql("some query")   // uses custom datasource under the hood through Session Extensions. 

scala> df.count //  here we want some kind of pre execution hook just before the query starts it's execution 

Is there a hook or some kind of callback that we can implement to achieve this?

Or similar to org.apache.spark.sql.util.QueryExecutionListener which provides callbacks for onSuccess and onFailure when query finishes, we want something like "beforeStart()".

Any ideas on how to implement this?