Spark Listeners for getting dataset partition information in streaming application

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark Listeners for getting dataset partition information in streaming application

Robin Kuttaiah
Hello,

Is there a way spark streaming application will get to know during the start and end of the data read from a dataset partition?
I want to create partition specific cache during the start and delete during the partition is read completely.

Thanks for you help in advance.

regards,
Robin Kuttaiah