How to register custom structured streaming source

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How to register custom structured streaming source

Farshid Zavareh
Hello.

I need to create a custom streaming source by extending FileStreamSource. The idea is to override commit, so that processed files (S3 objects in my case) are renamed to have a certain prefix. However, I don't know how to use this custom source. Obviously I don't want to compile Spark -- the application will be running on Amazon EMR clusters.

Thanks,
Farshid