How to register custom structured streaming source
I need to create a custom streaming source by extending FileStreamSource. The idea is to override commit, so that processed files (S3 objects in my case) are renamed to have a certain prefix. However, I don't know how to use this custom source. Obviously I don't want to compile Spark -- the application will be running on Amazon EMR clusters.