How to register custom structured streaming source

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to register custom structured streaming source

Farshid Zavareh
Hello.

I need to create a custom streaming source by extending FileStreamSource. The idea is to override commit, so that processed files (S3 objects in my case) are renamed to have a certain prefix. However, I don't know how to use this custom source. Obviously I don't want to compile Spark -- the application will be running on Amazon EMR clusters.

Thanks,
Farshid
Reply | Threaded
Open this post in threaded view
|

Re: How to register custom structured streaming source

Hien Luu
Hi Farshid,

Take a look at this example on github - https://github.com/hienluu/structured-streaming-sources.

Cheers,

Hien

On Thu, Jul 12, 2018 at 12:52 AM Farshid Zavareh <[hidden email]> wrote:
Hello.

I need to create a custom streaming source by extending FileStreamSource. The idea is to override commit, so that processed files (S3 objects in my case) are renamed to have a certain prefix. However, I don't know how to use this custom source. Obviously I don't want to compile Spark -- the application will be running on Amazon EMR clusters.

Thanks,
Farshid


--
Regards,