Converting spark batch to spark streaming

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Converting spark batch to spark streaming

mhd wrk
I'm trying to convert a spark batch application to a streaming application and wondering what function (or design pattern) I should use to execute a series of operations inside the driver upon arrival of each message (a text file inside an HDFS folder) before starting computation inside executors.

Thanks,
Mohammad
Reply | Threaded
Open this post in threaded view
|

Re: Converting spark batch to spark streaming

Jacek Laskowski
Hi,

Start with DataStreamWriter.foreachBatch.

On Thu, Jan 7, 2021 at 6:55 PM mhd wrk <[hidden email]> wrote:
I'm trying to convert a spark batch application to a streaming application and wondering what function (or design pattern) I should use to execute a series of operations inside the driver upon arrival of each message (a text file inside an HDFS folder) before starting computation inside executors.

Thanks,
Mohammad