Writing custom Structured Streaming receiver

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Writing custom Structured Streaming receiver

Daniel Haviv
Hi,
Is there a guide to writing a custom Structured Streaming receiver?

Thank you.
Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Writing custom Structured Streaming receiver

Tathagata Das
Structured Streaming source APIs are not yet public, so there isnt a guide. However, if you are adventurous enough, you can take a look at the source code in Spark.

Note that these APIs are still internal APIs and are very likely to change in future versions of Spark.


On Wed, Nov 1, 2017 at 5:45 PM, Daniel Haviv <[hidden email]> wrote:
Hi,
Is there a guide to writing a custom Structured Streaming receiver?

Thank you.
Daniel

Reply | Threaded
Open this post in threaded view
|

Re: Writing custom Structured Streaming receiver

Hien Luu
Hi TD,

I looked at DataStreamReader class and looks like we can specify an FQCN as
a source (provided that it implements trait Source). The
DataSource.lookupDataSource function will try to load this FQCN during the
creation of a DataSource object instance inside the DataStreamReader.load().
Will this work?

I am curious if anyone has tried this yet?

I am going to give a shot to see if this works.

Thanks,

Hien



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Writing custom Structured Streaming receiver

nezhazheng
Hi Hien,

You can write your own Source or Sink either through SPI(https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html).
Below in an example that implement kafka 0.8 source.

郑舒力

在 2017年11月21日,上午9:40,Hien Luu <[hidden email]> 写道:

Hi TD,

I looked at DataStreamReader class and looks like we can specify an FQCN as
a source (provided that it implements trait Source). The
DataSource.lookupDataSource function will try to load this FQCN during the
creation of a DataSource object instance inside the DataStreamReader.load().
Will this work?

I am curious if anyone has tried this yet?

I am going to give a shot to see if this works.

Thanks,

Hien



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Writing custom Structured Streaming receiver

Hien Luu
Cool.  Thanks nezhazheng.  I will give it a shot.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]