What to consider when implementing a custom streaming sink?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

What to consider when implementing a custom streaming sink?

kant kodali
Hi All,

I am trying to implement a custom sink and I have few questions mainly on output modes.

1) How does spark let the sink know that a new row is an update of an existing row? does it look at all the values of all columns of the new row and an existing row for an equality match or does it compute some sort of hash?

2) What else do I need to consider when writing a custom sink?

Thanks!