Is there a merge API available for writing DataFrame

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Is there a merge API available for writing DataFrame

Sivaprasanna
Hi, 

As the title implies, do we have a way of merging a DataFrame into a sink (either Table or a distribute filesystem)? I'm sure we cannot have a full fledged equivalent of Hive's MERGE INTO but maybe we can have a way of writing (updating) only those rows present in the DF, with the rest of the rows/data in the sink untouched.

Sivaprasanna
Reply | Threaded
Open this post in threaded view
|

Re: Is there a merge API available for writing DataFrame

ayan guha
You are probably looking for Spark Delta Lake tables

On Fri, 15 Nov 2019 at 7:48 pm, Sivaprasanna <[hidden email]> wrote:
Hi, 

As the title implies, do we have a way of merging a DataFrame into a sink (either Table or a distribute filesystem)? I'm sure we cannot have a full fledged equivalent of Hive's MERGE INTO but maybe we can have a way of writing (updating) only those rows present in the DF, with the rest of the rows/data in the sink untouched.

Sivaprasanna
--
Best Regards,
Ayan Guha