Spark :- Update record in partition.

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark :- Update record in partition.

Sunil kalra
Hi All,

If i have to update a record in partition using spark, do i have to read the whole partition and update the row and overwrite the partition?

Is there a way to only update 1 row like DBMS. Otherwise 1 row update takes a long time to rewrite the whole partition ?

Thanks
Sunil 




Reply | Threaded
Open this post in threaded view
|

Re: Spark :- Update record in partition.

ayan guha
Hi

Please look at delta.io which is a companion open source project. It addresses the exact use case you are after. 

On Mon, Jun 8, 2020 at 2:35 AM Sunil Kalra <[hidden email]> wrote:
Hi All,

If i have to update a record in partition using spark, do i have to read the whole partition and update the row and overwrite the partition?

Is there a way to only update 1 row like DBMS. Otherwise 1 row update takes a long time to rewrite the whole partition ?

Thanks
Sunil 






--
Best Regards,
Ayan Guha