avoid duplicate records when appending new data to a parquet

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

avoid duplicate records when appending new data to a parquet

Lian Jiang
I have a parquet which has an id field which is the hash of the composite key fields. Is it possible to maintain the uniqueness of the id field when appending new data which may duplicate with existing records in the parquet? Thanks!