Apache Spark - Using withWatermark for DataSets

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Apache Spark - Using withWatermark for DataSets

M Singh

I am working with DataSets so that I can use mapGroupsWithState for business logic and then use dropDuplicates over a set of fields.  I would like to use the withWatermark so that I can restrict the how much state is stored.  

From the API it looks like withWatermark takes a string - timestamp column name as argument.  Is it possible to use it with DataSets ?  If not, is there any alternative like withWatermark available for working with DataSets ?