Apache Spark - (2.2.0) - window function for DataSet

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache Spark - (2.2.0) - window function for DataSet

M Singh
Hi:
I would like to use window function on a DataSet stream (Spark 2.2.0)
The window function requires Column as argument and can be used with DataFrames by passing the column. Is there any analogous window function or pointers to how window function can be used for DataSets ?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark - (2.2.0) - window function for DataSet

Diogo Munaro Vieira
Window function requires a timestamp column because you will apply a function for each window (like an aggregation). You still can use UDF for customized tasks

Em 25 de dez de 2017 20:15, "M Singh" <[hidden email]> escreveu:
Hi:
I would like to use window function on a DataSet stream (Spark 2.2.0)
The window function requires Column as argument and can be used with DataFrames by passing the column. Is there any analogous window function or pointers to how window function can be used for DataSets ?

Thanks