Window Time Function

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Window Time Function

Simon Schiff
This post has NOT been accepted by the mailing list yet.
Hello,
I am using the Spark 2.1.0 release. I wrote a application to process a huge amount of time-tagged sensor data. For this I use the following function:

org.apache.spark.sql.window(Column timeColumn, String windowDuration, String slideDuration)

But i got a analysis exception, when slideDuration > windowDuration. I do this, because I want to define sampling windows over the data. I found here the corresponding code at github:

https://github.com/apache/spark/blob/39e2bad6a866d27c3ca594d15e574a1da3ee84cc/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TimeWindow.scala#L85

For a test I removed the if condition and compiled spark for myself. It worked and I was able to define a sampling window where slideDuration > windowDuration with correct results. My question: Why is it not allowed to define a sampling window? It would be really nice to have this feature included in spark. Then others are able to compile my application from source without compiling spark before. Would it be the correct way, when I do a pull request for this?
Loading...