A code with same flavor caused the following error (which in my understanding is related to the creation of the Java code generation):
As you can see in this usecase I have small steps and large window.
I write a sliding window analytic program and use the functions.window function (https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/functions.html#window(org.apache.spark.sql.Column,%20java.lang.String,%20java.lang.String))The code looks like this:
Column slidingWindow = functions.window(myDF.col("timestamp"), "24 hours", "1 seconds");
also I see hundreds of "Code generated in 4.979587 ms" plus it is taking quite a lot time to compute although it has only 10 rows of data.
Am I doing something wrong? is it a bug? what is the right way to use this function?
Please relate to Java when answering.
|Free forum by Nabble||Edit this page|