I am trying to aggregate Spark time-stamped structured stream to get per-device (source) averages for every second of incoming data.
I am using Spark 2.4.6.
Yet I am getting an exception in the query start():
even though there is a watermark on the stream.
Schema printout looks fine:
Actual data looks fine too. If I feed it to
then I am getting output
and then follow-up batches of a similar look.
There is no exception if I use COMPLETE output mode, but then old results (from the start of the timeline) are reported in every batch and that’s not what I want. I want only new query result records to be reported. Thus I want the APPEND mode – but it causes an exception.
Why is the exception and how can I make it work?
Tiny project that isolates the problem is here:
Thanks for advice.
try this :
On Wed, 23 Sep 2020 at 22:51, Sergey Oboguev <[hidden email]> wrote:
|Free forum by Nabble||Edit this page|