My sincere apologies for adding my question to this chain. For some reason, I am unable to see the messages which I write to the group ever appear back in it and I think that this might be related in a way that shows a few differences between traditional operations and Spark Streaming operations.
Can I please ask why does lines.count() throws the exception: org.apache.spark.sql.AnalysisException: Queries with streaming sources must be executed with writeStream.start();;
Whereas if I do lines.createOrReplaceTempView("test") and then run the sql "select count(*) ccount from test" it runs absolutely fine.
I can figure out from the exceptions that there is a check which is getting executed to find out whether isStreaming is true for lines or not, but a bit of explanation might help.
On Fri, Apr 13, 2018 at 3:53 AM, Tathagata Das <[hidden email]> wrote:
The traditional SQL windows with `over` is not supported in streaming. Only time-based windows, that is, `window("timestamp", "10 minutes")` is supported in streaming.