chaining flatMapGroupsWithState in append mode

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

chaining flatMapGroupsWithState in append mode

alex770
I wrote few examples chaining flatMapGroupsWithState in append mode and it
worked. Other examples did not.

The question is if the flatMapGroupsWithState chains in append mode are
officially supported.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: chaining flatMapGroupsWithState in append mode

Jungtaek Lim-2
That will heavily depend on your implementation of state functions on flatMapGroupsWithState, but assuming your state functions respect of the output mode, chaining flatMapGroupsWithState doesn't work - I don't mean it doesn't compile or doesn't run. I mean the query will not produce correct output.

The guide doc of Structured Streaming in Spark 3.0 will contain the limitation of global watermark which explains why the chaining of stateful operators in Structured Streaming doesn't work correctly. Spark 3.0 will also log the warning message if Spark detects the pattern. Reading the new part in doc requires understanding of semantic of watermark and how global watermark is calculated in prior, so a bit advanced topic. (Though I think end users should also need to understand this to make sure their query is "correct".)

You can see the change of doc previously in this PR: https://github.com/apache/spark/pull/24890

Thanks,
Jungtaek Lim (HeartSaVioR)

On Thu, Nov 28, 2019 at 9:55 PM alex770 <[hidden email]> wrote:
I wrote few examples chaining flatMapGroupsWithState in append mode and it
worked. Other examples did not.

The question is if the flatMapGroupsWithState chains in append mode are
officially supported.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]