Stateful Aggregation Using flatMapGroupsWithState

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Stateful Aggregation Using flatMapGroupsWithState

Sandip Mehta
Hi All,

I am getting following error message while applying flatMapGroupsWithState.

Exception in thread "main" org.apache.spark.sql.AnalysisException: flatMapGroupsWithState in update mode is not supported with aggregation on a streaming DataFrame/Dataset;;

Following is what I am trying to do.

- Read messages from Kafka & Parse It
- Group based on certain dimensions
- Ran UDAF for every group and calculated aggregation for each group. Agg doesn't return KeyValueGroupDataSet. So applying groupByKey on previous step output to group based on aggFunction column
- Merge this aggregates to previous state of stream using flatMapGroupsWithState.

Getting error message for the last step. 

Does this error means I cannot apply flatMapGroupsWithState after applying agg(....) on dataset?