Support for group aggregate pandas UDF in streaming aggregation for SPARK 3.0 python

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Support for group aggregate pandas UDF in streaming aggregation for SPARK 3.0 python

Aesha Dhar Roy
Hi,

Is there any plan to remove the limitation mentioned below?

Streaming aggregation doesn't support group aggregate pandas UDF 

We want to run our data modelling jobs real time using Spark 3.0 and kafka 2.4 and need to have support for custom aggregate pandas UDF on stream windows.
Is there any plan for this in the upcoming releases ?

Regards,
Aesha