Count() not working on streaming dataframe/structured streaming

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Count() not working on streaming dataframe/structured streaming

Ritesh Shah

I have written this simple code to try streaming aggregation in spark 2.4. Somehow, job keeps running but not returning any result. It returns me 3 columns JobType, Timestamp and TS if I remove groupby and count aggregation function.

 

Would really appreciate any help.

 

       val edgeDF = spark

        .readStream

        .format("kafka")

        .option("kafka.bootstrap.servers", "…")

        .option("subscribe", "edgemw")

        .load()

       

       val edgeSelectDF = edgeDF.select(get_json_object(($"value").cast("string"),"$.after.JOBTYPE").alias("JobType"),

                                       get_json_object(($"value").cast("string"),"$.current_ts").alias("Timestamp"))

                               .select($"JobType",$"Timestamp",from_utc_timestamp($"Timestamp", "UTC").alias("TS"))

                               .groupBy("JobType")

                               .count()

                                                                           

                               

       val query = edgeSelectDF

              .writeStream

              .format("console")

              .outputMode(OutputMode.Complete())

              .trigger(Trigger.ProcessingTime(5.second))

              .start()

              .awaitTermination()

 

Thanks & regards,

Ritesh

============================================================================================================================

Disclaimer:  This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra.

============================================================================================================================