Structured Streaming: How to add a listener for when a batch is complete

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Structured Streaming: How to add a listener for when a batch is complete

Natalie Ruiz

Hello all,

 

I’m a beginner, new to Spark and wanted to know if there was an equivalent to Spark Streaming’s StreamingListenerBatchCompleted in Structured Streaming? I want to add a listener for when a batch is complete but the documentation and examples I find are for Spark Streaming and not Structured Streaming and registering it to a StreamingContext, which from my understanding is just for Spark Streaming

 

Thanks!

Reply | Threaded
Open this post in threaded view
|

Re: Structured Streaming: How to add a listener for when a batch is complete

tdas@databricks.com

On Tue, Sep 3, 2019, 3:26 PM Natalie Ruiz <[hidden email]> wrote:

Hello all,

 

I’m a beginner, new to Spark and wanted to know if there was an equivalent to Spark Streaming’s StreamingListenerBatchCompleted in Structured Streaming? I want to add a listener for when a batch is complete but the documentation and examples I find are for Spark Streaming and not Structured Streaming and registering it to a StreamingContext, which from my understanding is just for Spark Streaming

 

Thanks!

Reply | Threaded
Open this post in threaded view
|

Re: Structured Streaming: How to add a listener for when a batch is complete

Gourav Sengupta
Hi Natalie,
in case we are using Python then listener is not available, we can use the SPARK UI REST API's .

For that to happen we have to start the SPARK session with following settings: spark.sql("SET spark.sql.streaming.metricsEnabled=true")
For  the UI from which we can use the JSON results from the below:

In case you are working on AWS EMR, the UI URL is a bit different, let me know in that case and I will email to you.


Hi TD,
What will be the best way to stop a long running SPARK streaming job gracefully? I will be really grateful if you could kindly help us out.


Regards,
Gourav Sengupta

On Wed, Sep 4, 2019 at 2:28 AM Tathagata Das <[hidden email]> wrote:

On Tue, Sep 3, 2019, 3:26 PM Natalie Ruiz <[hidden email]> wrote:

Hello all,

 

I’m a beginner, new to Spark and wanted to know if there was an equivalent to Spark Streaming’s StreamingListenerBatchCompleted in Structured Streaming? I want to add a listener for when a batch is complete but the documentation and examples I find are for Spark Streaming and not Structured Streaming and registering it to a StreamingContext, which from my understanding is just for Spark Streaming

 

Thanks!