[Structured Streaming] Last processed event time always behind Direct Streaming

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[Structured Streaming] Last processed event time always behind Direct Streaming

subramgr
Hi

We are migrating our Direct Streaming Spark job to Structured Streaming. We
have a batch size of 1 minute.

I am consistently seeing that the Structured Streaming job is always (3-5
minutes) behind the Direct Streaming job. Is there some kinda fine tuning
that will help Structured Streaming catch up ?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]