[structured-streaming][kafka] Will the Kafka readstream timeout after connections.max.idle.ms 540000 ms ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[structured-streaming][kafka] Will the Kafka readstream timeout after connections.max.idle.ms 540000 ms ?

karthikjay
Hi all,

We are running into a scenario where the structured streaming job is exiting
after a while specifically when the Kafka topic is not getting any data.
From the job logs, I see this connections.max.idle.ms = 540000. Does that
mean the spark readstream will close when it does not get data for 540000
milliseconds ? If yes, how do I override this ?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [structured-streaming][kafka] Will the Kafka readstream timeout after connections.max.idle.ms 540000 ms ?

Shixiong(Ryan) Zhu
The streaming query should keep polling data from Kafka. When the query was stopped, did you see any exception?

Best Regards,

Shixiong Zhu
Databricks Inc.


On Tue, May 15, 2018 at 6:25 PM, karthikjay <[hidden email]> wrote:
Hi all,

We are running into a scenario where the structured streaming job is exiting
after a while specifically when the Kafka topic is not getting any data.
From the job logs, I see this connections.max.idle.ms = 540000. Does that
mean the spark readstream will close when it does not get data for 540000
milliseconds ? If yes, how do I override this ?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]