Kafka spark structure streaming out of memory issue

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Kafka spark structure streaming out of memory issue

km.santanu
Hi
I am using Kafka stateless structure streaming.i have enabled watermark as 1
hour.after long running about 2 hour my job is terminating
automatically.check point has been enabled.
I am doing average on input data.
Can you please suggest how to avoid out of memory error



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Kafka spark structure streaming out of memory issue

Srinivas V
It depends on how much memory is available and how much data you are processing. Please provide data size and cluster details to help.

On Fri, Aug 14, 2020 at 12:54 AM km.santanu <[hidden email]> wrote:
Hi
I am using Kafka stateless structure streaming.i have enabled watermark as 1
hour.after long running about 2 hour my job is terminating
automatically.check point has been enabled.
I am doing average on input data.
Can you please suggest how to avoid out of memory error



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]