Any impact on Driver out memory with long batch queue.

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Any impact on Driver out memory with long batch queue.

sd.hrishi
Hi All, 

Has anyone observed that having long streaming queues impact Driver memory. Our Spark Driver is getting killed due Java heap out memory (4GB). We are not doing any computation / aggregation in Driver just consuming from Kafka topics. But there around 100-500 queued batches. 

Spark Version: 2.4 
Kafka Direct stream. 

image.png


Regards
Hrishi