spark stream kafka wait for all data process done

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

spark stream kafka wait for all data process done

zenglong chen
How can kafka wait for tasks process done then begin receive next batch?I want to process 5000 record once by pandas and it may take too long time to process.
Reply | Threaded
Open this post in threaded view
|

Re: spark stream kafka wait for all data process done

刘 勇
Hi,
You can set spark.streaming.kafka.backpressure.enable=true.
If your tasks can't process larger data that this variable can control the kafka data into streaming speed. And you can increment your streaming process time window.



Sent from my Samsung Galaxy smartphone.


-------- Original message --------
From: zenglong chen <[hidden email]>
Date: 8/2/19 09:59 (GMT+08:00)
Subject: spark stream kafka wait for all data process done

How can kafka wait for tasks process done then begin receive next batch?I want to process 5000 record once by pandas and it may take too long time to process.