Spark streaming with kafka input stuck in (Re-)joing group because of group rebalancing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark streaming with kafka input stuck in (Re-)joing group because of group rebalancing

Junfeng Chen
When I terminate a spark streaming application and restart it, it always stuck in this step:
Revoking previously assigned partitions [] for group [mygroup]
(Re-)joing group [mygroup]

If I use a new group id, even though it works fine, I may lose the data from the last time I read the previous group id. 

So how to solve it? 

 
Regard,
Junfeng Chen