assertion failed: Beginning offset 34242088 is after the ending offset 34242084 for topic <topic_name> partition 2. You either provided an invalid fromOffset, or the Kafka topic has been damaged

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

assertion failed: Beginning offset 34242088 is after the ending offset 34242084 for topic <topic_name> partition 2. You either provided an invalid fromOffset, or the Kafka topic has been damaged

ravidspark
Hi Community,

Seeing the below message in the logs and Spark application is getting
terminated. There is an issue with our Kafka service and it auto restarts
during which leader reelection happens.

*Exception:*
assertion failed: Beginning offset 34242088 is after the ending offset
34242084 for topic <topic_name> partition 2. You either provided an invalid
fromOffset, or the Kafka topic has been damaged.

After analyzing it, I understand that the problem is that the Spark consumer
has an earliest offset which it processed and Kafka returns an old offset
(correct me if I am wrong). Is this behavior intended? Is there a way we can
handle these situations without letting the app die?

Any inputs are greatly appreciated.

Thanks,
Ravi



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]