Kafka offset committer tool for structured streaming query

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Kafka offset committer tool for structured streaming query

Jungtaek Lim
Hi Spark users, especially Structured Streaming users who are using Kafka as data source,

I'm pleased to introduce Kafka offset committer, which enables commit offsets which batch has been processed. The tool is basically an implementation of streaming query listener, which listens for events and commit offsets for each batch. Please refer README.md in the repository to see more details.

Currently it hasn't be published to Maven central, so you might need to build the source and add jar via "--jars" option until artifact is published.
I'd be happy to hear new ideas of improvements, and much appreciated for contributions!

Enjoy!

Thanks,
Jungtaek Lim (HeartSaVioR)