[Structured streaming, V2] commit on ContinuousReader
Version: 2.3, DataSourceV2, ContinuousReader
We're creating a new data source to fetch data from streaming source that requires commiting received data and we would like to commit data once in a while after it has been retrieved and correctly processed and then fetch more.
One option could be to rely on spark committing already read data using commit(end: Offset) that is present in ContinuousReader (v2.reader.streaming), but it seems that this method is never called.
The question is if this method commit(end: Offset) is ever used and when? I went through part of Spark code base, but haven't really found any place where it could be called.