[Structured streaming, V2] commit on ContinuousReader

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[Structured streaming, V2] commit on ContinuousReader

Version: 2.3, DataSourceV2, ContinuousReader


We're creating a new data source to fetch data from streaming source that requires commiting received data and we would like to commit data once in a while after it has been retrieved and correctly processed and then fetch more. 

One option could be to rely on spark committing already read data using commit(end: Offset) that is present in ContinuousReader (v2.reader.streaming), but it seems that this method is never called.

The question is if this method commit(end: Offset) is ever used and when? I went through part of Spark code base, but haven't really found any place where it could be called.