DataSourceV2 with ordering pushdown

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

DataSourceV2 with ordering pushdown

Kohki Nishio
I'm trying to connect Spark with Lucene indices and noticed that I can't really tell what ordering Spark can expect from my Batch / PartitionReader.

Spark ended up retrieving all rows then doing ordering if there's any orderBy, is there anyway I can tell spark that this partition is ordered ? Is working with a physical plan the only way to achieve this ?

Thanks
-- 
Kohki Nishio