How to implement "getPreferredLocations" in Data source v2?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to implement "getPreferredLocations" in Data source v2?

kineret M
Hi, 
I would like to support data locality in Spark data source v2. How can I provide Spark the ability to read and process data on the same node? 

I didn't find any interface that supports 'getPreferredLocations' (or equivalent).

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: How to implement "getPreferredLocations" in Data source v2?

Russell Spitzer

On Sat, Jan 18, 2020, 1:44 PM kineret M <[hidden email]> wrote:
Hi, 
I would like to support data locality in Spark data source v2. How can I provide Spark the ability to read and process data on the same node? 

I didn't find any interface that supports 'getPreferredLocations' (or equivalent).

Thanks!