How to migrate DataSourceV2 into Spark 3.0.0

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to migrate DataSourceV2 into Spark 3.0.0

rafaelkyrdan

Hey guys,

 

I’m trying to migrate my package where I’m using DataSourceV2 into Spark 3.0.0

Unfortunately, neither migration guide nor JIRA ticket under which this API was refactored says nothing about how to do it.

https://issues.apache.org/jira/browse/SPARK-25390 I was suggested to send my question in the mailing list.

 

Here the gist file with an example which I will help to understand the issue https://gist.github.com/rafaelkyrdan/2bea8385aadd71be5bf67cddeec59581

In general a few notes about what changes happened with DataSourceV2 into Spark 3.0.0 and what interfaces we should use now would be helpful.

 

Thank you

--
Rafael Kyrdan

Skype:  raphael.kyrdan
Mob: +380633040707

Reply | Threaded
Open this post in threaded view
|

Re: How to migrate DataSourceV2 into Spark 3.0.0

rafaelkyrdan
Sorry for late response.
I was able to migrate my project to Spark 3.0.0
Here some hints what I did:
https://gist.github.com/rafaelkyrdan/2bea8385aadd71be5bf67cddeec59581



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]