Apache Spark Connector for SQL Server and Azure SQL

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache Spark Connector for SQL Server and Azure SQL

alejandra.lemmo

Hi,

 

In a project where I work with Databricks, we use this connector to read / write data to Azure SQL Database. Currently with Spark 2.4.5 and Scala 2.11.

 

But those setups are getting old. What happens if we update Spark to 3.0.1 or higher and Scala 2.12.

This connector does not work according to the versions it supports. What should we do? Don't use the connector or is there another way to work?

 

I appreciate any type of information that helps me.

 

Med vänlig hälsning / Best regards

Alejandra Lemmo
Data Engineer

Customer Analytic

Address: Evenemangsgatan 13, 169 56 Solna, 16956 Solna

D +46735249832

M +46735249832


[hidden email]
www.vattenfall.se

Please consider the environment before printing this e-mail


Confidentiality: C2 - Internal

Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark Connector for SQL Server and Azure SQL

ayan guha
I would suggest to ask microsoft and databricks, this forum is for apache spark. 

if you are interested please drop me a note separately as I m keen to understand the issue as we use same setup

Ayan

On Mon, 26 Oct 2020 at 11:53 pm, <[hidden email]> wrote:

Hi,

 

In a project where I work with Databricks, we use this connector to read / write data to Azure SQL Database. Currently with Spark 2.4.5 and Scala 2.11.

 

But those setups are getting old. What happens if we update Spark to 3.0.1 or higher and Scala 2.12.

This connector does not work according to the versions it supports. What should we do? Don't use the connector or is there another way to work?

 

I appreciate any type of information that helps me.

 

Med vänlig hälsning / Best regards

Alejandra Lemmo
Data Engineer

Customer Analytic

Address: Evenemangsgatan 13, 169 56 Solna, 16956 Solna

D +46735249832

M +46735249832


[hidden email]
www.vattenfall.se

Please consider the environment before printing this e-mail


Confidentiality: C2 - Internal

--
Best Regards,
Ayan Guha
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark Connector for SQL Server and Azure SQL

Artemis User
In reply to this post by alejandra.lemmo

The best option certainly would be to recompile the Spark Connector for MS SQL server using the Spark 3.0.1/Scala 2.12 dependencies, and just fix the compiler errors as you go. The code is open source on github (https://github.com/microsoft/sql-spark-connector).  Looks like this connector is using Data Frame instead of RDD, so I would expect there shouldn't be many API changes that would cause compiler errors.

You may also want to go through the forks and it looks like some people have already tried to convert the connector to Spark 3.0.1, like this one https://github.com/datarootsio/sql-spark-connector-spark-3

-- ND

On 10/26/20 6:18 AM, [hidden email] wrote:

Hi,

 

In a project where I work with Databricks, we use this connector to read / write data to Azure SQL Database. Currently with Spark 2.4.5 and Scala 2.11.

 

But those setups are getting old. What happens if we update Spark to 3.0.1 or higher and Scala 2.12.

This connector does not work according to the versions it supports. What should we do? Don't use the connector or is there another way to work?

 

I appreciate any type of information that helps me.

 

Med vänlig hälsning / Best regards

Alejandra Lemmo
Data Engineer

Customer Analytic

Address: Evenemangsgatan 13, 169 56 Solna, 16956 Solna

D +46735249832


M +46735249832



[hidden email]
www.vattenfall.se

Please consider the environment before printing this e-mail


Confidentiality: C2 - Internal