Custom JdbcConnectionProvider

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Custom JdbcConnectionProvider

rafaelkyrdan
Guys do you know how I can use the custom implementation of
JdbcConnectionProvider?

As far as I understand in the spark jdbc we can use custom Driver, like
this:
*val jdbcDF = spark.read
  .format("jdbc")
  .option("url", "jdbc:postgresql:dbserver").option("driver", "my.drivier")
*
And we need a matching JdbcConnectionProvider which will override the
property:
* override val driverClass = "my.driver"*

I have both but I see that they are not used. Do I need to register somehow
them? Could someone share a relevant example?
Thx.




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Custom JdbcConnectionProvider

Takeshi Yamamuro
Hi,

Since it depends on the service loader, I think you need to add a configuration file in META-INF/services.

Bests,
Takeshi

On Tue, Oct 27, 2020 at 9:50 PM rafaelkyrdan <[hidden email]> wrote:
Guys do you know how I can use the custom implementation of
JdbcConnectionProvider?

As far as I understand in the spark jdbc we can use custom Driver, like
this:
*val jdbcDF = spark.read
  .format("jdbc")
  .option("url", "jdbc:postgresql:dbserver").option("driver", "my.drivier")
*
And we need a matching JdbcConnectionProvider which will override the
property:
* override val driverClass = "my.driver"*

I have both but I see that they are not used. Do I need to register somehow
them? Could someone share a relevant example?
Thx.




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Custom JdbcConnectionProvider

Gabor Somogyi
Thanks Takeshi for sharing it, that can be used as an example.
The user and developer guide will come soon...

On Tue, Oct 27, 2020 at 2:31 PM Takeshi Yamamuro <[hidden email]> wrote:
Hi,

Since it depends on the service loader, I think you need to add a configuration file in META-INF/services.

Bests,
Takeshi

On Tue, Oct 27, 2020 at 9:50 PM rafaelkyrdan <[hidden email]> wrote:
Guys do you know how I can use the custom implementation of
JdbcConnectionProvider?

As far as I understand in the spark jdbc we can use custom Driver, like
this:
*val jdbcDF = spark.read
  .format("jdbc")
  .option("url", "jdbc:postgresql:dbserver").option("driver", "my.drivier")
*
And we need a matching JdbcConnectionProvider which will override the
property:
* override val driverClass = "my.driver"*

I have both but I see that they are not used. Do I need to register somehow
them? Could someone share a relevant example?
Thx.




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Custom JdbcConnectionProvider

Takeshi Yamamuro
> the user and developer guide will come soon...

Yea, it looks nice! Thanks for the work.

On Tue, Oct 27, 2020 at 11:21 PM Gabor Somogyi <[hidden email]> wrote:
Thanks Takeshi for sharing it, that can be used as an example.
The user and developer guide will come soon...

On Tue, Oct 27, 2020 at 2:31 PM Takeshi Yamamuro <[hidden email]> wrote:
Hi,

Since it depends on the service loader, I think you need to add a configuration file in META-INF/services.

Bests,
Takeshi

On Tue, Oct 27, 2020 at 9:50 PM rafaelkyrdan <[hidden email]> wrote:
Guys do you know how I can use the custom implementation of
JdbcConnectionProvider?

As far as I understand in the spark jdbc we can use custom Driver, like
this:
*val jdbcDF = spark.read
  .format("jdbc")
  .option("url", "jdbc:postgresql:dbserver").option("driver", "my.drivier")
*
And we need a matching JdbcConnectionProvider which will override the
property:
* override val driverClass = "my.driver"*

I have both but I see that they are not used. Do I need to register somehow
them? Could someone share a relevant example?
Thx.




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
---
Takeshi Yamamuro


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: Custom JdbcConnectionProvider

rafaelkyrdan
In reply to this post by Takeshi Yamamuro
Thank you for a quick response.
Your example is clear but I wasn't able to extend *JdbcConnectionProvider  *
in my application. I'm getting compile errors:

/[error]
C:\Users\Rafael_Kyrdan\projects\tdm_engine2\src\main\scala\com\tdm\engine\connection\BasicConnectionProvider.scala:10:39:
not found: type JdbcConnectionProvider
[error] class BasicConnectionProvider extends JdbcConnectionProvider with
Logging {
[error]                                       ^/

/[error]
C:\Users\Rafael_Kyrdan\projects\tdm_engine2\src\main\scala\com\spark\ExampleJdbcConnectionProvider.scala:8:45:
not found: type JdbcConnectionProvider
[error] class ExampleJdbcConnectionProvider extends JdbcConnectionProvider
with Logging {/

Now I noticed that it is marked as developer API. Does it mean that I should
do only fork of spark? Or what I need to do to be able to extend it?

My context: In my application, I'm using spark JDBC to read and write data
to different DBs. I wanna add a custom connection provider to connect to DB
where JDBC URL is not enough.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Custom JdbcConnectionProvider

rafaelkyrdan
In reply to this post by Takeshi Yamamuro
I cloned your example and it is also cannot be compiled
<http://apache-spark-user-list.1001560.n3.nabble.com/file/t10977/Screenshot_2020-10-28_182213.png>

[ERROR]
C:\Users\Rafael_Kyrdan\projects\spark-jdbc-connection-provider\src\main\scala\com\spark\ExampleJdbcConnectionProvider.scala:23:
error: object JdbcConnectionProvider is not a member of package
org.apache.spark.sql.jdbc
[ERROR] import org.apache.spark.sql.jdbc.JdbcConnectionProvider
[ERROR]        ^
[ERROR]
C:\Users\Rafael_Kyrdan\projects\spark-jdbc-connection-provider\src\main\scala\com\spark\ExampleJdbcConnectionProvider.scala:25:
error: not found: type JdbcConnectionProvider
[ERROR] class ExampleJdbcConnectionProvider extends JdbcConnectionProvider
with Logging {
[ERROR]                                             ^
[ERROR] two errors found
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time:  37.339 s
[INFO] Finished at: 2020-10-28T18:21:08+02:00
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:3.3.2:compile (default) on project
spark-jdbc-connection-provider: wrap:
org.apache.commons.exec.ExecuteException: Process exited with an error: 1
(Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

C:\Users\Rafael_Kyrdan\projects\spark-jdbc-connection-provider>

Could you highlight what am I missing? How the custom connection provider
could be used properly?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Custom JdbcConnectionProvider

Gabor Somogyi
Please use the latest snapshot as dependency + update force the snapshots.

On Wed, Oct 28, 2020 at 5:42 PM rafaelkyrdan <[hidden email]> wrote:
I cloned your example and it is also cannot be compiled
<http://apache-spark-user-list.1001560.n3.nabble.com/file/t10977/Screenshot_2020-10-28_182213.png>

[ERROR]
C:\Users\Rafael_Kyrdan\projects\spark-jdbc-connection-provider\src\main\scala\com\spark\ExampleJdbcConnectionProvider.scala:23:
error: object JdbcConnectionProvider is not a member of package
org.apache.spark.sql.jdbc
[ERROR] import org.apache.spark.sql.jdbc.JdbcConnectionProvider
[ERROR]        ^
[ERROR]
C:\Users\Rafael_Kyrdan\projects\spark-jdbc-connection-provider\src\main\scala\com\spark\ExampleJdbcConnectionProvider.scala:25:
error: not found: type JdbcConnectionProvider
[ERROR] class ExampleJdbcConnectionProvider extends JdbcConnectionProvider
with Logging {
[ERROR]                                             ^
[ERROR] two errors found
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time:  37.339 s
[INFO] Finished at: 2020-10-28T18:21:08+02:00
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:3.3.2:compile (default) on project
spark-jdbc-connection-provider: wrap:
org.apache.commons.exec.ExecuteException: Process exited with an error: 1
(Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

C:\Users\Rafael_Kyrdan\projects\spark-jdbc-connection-provider>

Could you highlight what am I missing? How the custom connection provider
could be used properly?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Custom JdbcConnectionProvider

Mich Talebzadeh
In reply to this post by rafaelkyrdan
I think you can pickup your custom build driver from the command line itself

Here I am using a custom build third-party driver to access Oracle Table on-premisses from cloud

val jdbUrl = "jdbc:datadirect:ddhybrid://"+HybridServer+":"+HybridPort+";hybridDataPipelineDataSource="+ hybridDataPipelineDataSource+";datasourceUserId="+OracleUserName+";datasourcePassword="+OracleUserPassword+";encryptionMethod=noEncryption;"
   
// Read Oracle table on prem
val OracleDF = Try(
     spark.read.
     format("jdbc").
     option("url", jdbUrl).
     option("dbtable", OracleSchema+"."+OracleTable).
     option("user", HybridServerUserName).
     option("password", HybridServerPassword).
     load()
    ) match {
                   case Success(df) => df
                   case Failure(e) =>
                     println(e)
                     sys.exit(1)
     }

if (OracleDF.take(1).isEmpty){
  println("\nSource table "+OracleSchema+"."+OracleTable +" is empty. exiting")
  sys.exit(1)
}

However, on spark-submit, I explicitly specify the driver class path

 spark-shell --driver-class-path /home/hduser/jars/ddhybrid.jar --jars ..

Assuming I understood your question correctly.

HTH



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 



On Tue, 27 Oct 2020 at 12:51, rafaelkyrdan <[hidden email]> wrote:
Guys do you know how I can use the custom implementation of
JdbcConnectionProvider?

As far as I understand in the spark jdbc we can use custom Driver, like
this:
*val jdbcDF = spark.read
  .format("jdbc")
  .option("url", "jdbc:postgresql:dbserver").option("driver", "my.drivier")
*
And we need a matching JdbcConnectionProvider which will override the
property:
* override val driverClass = "my.driver"*

I have both but I see that they are not used. Do I need to register somehow
them? Could someone share a relevant example?
Thx.




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Custom JdbcConnectionProvider

rafaelkyrdan
In reply to this post by rafaelkyrdan
Thank you Gabor Somogyi for highlighting the issue.

I cannot compile the example because 3.1.0-SNAPSHOT is not available in
maven.
What resolvers do you use? I tried one which suggested in SBT documentation
SBT documentation
<https://www.scala-sbt.org/1.x/docs/Resolvers.html#Maven+resolvers>   but no
luck.

*resolvers +=
  "Sonatype OSS Snapshots" at
"https://oss.sonatype.org/content/repositories/snapshots"*



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Custom JdbcConnectionProvider

Gabor Somogyi
I've compiled the latest master locally and then put it into my m2 repo.

mvn clean install...


On Wed, Oct 28, 2020 at 8:36 PM rafaelkyrdan <[hidden email]> wrote:
Thank you Gabor Somogyi for highlighting the issue.

I cannot compile the example because 3.1.0-SNAPSHOT is not available in
maven.
What resolvers do you use? I tried one which suggested in SBT documentation
SBT documentation
<https://www.scala-sbt.org/1.x/docs/Resolvers.html#Maven+resolvers>   but no
luck.

*resolvers +=
  "Sonatype OSS Snapshots" at
"https://oss.sonatype.org/content/repositories/snapshots"*



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]