Executor unable to pick postgres driver in Spark standalone cluster

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Executor unable to pick postgres driver in Spark standalone cluster

Rishikesh Teke
This post has NOT been accepted by the mailing list yet.

Hi all,

I was submitting the play application to spark 2.1 standalone cluster . In play application postgres dependency is also added and application works on local spark libraries. But at run time on standalone cluster it gives me error :

o.a.s.s.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 1, 172.31.21.3, executor 1): java.lang.ClassNotFoundException: org.postgresql
.Driver

I have placed following in spark-defaults.conf directory

spark.executor.extraClassPath   /home/ubuntu/downloads/postgres/postgresql-9.4-1200-jdbc41.jar
spark.driver.extraClassPath     /home/ubuntu/downloads/postgres/postgresql-9.4-1200-jdbc41.jar

Still executors unable to pick the driver.
Am i missing something? Need help .
Thanks.

Loading...