This post has NOT been accepted by the mailing list yet.
The issue seems to be with primordial class loader. I cannot load the drivers to all the nodes at the same location but have loaded the jars to HDFS. I have tried SPARK_YARN_DIST_FILES as well as SPARK_CLASSPATH on the edge node with no luck. Is there another way to load these jars through primordial class loader in YARN CLUSTER mode ? or do I have to add them inside SPARK assembly jar?