ClassNotFoundException while unmarshalling a remote RDD on Spark 1.5.1

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

ClassNotFoundException while unmarshalling a remote RDD on Spark 1.5.1

PICARD Damien

Hi !

 

I’m facing a Classloader problem using Spark 1.5.1

 

I use javax.validation and hibernate validation annotations on some of my beans :

 

  @NotBlank

  @Valid

  private String attribute1 ;

 

  @Valid

  private String attribute2 ;

 

When Spark tries to unmarshall these beans (after a remote RDD), I get the ClassNotFoundException :

17/09/07 09:19:25 INFO storage.BlockManager: Found block rdd_8_1 remotely

17/09/07 09:19:25 ERROR executor.Executor: Exception in task 3.0 in stage 2.0 (TID 6)

java.lang.ClassNotFoundException: org.hibernate.validator.constraints.NotBlank

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:348)

        at java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:700)

        at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1566)

        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)

        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)

   …

 

Indeed, it means that the annotation class is not found, because it is not in the classpath. Why ? I don’t know, because I make a uber JAR that contains this class. I suppose that at the time the job tries to unmarshall the RDD, the uber jar is not loaded.

 

So, I try to add the hibernate JAR to the class loader manually, using this spark-submit command :

 

spark-submit --queue default \

--class com.my.Launcher \

--deploy-mode cluster \

--master yarn-cluster \

--driver-java-options "-Dfile.encoding=UTF-8" \

--jars /home/user/hibernate-validator-5.2.2.Final.jar \

--driver-class-path /home/user/hibernate-validator-5.2.2.Final.jar \

--conf "spark.executor.extraClassPath=/home/user/hibernate-validator-5.2.2.Final.jar" \

/home/user/uberjar-job.jar

 

Without effects. So, is there a way to add this class to the classloader ?

 

Thank you in advance.

 

Damien

 

 

=========================================================

Ce message et toutes les pieces jointes (ci-apres le "message")
sont confidentiels et susceptibles de contenir des informations
couvertes par le secret professionnel. Ce message est etabli
a l'intention exclusive de ses destinataires. Toute utilisation
ou diffusion non autorisee interdite.
Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE
et ses filiales declinent toute responsabilite au titre de ce message
s'il a ete altere, deforme falsifie.

=========================================================

This message and any attachments (the "message") are confidential,
intended solely for the addresses, and may contain legally privileged
information. Any unauthorized use or dissemination is prohibited.
E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any
of its subsidiaries or affiliates shall be liable for the message
if altered, changed or falsified.

=========================================================

Reply | Threaded
Open this post in threaded view
|

RE: ClassNotFoundException while unmarshalling a remote RDD on Spark 1.5.1

PICARD Damien

Ok, it just seems to be an issue with the syntax of the spark-submit command. It should be :

 

spark-submit --queue default \

--class com.my.Launcher \

--deploy-mode cluster \

--master yarn-cluster \

--driver-java-options "-Dfile.encoding=UTF-8" \

--jars /home/user/hibernate-validator-5.2.2.Final.jar \

--driver-class-path hibernate-validator-5.2.2.Final.jar \

--conf "spark.executor.extraClassPath=hibernate -validator-5.2.2.Final.jar" \

/home/user/uberjar-job.jar

 

I also have to add some others jars, like jboss-logging to meet the needs of hibernate-validator.

 

De : PICARD Damien (EXT) AssuResPriSmsAts
Envoyé : lundi 11 septembre 2017 08:53
À : '[hidden email]'
Objet : ClassNotFoundException while unmarshalling a remote RDD on Spark 1.5.1

 

Hi !

 

I’m facing a Classloader problem using Spark 1.5.1

 

I use javax.validation and hibernate validation annotations on some of my beans :

 

  @NotBlank

  @Valid

  private String attribute1 ;

 

  @Valid

  private String attribute2 ;

 

When Spark tries to unmarshall these beans (after a remote RDD), I get the ClassNotFoundException :

17/09/07 09:19:25 INFO storage.BlockManager: Found block rdd_8_1 remotely

17/09/07 09:19:25 ERROR executor.Executor: Exception in task 3.0 in stage 2.0 (TID 6)

java.lang.ClassNotFoundException: org.hibernate.validator.constraints.NotBlank

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:348)

        at java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:700)

        at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1566)

        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)

        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)

   …

 

Indeed, it means that the annotation class is not found, because it is not in the classpath. Why ? I don’t know, because I make a uber JAR that contains this class. I suppose that at the time the job tries to unmarshall the RDD, the uber jar is not loaded.

 

So, I try to add the hibernate JAR to the class loader manually, using this spark-submit command :

 

spark-submit --queue default \

--class com.my.Launcher \

--deploy-mode cluster \

--master yarn-cluster \

--driver-java-options "-Dfile.encoding=UTF-8" \

--jars /home/user/hibernate-validator-5.2.2.Final.jar \

--driver-class-path /home/user/hibernate-validator-5.2.2.Final.jar \

--conf "spark.executor.extraClassPath=/home/user/hibernate-validator-5.2.2.Final.jar" \

/home/user/uberjar-job.jar

 

Without effects. So, is there a way to add this class to the classloader ?

 

Thank you in advance.

 

Damien

 

 

=========================================================

Ce message et toutes les pieces jointes (ci-apres le "message")
sont confidentiels et susceptibles de contenir des informations
couvertes par le secret professionnel. Ce message est etabli
a l'intention exclusive de ses destinataires. Toute utilisation
ou diffusion non autorisee interdite.
Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE
et ses filiales declinent toute responsabilite au titre de ce message
s'il a ete altere, deforme falsifie.

=========================================================

This message and any attachments (the "message") are confidential,
intended solely for the addresses, and may contain legally privileged
information. Any unauthorized use or dissemination is prohibited.
E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any
of its subsidiaries or affiliates shall be liable for the message
if altered, changed or falsified.

=========================================================