why I have to specify the dependency on commons-configuration explicitly after compiling with a customized Hadoop

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

why I have to specify the dependency on commons-configuration explicitly after compiling with a customized Hadoop

Nan Zhu
Hi, all

Recently, I’m working on running Spark on a customized HDFS implementation. 

I installed the customized Hadoop jar on local maven repo and compile the spark successfully after adding the dependency in SparkBuild.scala

however, when I run the spark-shell, it always tell me that 

java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration

at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)

at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)

at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)

at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:217)

at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:185)

at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:237)

at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:79)

at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:210)

at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:227)

at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:34)

at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:79)

at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)

at org.apache.spark.SparkContext.<init>(SparkContext.scala:283)

at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:862)

at <init>(<console>:10)

at <init>(<console>:22)

at <init>(<console>:24)

at .<init>(<console>:28)

at .<clinit>(<console>)

at .<init>(<console>:7)

at .<clinit>(<console>)

at $export(<console>)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:629)

at org.apache.spark.repl.SparkIMain$Request$$anonfun$10.apply(SparkIMain.scala:897)

at scala.tools.nsc.interpreter.Line$$anonfun$1.apply$mcV$sp(Line.scala:43)

at scala.tools.nsc.io.package$$anon$2.run(package.scala:25)

at java.lang.Thread.run(Thread.java:744)

Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

... 31 more



I solved the issue by adding the dependency on commons-configuration in SparkBuild.scala

What I’m wondering is why I have to do this? 

Best,

-- 
Nan Zhu

Loading...