Trouble on runnig spark's hbasetest example

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Trouble on runnig spark's hbasetest example

林武康
Hi all,
I got a trouble on running Spark's HBaseTest example, It failed with Exception below:

14/02/14 14:41:58 DEBUG util.Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:225)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:250)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:345)
at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.newConfiguration(YarnSparkHadoopUtil.scala:36)
at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:32)
at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.<init>(YarnSparkHadoopUtil.scala:29)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:76)
at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:74)
at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

I have setted HADOOP_HOME on /root/.bash_profile and /root/.bashrc, It not works.
I changed the src of this excample, add HADOOP_HOME in the SparkContext like this:
    val sc = new SparkContext(args(0),
         "HBaseTest",
       System.getenv("SPARK_HOME"),
       SparkContext.jarOfClass(this.getClass),
       Map("HADOOP_HOME" -> "/usr/lib/cdh5/hadoop"))

It still not work, what's wrong with this? how can I run this successfully?
Thank you for any help.


--------------------------------------------------------------------------------

Best Regards!