spark-sql on windows throws Exception in thread "main" java.lang.UnsatisfiedLinkError:

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

spark-sql on windows throws Exception in thread "main" java.lang.UnsatisfiedLinkError:

Mich Talebzadeh

Need to create some hive test tables for pyCharm

SPARK_HOME is set up as

D:\temp\spark-3.0.1-bin-hadoop2.7

HADOOP_HOME is

c:\hadoop\

spark-shell works. Trying to run spark-sql, I get the following errors

PS C:\tmp\hive> spark-sql
log4j:WARN No appenders could be found for logger (org.apache.hadoop.hive.conf.HiveConf).
log4j:WARN Please initialize the log4j system properly.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/11/16 21:12:34 INFO SharedState: loading hive config file: file:/D:/temp/spark-3.0.1-bin-hadoop2.7/conf/hive-site.xml
20/11/16 21:12:34 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/tmp/hive/warehouse').
20/11/16 21:12:34 INFO SharedState: Warehouse path is '/tmp/hive/warehouse'.
20/11/16 21:12:34 INFO SessionState: Created HDFS directory: /tmp/hive/admin/e32257b3-6c6b-46d7-921e-2782e2c15546
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
        at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
        at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:478)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:532)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:509)
        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:312)
        at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:751)
        at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:688)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:586)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:548)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:135)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/11/16 21:12:34 INFO ShutdownHookManager: Shutdown hook called
20/11/16 21:12:34 INFO ShutdownHookManager: Deleting directory C:\Users\admin\AppData\Local\Temp\spark-51bb8e21-5fd3-4b86-bb0b-5e331c92bd4f
PS C:\tmp\hive>


Any workarounds is appreciated.


Mich


LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.