spark-sql on windows throws Exception in thread "main" java.lang.UnsatisfiedLinkError:

[Mich Talebzadeh]

Need to create some hive test tables for pyCharm

SPARK_HOME is set up as

D:\temp\spark-3.0.1-bin-hadoop2.7

HADOOP_HOME is

c:\hadoop\

spark-shell works. Trying to run spark-sql, I get the following errors

PS C:\tmp\hive> spark-sql

log4j:WARN No appenders could be found for logger (org.apache.hadoop.hive.conf.HiveConf).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

20/11/16 21:12:34 INFO SharedState: loading hive config file: file:/D:/temp/spark-3.0.1-bin-hadoop2.7/conf/hive-site.xml

20/11/16 21:12:34 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/tmp/hive/warehouse').

20/11/16 21:12:34 INFO SharedState: Warehouse path is '/tmp/hive/warehouse'.

20/11/16 21:12:34 INFO SessionState: Created HDFS directory: /tmp/hive/admin/e32257b3-6c6b-46d7-921e-2782e2c15546

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V

        at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)

        at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)

        at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:478)

        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:532)

        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:509)

        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:312)

        at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:751)

        at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:688)

        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:586)

        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:548)

        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:135)

        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)

        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)

        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)

        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)

        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

20/11/16 21:12:34 INFO ShutdownHookManager: Shutdown hook called

20/11/16 21:12:34 INFO ShutdownHookManager: Deleting directory C:\Users\admin\AppData\Local\Temp\spark-51bb8e21-5fd3-4b86-bb0b-5e331c92bd4f

PS C:\tmp\hive>