Quantcast

Spark 2.1.0 issue with spark-shell and pyspark

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Spark 2.1.0 issue with spark-shell and pyspark

Aleksei Aleksinov
This post has NOT been accepted by the mailing list yet.
Dear Spark Community

I have an issue with Spark 2.1.0, the same worked fine with Spark 2.0.0 and Spark 2.0.2

With a fresh/clean install of Spark 2.1.0 (unmodified from the tar spark-2.1.0-bin-hadoop2.7) I receive the following errors when launching pyspark and spark-shell respectively:

pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':"

And with spark-shell:

java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState'

As mentioned, launching those two shells with Spark 2.0.2 (also unmodified from the respective tar) did not cause any issues.

I'm missing something, but I'm not sure what? Any help/hints greatly appreciated.

Many thanks in advance and with best wishes,

Alex
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark 2.1.0 issue with spark-shell and pyspark

jerrytim
This post has NOT been accepted by the mailing list yet.
I came across the same problem while I ran my code at "model.save(sc, path)"

Error info:
IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':"

My platform is Mac, I installed Spark with Hadoop prebuilt. Then I integrated PySpark with Jupyter.

Anyone has any ideas?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark 2.1.0 issue with spark-shell and pyspark

heramerom
This post has NOT been accepted by the mailing list yet.
In reply to this post by Aleksei Aleksinov
Also handle this error. Anyone else has idea?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark 2.1.0 issue with spark-shell and pyspark

romain.jouin
This post has NOT been accepted by the mailing list yet.
This post was updated on .
Hi,

master   = "spark://193.xx.xx.xx:7077"
appName  = "romain2"
spark = SparkSession.builder.master(master).appName(appName).getOrCreate()

also gives me an error :


IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':"


Any way out ?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark 2.1.0 issue with spark-shell and pyspark

timesking
This post has NOT been accepted by the mailing list yet.
I got the same problem on my Win10 x64.

The key to fix it, is you should download correct winutils.exe for your spark prebuild.


for example:
if you download spark-2.1.0-bin-hadoop2.7, you have to download winutils.exe in folder hadoop-2.7.1 of https://github.com/steveloughran/winutils.


do not forget to run "winutils.exe chmod 777 /tmp/hive" in CommandLine as Administrator, and in the correct hard drive.

you could get further information :
https://hernandezpaul.wordpress.com/2016/01/24/apache-spark-installation-on-windows-10/
http://letstalkspark.blogspot.kr/2016/02/getting-started-with-spark-on-window-64.html

Loading...