Quantcast

HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

Chang Lim
This post has NOT been accepted by the mailing list yet.
Hi,

I am on Spark 2.0 Review release.  According to Spark 2.0 docs, to share TempTable/View, I need to:
    "to run the Thrift server in the old single-session mode, please set option spark.sql.hive.thriftServer.singleSession to true."
Question: When using HiveThriftServer2.startWithContext(), where do I set the above setting?  I tried the following 4 possible places to set the flag but it does not seem to work.  What am I missing?

val spark = SparkSession.builder.master("local[2]")
   .enableHiveSupport().appName("App")
   .config("spark.sql.hive.thriftServer.singleSession", "true")  // <--- 1. is this the correct place to set??
   .getOrCreate()

//starts Thrift Server
//spark.conf.set("spark.sql.hive.thriftServer.singleSession", true)  <-- 2. Tried this, don't seem to work
//spark.sparkContext.getConf.set("spark.sql.hive.thriftServer.singleSession", "true")  <-- 3. Tried this, don't seem to work
val sql = new org.apache.spark.sql.SQLContext(spark.sparkContext)
sql.setConf("spark.sql.hive.thriftServer.singleSession", "true")  // <-- 4. Tried this, don't seem to work
HiveThriftServer2.startWithContext(sql)

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

Chang Lim
This post has NOT been accepted by the mailing list yet.
Would appreciate if someone:
1. Can confirm if this is an issue or
2. Share on how to get HiveThriftServer2.startWithContext working with shared temp table.

I am using Beeline as the JDBC client to access the temp tables of the running Spark app.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

Chang Lim
This post has NOT been accepted by the mailing list yet.
It's an issue with the preview build.  Switched to RC5 and all is working. Thanks to Michael Armbrust.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

Richard M
This post has NOT been accepted by the mailing list yet.
I am running HiveServer2 as well and when I connect with beeline I get the following:

 org.apache.spark.sql.internal.SessionState cannot be cast to org.apache.spark.sql.hive.HiveSessionState


Do you know how to resolve this?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

Chang Lim
This post has NOT been accepted by the mailing list yet.
What command did you use to connect?  Try this:

beeline>  !connect jdbc:hive2://localhost:10000?hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice



On Thu, Aug 11, 2016 at 9:23 AM, Richard M [via Apache Spark User List] <[hidden email]> wrote:
I am running HiveServer2 as well and when I connect with beeline I get the following:

 org.apache.spark.sql.internal.SessionState cannot be cast to org.apache.spark.sql.hive.HiveSessionState


Do you know how to resolve this?


To unsubscribe from HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting, click here.
NAML

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

Richard M
This post has NOT been accepted by the mailing list yet.
I was using the 1.1 driver. I upgraded that library to 2.1 and it resolved my problem.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HiveThriftServer and spark.sql.hive.thriftServer.singleSession setting

funes79
This post has NOT been accepted by the mailing list yet.
In reply to this post by Chang Lim
Hello Chang, you mentioned that the new release fixed tge issue and you can connect via jdbc and query a temp table. Can you please give me more details how tu run this and which way of configuration worked for you? Thanks
Loading...