Error: NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT while running a Spark-Hive Job

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Error: NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT while running a Spark-Hive Job

Rishikesh Gawade
Hello there,
I am using spark-2.3.0 compiled using the following Maven Command:
mvn -Pyarn -Phive -Phive-thriftserver -DskipTests clean install.
I have configured it to run with Hive v2.3.3. Also, all the Hive related jars (v1.2.1) in the Spark's JAR folder have been replaced by all the JARs available in Hive's lib folder and i have also configured Hive to use spark as an execution engine.

After that, in my Java application i have written the following lines:
SparkSession spark = SparkSession
.builder()
.appName("Java Spark Hive Example")
.config("hive.metastore.warehouse.dir","/user/hive/warehouse")
.config("hive.metastore.uris","thrift://hadoopmaster:9083")
.enableHiveSupport()
.getOrCreate();
HiveContext hc = new HiveContext(spark);
hc.sql("SHOW DATABASES").show();
I built the project thereafter (no compilation errors) and tried running the job using spark-submit command as follows:

spark-submit --master yarn --class org.adbms.SpamFilter IdeaProjects/mlproject/target/mlproject-1.0-SNAPSHOT.jar

On doing so, i received the following error:
Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
    at org.apache.spark.sql.hive.HiveUtils$.formatTimeVarsForHiveClient(HiveUtils.scala:205)
    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:286)
    at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
    at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:195)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:195)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:195)
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
    at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:194)
    and more...


I have no idea what this is about. Is this because Spark v2.3.0 and Hive v2.3.3 aren't compatible with each other?
If i have done anything wrong, i request you to point it out and suggest me the required changes. Also, if it's the case that i might have misconfigured spark and hive, please suggest me the changes in configuration, a link guiding through all necessary configs would also be appreciated.
Thank you in anticipation.
Regards,
Rishikesh Gawade