Spark 3.0.0-preview and s3a

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark 3.0.0-preview and s3a

vincent gromakowski
Hi Spark users,
I am testing the preview of Spark 3 with s3a and hadoop 3.2 but I have got NoClassDefFoundError and cannot find what is the issue. I suppose there is some lib conflict. Can someone provide a working configuration?

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:816)
at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:792)



Here is my SBT file

libraryDependencies ++= {
Seq(
"org.apache.spark" %% "spark-core" % "3.0.0-preview" % "provided",
"org.apache.spark" %% "spark-sql" % "3.0.0-preview" % "provided",
"org.apache.hadoop" % "hadoop-cloud-storage" % "3.2.1",
"org.scalactic" %% "scalactic" % "3.1.0",
"org.scalatest" %% "scalatest" % "3.1.0" % Test
)
}

assemblyShadeRules in assembly := Seq(
ShadeRule.rename("javax.xml.stream.**" -> "shaded-javax.xml.stream.@1").inLibrary("javax.xml.stream" % "stax-api" % "1.0-2"),
ShadeRule.rename("*" -> "shaded-@1").inLibrary("com.fasterxml.jackson.core" % "jackson-core" % "2.10.0"),
ShadeRule.rename("*" -> "shaded2-@1").inLibrary("com.fasterxml.jackson.core" % "jackson-databind" % "2.10.0"),
ShadeRule.rename("mozilla.**" -> "shaded-mozilla.@1").inLibrary("com.amazonaws" % "aws-java-sdk-bundle" % "1.11.375"),
)

assemblyMergeStrategy in assembly := {
case "mime.types" => MergeStrategy.rename
case x if x.contains("versions.properties") => MergeStrategy.rename
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}