Dependency error due to scala version mismatch in SBT and Spark 2.1

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Dependency error due to scala version mismatch in SBT and Spark 2.1

patel kumar
Hi,

I am using CDH cluster with Spark 2.1 with Scala Version 2.11.8.
sbt version is 1.0.2.

While doing assembly , I am getting error as 
[error] java.lang.RuntimeException: Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.
modules:scala-parser-combinators

I tried to override the version mismatch using dependencyOverrides and force(), but none of the solution worked.

Please help me to resolve this version Conflict.

Details of the configuration are mentioned below :- 

build.sbt 

***********************************************************************
name := "newtest"
version := "0.0.2"

scalaVersion := "2.11.8" 

sbtPlugin := true

val sparkVersion = "2.1.0"

mainClass in (Compile, run) := Some("com.testpackage.sq.newsparktest")

assemblyJarName in assembly := "newtest.jar"


libraryDependencies ++= Seq(
   "org.apache.spark" % "spark-core_2.11" % "2.1.1" % "provided",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.1" % "provided",
  "com.databricks" % "spark-avro_2.11" % "3.2.0",
    "org.apache.spark" % "spark-hive_2.11" % "2.1.1" % "provided"
   )

 
libraryDependencies +=
     "log4j" % "log4j" % "1.2.15" excludeAll(
       ExclusionRule(organization = "com.sun.jdmk"),
       ExclusionRule(organization = "com.sun.jmx"),
       ExclusionRule(organization = "javax.jms")
     )
     
resolvers += "SparkPackages" at "https://dl.bintray.com/spark-packages/maven/"
resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)

assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
}      
   
***************************************************
plugins.sbt

**********************************

dependencyOverrides += ("org.scala-lang.modules" % "scala-xml_2.11" % "1.0.4")
dependencyOverrides += ("org.scala-lang.modules" % "scala-parser-combinators_2.11" % "1.0.4")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
resolvers += Resolver.url("bintray-sbt-plugins", url("https://dl.bintray.com/eed3si9n/sbt-plugins/"))(Resolver.ivyStylePatterns)

*****************************************************
Error Message after assembly

**************************************************
[error] Modules were resolved with conflicting cross-version suffixes in {file:/D:/Tools/scala_ide/test_workspace/test/NewSp
arkTest/}newsparktest:
[error]    org.scala-lang.modules:scala-xml _2.11, _2.12
[error]    org.scala-lang.modules:scala-parser-combinators _2.11, _2.12
[error] java.lang.RuntimeException: Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.
modules:scala-parser-combinators
[error]         at scala.sys.package$.error(package.scala:27)
[error]         at sbt.librarymanagement.ConflictWarning$.processCrossVersioned(ConflictWarning.scala:39)
[error]         at sbt.librarymanagement.ConflictWarning$.apply(ConflictWarning.scala:19)
[error]         at sbt.Classpaths$.$anonfun$ivyBaseSettings$64(Defaults.scala:1971)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:42)
[error]         at sbt.std.Transform$$anon$4.work(System.scala:64)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:257)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error]         at sbt.Execute.work(Execute.scala:266)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:257)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:167)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:32)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] (*:update) Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.modules:scala-par
ser-combinators
[error] Total time: 413 s, completed Oct 12, 2017 3:28:02 AM

**************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Dependency error due to scala version mismatch in SBT and Spark 2.1

patel kumar
 This is not the correct way to build Spark with sbt. Why ?


On Sun, Oct 15, 2017 at 11:54 PM, Mark Hamstra <[hidden email]> wrote:
I am building Spark using build.sbt.

Which just gets me back to my original question: Why? This is not the correct way to build Spark with sbt. 

On Sun, Oct 15, 2017 at 11:40 PM, patel kumar <[hidden email]> wrote:
I am building Spark using build.sbt. Details are mentioned in the original mail chain.

When I was using Spark1.6 using scala 2.10, everything was working fine.

Issue arises when I am updating my code to make it compatible with spark 2.1.

It is failing while doing sbt assembly.



On Sun, Oct 15, 2017 at 11:29 PM, Mark Hamstra <[hidden email]> wrote:
I don't understand. Are you building Spark or something else?

If you are trying to build Spark, it doesn't look like you are doing it the right way. As I mentioned before, and is explained in detail in the link I provided, building Spark with sbt is done via build/sbt, not by directly invoking your choice of an sbt version.  

On Sun, Oct 15, 2017 at 11:26 PM, patel kumar <[hidden email]> wrote:
Because earlier, I was using sbt 0.13.13.1, and I was getting another version conflict i.e.

[error] Modules were resolved with conflicting cross-version suffixes in {file:/D:/Tools/scala_ide/test_workspace/test/NewSp
arkTest/}newsparktest:
[error]    org.json4s:json4s-ast _2.10, _2.11
[error]    org.json4s:json4s-core _2.10, _2.11
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) Conflicting cross-version suffixes in: org.json4s:json4s-ast, org.json4s:json4s-core


Actually issue is because of SBT compilation. sbt 0.13.x is originally compiled with scala 2.10 and gives support to 2.11.x.

When I execute sbt , all the default jars are automatically downloaded with suffuix _10 (for scala 2.10). When execute assembly json4s jars for 2.11 is also downloaded.

I upgraded the sbt with the hope that this issue might be fixed with new version of sbt.

Sbt 1.02 introduced the same problem for XML related jars as sbt 1.0.2 originally compiled with scala 2.12.

I looked up the sbt version which would by default download 2.11.x jars, but failed to find any such version.

So, now I am looking for a solution by which I could override the jars that are downloaded by default during sbt.


On Sun, Oct 15, 2017 at 11:03 PM, Mark Hamstra <[hidden email]> wrote:
sbt version is 1.0.2.

Why?

Building Spark with sbt is done via build/sbt, which will give you sbt 0.13.11 when building Spark 2.1.0. 

 

On Sun, Oct 15, 2017 at 10:42 PM, patel kumar <[hidden email]> wrote:
Hi,

I am using CDH cluster with Spark 2.1 with Scala Version 2.11.8.
sbt version is 1.0.2.

While doing assembly , I am getting error as 
[error] java.lang.RuntimeException: Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.
modules:scala-parser-combinators

I tried to override the version mismatch using dependencyOverrides and force(), but none of the solution worked.

Please help me to resolve this version Conflict.

Details of the configuration are mentioned below :- 

build.sbt 

***********************************************************************
name := "newtest"
version := "0.0.2"

scalaVersion := "2.11.8" 

sbtPlugin := true

val sparkVersion = "2.1.0"

mainClass in (Compile, run) := Some("com.testpackage.sq.newsparktest")

assemblyJarName in assembly := "newtest.jar"


libraryDependencies ++= Seq(
   "org.apache.spark" % "spark-core_2.11" % "2.1.1" % "provided",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.1" % "provided",
  "com.databricks" % "spark-avro_2.11" % "3.2.0",
    "org.apache.spark" % "spark-hive_2.11" % "2.1.1" % "provided"
   )

 
libraryDependencies +=
     "log4j" % "log4j" % "1.2.15" excludeAll(
       ExclusionRule(organization = "com.sun.jdmk"),
       ExclusionRule(organization = "com.sun.jmx"),
       ExclusionRule(organization = "javax.jms")
     )
     
resolvers += "SparkPackages" at "https://dl.bintray.com/spark-packages/maven/"
resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)

assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
}      
   
***************************************************
plugins.sbt

**********************************

dependencyOverrides += ("org.scala-lang.modules" % "scala-xml_2.11" % "1.0.4")
dependencyOverrides += ("org.scala-lang.modules" % "scala-parser-combinators_2.11" % "1.0.4")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
resolvers += Resolver.url("bintray-sbt-plugins", url("https://dl.bintray.com/eed3si9n/sbt-plugins/"))(Resolver.ivyStylePatterns)

*****************************************************
Error Message after assembly

**************************************************
[error] Modules were resolved with conflicting cross-version suffixes in {file:/D:/Tools/scala_ide/test_workspace/test/NewSp
arkTest/}newsparktest:
[error]    org.scala-lang.modules:scala-xml _2.11, _2.12
[error]    org.scala-lang.modules:scala-parser-combinators _2.11, _2.12
[error] java.lang.RuntimeException: Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.
modules:scala-parser-combinators
[error]         at scala.sys.package$.error(package.scala:27)
[error]         at sbt.librarymanagement.ConflictWarning$.processCrossVersioned(ConflictWarning.scala:39)
[error]         at sbt.librarymanagement.ConflictWarning$.apply(ConflictWarning.scala:19)
[error]         at sbt.Classpaths$.$anonfun$ivyBaseSettings$64(Defaults.scala:1971)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:42)
[error]         at sbt.std.Transform$$anon$4.work(System.scala:64)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:257)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error]         at sbt.Execute.work(Execute.scala:266)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:257)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:167)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:32)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] (*:update) Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.modules:scala-par
ser-combinators
[error] Total time: 413 s, completed Oct 12, 2017 3:28:02 AM

**************************************************






Reply | Threaded
Open this post in threaded view
|

Re: Dependency error due to scala version mismatch in SBT and Spark 2.1

Mark Hamstra
The canonical build of Spark is done using maven, not sbt. Maven and sbt do things a bit differently. In order to get maven and sbt to each build Spark quite similar to the way the other does, the builds are each driven through a customization script -- build/mvn and build/sbt respectively. A lot of work has gone into those scripts, configurations, etc. They work. Using an arbitrary sbt version outside of the defined Spark build process is going to lead to a lot of issues that are completely avoided by using the documented Spark build procedure. So, back to my original question: Why? Why do something that doesn't work when there is a well-defined, well-maintained, documented way to build Spark with either maven or sbt? https://spark.apache.org/docs/latest/building-spark.html
 
On Mon, Oct 16, 2017 at 12:26 AM, patel kumar <[hidden email]> wrote:
 This is not the correct way to build Spark with sbt. Why ?


On Sun, Oct 15, 2017 at 11:54 PM, Mark Hamstra <[hidden email]> wrote:
I am building Spark using build.sbt.

Which just gets me back to my original question: Why? This is not the correct way to build Spark with sbt. 

On Sun, Oct 15, 2017 at 11:40 PM, patel kumar <[hidden email]> wrote:
I am building Spark using build.sbt. Details are mentioned in the original mail chain.

When I was using Spark1.6 using scala 2.10, everything was working fine.

Issue arises when I am updating my code to make it compatible with spark 2.1.

It is failing while doing sbt assembly.



On Sun, Oct 15, 2017 at 11:29 PM, Mark Hamstra <[hidden email]> wrote:
I don't understand. Are you building Spark or something else?

If you are trying to build Spark, it doesn't look like you are doing it the right way. As I mentioned before, and is explained in detail in the link I provided, building Spark with sbt is done via build/sbt, not by directly invoking your choice of an sbt version.  

On Sun, Oct 15, 2017 at 11:26 PM, patel kumar <[hidden email]> wrote:
Because earlier, I was using sbt 0.13.13.1, and I was getting another version conflict i.e.

[error] Modules were resolved with conflicting cross-version suffixes in {file:/D:/Tools/scala_ide/test_workspace/test/NewSp
arkTest/}newsparktest:
[error]    org.json4s:json4s-ast _2.10, _2.11
[error]    org.json4s:json4s-core _2.10, _2.11
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) Conflicting cross-version suffixes in: org.json4s:json4s-ast, org.json4s:json4s-core


Actually issue is because of SBT compilation. sbt 0.13.x is originally compiled with scala 2.10 and gives support to 2.11.x.

When I execute sbt , all the default jars are automatically downloaded with suffuix _10 (for scala 2.10). When execute assembly json4s jars for 2.11 is also downloaded.

I upgraded the sbt with the hope that this issue might be fixed with new version of sbt.

Sbt 1.02 introduced the same problem for XML related jars as sbt 1.0.2 originally compiled with scala 2.12.

I looked up the sbt version which would by default download 2.11.x jars, but failed to find any such version.

So, now I am looking for a solution by which I could override the jars that are downloaded by default during sbt.


On Sun, Oct 15, 2017 at 11:03 PM, Mark Hamstra <[hidden email]> wrote:
sbt version is 1.0.2.

Why?

Building Spark with sbt is done via build/sbt, which will give you sbt 0.13.11 when building Spark 2.1.0. 

 

On Sun, Oct 15, 2017 at 10:42 PM, patel kumar <[hidden email]> wrote:
Hi,

I am using CDH cluster with Spark 2.1 with Scala Version 2.11.8.
sbt version is 1.0.2.

While doing assembly , I am getting error as 
[error] java.lang.RuntimeException: Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.
modules:scala-parser-combinators

I tried to override the version mismatch using dependencyOverrides and force(), but none of the solution worked.

Please help me to resolve this version Conflict.

Details of the configuration are mentioned below :- 

build.sbt 

***********************************************************************
name := "newtest"
version := "0.0.2"

scalaVersion := "2.11.8" 

sbtPlugin := true

val sparkVersion = "2.1.0"

mainClass in (Compile, run) := Some("com.testpackage.sq.newsparktest")

assemblyJarName in assembly := "newtest.jar"


libraryDependencies ++= Seq(
   "org.apache.spark" % "spark-core_2.11" % "2.1.1" % "provided",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.1" % "provided",
  "com.databricks" % "spark-avro_2.11" % "3.2.0",
    "org.apache.spark" % "spark-hive_2.11" % "2.1.1" % "provided"
   )

 
libraryDependencies +=
     "log4j" % "log4j" % "1.2.15" excludeAll(
       ExclusionRule(organization = "com.sun.jdmk"),
       ExclusionRule(organization = "com.sun.jmx"),
       ExclusionRule(organization = "javax.jms")
     )
     
resolvers += "SparkPackages" at "https://dl.bintray.com/spark-packages/maven/"
resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)

assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
}      
   
***************************************************
plugins.sbt

**********************************

dependencyOverrides += ("org.scala-lang.modules" % "scala-xml_2.11" % "1.0.4")
dependencyOverrides += ("org.scala-lang.modules" % "scala-parser-combinators_2.11" % "1.0.4")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
resolvers += Resolver.url("bintray-sbt-plugins", url("https://dl.bintray.com/eed3si9n/sbt-plugins/"))(Resolver.ivyStylePatterns)

*****************************************************
Error Message after assembly

**************************************************
[error] Modules were resolved with conflicting cross-version suffixes in {file:/D:/Tools/scala_ide/test_workspace/test/NewSp
arkTest/}newsparktest:
[error]    org.scala-lang.modules:scala-xml _2.11, _2.12
[error]    org.scala-lang.modules:scala-parser-combinators _2.11, _2.12
[error] java.lang.RuntimeException: Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.
modules:scala-parser-combinators
[error]         at scala.sys.package$.error(package.scala:27)
[error]         at sbt.librarymanagement.ConflictWarning$.processCrossVersioned(ConflictWarning.scala:39)
[error]         at sbt.librarymanagement.ConflictWarning$.apply(ConflictWarning.scala:19)
[error]         at sbt.Classpaths$.$anonfun$ivyBaseSettings$64(Defaults.scala:1971)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:42)
[error]         at sbt.std.Transform$$anon$4.work(System.scala:64)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:257)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error]         at sbt.Execute.work(Execute.scala:266)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:257)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:167)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:32)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] (*:update) Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml, org.scala-lang.modules:scala-par
ser-combinators
[error] Total time: 413 s, completed Oct 12, 2017 3:28:02 AM

**************************************************