We are trying to upgrade our application from spark 2.3.1 to 2.4.3, and came across a weird problem.
We are using Gradle for dependency management.
Spark depends on twitter-chill, which depends on kryo-shaded. All our dependencies to kryo-shaded come from twitter, and all request version 4.0.2
Gradle, however, in its infinite wisdom, pulls version 3.0.3 instead, and that won't even compile.
dependencyInsight returns the following:
com.esotericsoftware:kryo-shaded:3.0.3 (selected by rule) variant "runtime" [ org.gradle.status = release (not requested) Requested attributes not found in the selected variant: org.gradle.usage = java-api ]
I presume this means that Gradle can't find the property/value org.gradle.usage=java-api in kryo-shaded version 4.0.2, but it can in 3.0.3?
Does anyone know why this might occur? I see no reference to org.gradle.usage in either our or spark's build files, so (assuming I even understand the problem correctly) I have no idea where this requirement is coming from.
We can work around the problem by setting the kryo-shaded version explicitly, but of course this means we would have to keep setting it as we upgrade in the future, so of course this is not ideal.
I realize this is likely (though not certainly) a gradle, not a spark, problem, but I'm hoping someone else here has encountered this before?