alternatives to shading

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

alternatives to shading

jasonnerothin@gmail.com
Our build is complex; it uses a large number of third party jars and generates an uber jar that is shaded before we pass it to spark submit. We shade to avoid ClassLoader collisions with Spark platform dependencies (e.g. protobuf 3).

Managing the dependencies/shade is cumbersome and error prone. The shade plugin itself takes a long time to run. Some jars (e.g. apache commons products) use classpath reflection - therefore the build does not fail until runtime.

We have attempted the spark.{user,executor}.userClasspathFirst settings, but they're marked as experimental and fail sometimes.

We are considering implementing our own ClassLoaders and/or rebuilding and shading the spark distribution.

Are there better alternatives?

--
Thanks,
Jason