Migration issue upgrading from Spark 2.4.5 to spark 3.0.1

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Migration issue upgrading from Spark 2.4.5 to spark 3.0.1

Nathan Kronenfeld-2
Hi, everyone.

We are in the process of upgrading our Spark version, and are hitting what appears to be the bug described in https://issues.apache.org/jira/browse/SPARK-29497 - namely, that when we pass anonymous functions to the workers, we get an error 
Caused by: java.lang.ClassCastException: cannot assign instance 
of java.lang.invoke.SerializedLambda to field <XYZ>
of type scala.Function1 in instance of <xyz>
As we are using Gradle instead of Maven, the suggested fixes there, even as far as they go, don't work for us.

Has anyone else seen this? Does anyone have any suggestions on how to get around it, other than to stop passing around anonymous functions?

Thanks,

            - Nathan Kronenfeld
            - Uncharted Software