UDF error with Spark 2.4 on scala 2.12

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

UDF error with Spark 2.4 on scala 2.12

Andres Ivaldi
Hello I'm having problems with UDF, I was reading a bit about it , and it look's like a closure issue, but I don't know hoy to fix it, it works fine on 2.11.

my code for udf definition (I tried several posibilites this is the las one)

    val o:org.apache.spark.sql.api.java.UDF2[java.sql.Timestamp,String, Int] =  (date:java.sql.Timestamp, periodExp:String) => {
      val o2:org.apache.spark.sql.api.java.UDF2[java.sql.Date,String, Int] =  (date:java.sql.Date, periodExp:String) => {
   sc.udf.register ("timePeriod",o2, DataTypes.IntegerType )
   sc.udf.register("timePeriod", o , DataTypes.IntegerType )

Caused by: java.io.NotSerializableException: scala.runtime.LazyRef
[info] Serialization stack:
[info]  - object not serializable (class: scala.runtime.LazyRef, value: LazyRef thunk)
[info]  - element of array (index: 3)
[info]  - array (class [Ljava.lang.Object;, size 5)
[info]  - field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;)
[info]  - object (class java.lang.invoke.SerializedLambda, SerializedLambda[capturingClass=class org.apache.spark.sql.catalyst.expressions.ScalaUDF, functionalInterfaceMethod=scala/Function1.apply:(Ljava/lang/Object;)Ljava/lang/Object;, implementation=invokeStatic org/apache/spark/sql/catalyst/expressions/ScalaUDF.$anonfun$f$3:(Lscala/Function2;Lorg/apache/spark/sql/catalyst/expressions/Expression;Lorg/apache/spark/sql/catalyst/expressions/Expression;Lscala/runtime/LazyRef;Lscala/runtime/LazyRef;Lorg/apache/spark/sql/catalyst/InternalRow;)Ljava/lang/Object;, instantiatedMethodType=(Lorg/apache/spark/sql/catalyst/InternalRow;)Ljava/lang/Object;, numCaptured=5])

Ing. Ivaldi Andres