[SPARK-SQL] How to return GenericInternalRow from spark udf

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[SPARK-SQL] How to return GenericInternalRow from spark udf

Amit Joshi
Hi,

I have a spark udf written in scala that takes couuple of columns and apply some logic and output InternalRow. There is spark schema of StructType also present. But when I try to return the InternalRow from UDF there is exception

java.lang.UnsupportedOperationException: Schema for type org.apache.spark.sql.catalyst.GenericInternalRow is not supported

  val getData = (hash : String, type : String) => {
    val schema = hash match {
      case "people" =>
        peopleSchema
      case "empl" =>  emplSchema
    }
    getGenericInternalRow(schema)
  }

  val data = udf(getData)

Spark Version : 2.4.5


Please Help.


Regards

Amit Joshi

Reply | Threaded
Open this post in threaded view
|

Re: [SPARK-SQL] How to return GenericInternalRow from spark udf

srowen
The UDF should return the result value you want, not a whole Row. In
Scala it figures out the schema of the UDF's result from the
signature.

On Thu, Aug 6, 2020 at 7:56 AM Amit Joshi <[hidden email]> wrote:

>
> Hi,
>
> I have a spark udf written in scala that takes couuple of columns and apply some logic and output InternalRow. There is spark schema of StructType also present. But when I try to return the InternalRow from UDF there is exception
>
> java.lang.UnsupportedOperationException: Schema for type org.apache.spark.sql.catalyst.GenericInternalRow is not supported
>
>   val getData = (hash : String, type : String) => {
>     val schema = hash match {
>       case "people" =>
>         peopleSchema
>       case "empl" =>  emplSchema
>     }
>     getGenericInternalRow(schema)
>   }
>
>   val data = udf(getData)
>
> Spark Version : 2.4.5
>
>
> Please Help.
>
>
> Regards
>
> Amit Joshi

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]