UDTF registration fails for hiveEnabled SQLContext

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

UDTF registration fails for hiveEnabled SQLContext

Mick Davies
Hi,

If I try to register a UDTF using SQLContext ( with enableHiveSupport set)
using the code:

I get the following error:


It works OK if I use deprecated HiveContext.


Is there a way to register UDTF without using deprecated code?

This is happening in some tests I am writing using

but I don't think it is related to that.

Thanks for any help.

Mick






--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: UDTF registration fails for hiveEnabled SQLContext

Mick Davies
The examples were lost by formatting:

Exception is:

No handler for UDAF 'com.iqvia.rwas.omop.udtf.ParallelExplode'. Use
sparkSession.udf.register(...) instead.; line 1 pos 7
org.apache.spark.sql.AnalysisException: No handler for UDAF
'com.iqvia.rwas.omop.udtf.ParallelExplode'. Use
sparkSession.udf.register(...) instead.; line 1 pos 7
        at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.makeFunctionExpression(SessionCatalog.scala:1134)
        at
org.apache.spark.sql.catalyst.catalog.SessionCatalog$$anonfun$org$apache$spark$sql$catalyst$catalog$SessionCatalog$$makeFunctionBuilder$1.apply(SessionCatalog.scala:1114)
        at
org.apache.spark.sql.catalyst.catalog.SessionCatalog$$anonfun$org$apache$spark$sql$catalyst$catalog$SessionCatalog$$makeFunctionBuilder$1.apply(SessionCatalog.scala:1114)
        at
org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.lookupFunction(FunctionRegistry.scala:115)
        at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1245)






--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: UDTF registration fails for hiveEnabled SQLContext

Gourav Sengupta
Hi Mick,

the error practically gives you the suggestion to correct it "use sparkSession.udf.register(...) instead".

Regards,
Gourav Sengupta

On Mon, May 14, 2018 at 8:59 AM, Mick Davies <[hidden email]> wrote:
The examples were lost by formatting:

Exception is:

No handler for UDAF 'com.iqvia.rwas.omop.udtf.ParallelExplode'. Use
sparkSession.udf.register(...) instead.; line 1 pos 7
org.apache.spark.sql.AnalysisException: No handler for UDAF
'com.iqvia.rwas.omop.udtf.ParallelExplode'. Use
sparkSession.udf.register(...) instead.; line 1 pos 7
        at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.makeFunctionExpression(SessionCatalog.scala:1134)
        at
org.apache.spark.sql.catalyst.catalog.SessionCatalog$$anonfun$org$apache$spark$sql$catalyst$catalog$SessionCatalog$$makeFunctionBuilder$1.apply(SessionCatalog.scala:1114)
        at
org.apache.spark.sql.catalyst.catalog.SessionCatalog$$anonfun$org$apache$spark$sql$catalyst$catalog$SessionCatalog$$makeFunctionBuilder$1.apply(SessionCatalog.scala:1114)
        at
org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.lookupFunction(FunctionRegistry.scala:115)
        at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1245)






--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: UDTF registration fails for hiveEnabled SQLContext

Mick Davies
Hi Gourav,

I don't think you can register UDTFs via sparkSession.udf.register

Mick



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: UDTF registration fails for hiveEnabled SQLContext

Ajay-2
You can register udf's by using the in-built udf function as well using (import org.apache.spark.sql.functions._)
Something along the lines of 
val flattenUdf = udf(udfutils.flatten)
where udfutils is another object and flatten is a method in it. 

On Tue, May 15, 2018 at 3:27 AM Mick Davies <[hidden email]> wrote:
Hi Gourav,

I don't think you can register UDTFs via sparkSession.udf.register

Mick



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Thanks,
Ajay
Reply | Threaded
Open this post in threaded view
|

Re: UDTF registration fails for hiveEnabled SQLContext

Mick Davies

I am trying to register a UDTF not a UDF.

So I don't think this applies

Mick



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]