In Catalyst expressions, when is it appropriate to use codegen

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

In Catalyst expressions, when is it appropriate to use codegen

Arwin Tio
Hi,

I am exploring the usage of Catalyst expression functions to avoid the performance issues associated with UDFs. 

One thing that I noticed is that there is a trait called CodegenFallback and there are some Catalyst expressions in Spark that inherit from it [0].

My question is, is there a technical limitation for some Catalyst expressions, like datetimeExpressions, that make codegen unsuitable? How do you evaluate whether or not a Catalyst expression should use codegen?

Thanks,

Arwin