[PySpark Profiler]: Does empty profile mean no execution in Python Interpreter?
I ran into an interesting scenario with no profile output
today. I have a PySpark application that primarily uses the
Spark SQL APIs. I understand that parts of the Spark SQL API may
not generate data in the PySpark profile dumps, but I was
surprised when I had code containing a UDF that did not generate
any profile output. I had thought anytime I used a UDF with
Spark SQL that code would have to execute in a Python
interpreter on the executor. Is that not the case? This went
against my mental model for how this works in Spark, so I'm
trying to understand what is happening here to cause no profile
output, which made me wonder if the UDF had ran in the JVM.