Spark version performance

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Spark version performance

Andres Ivaldi
Hello list, 

I'm having a little performance issue, with different Spark versions.

I've a spark embedded application written in scala, Initially I've use Spark 2.0.2, and works fine, with good speed response, but when I updated to 2.3.2 , with no any code changes It becomes slower.

Mainly what the application do is to gather information from a source, apply transformations with filters and performs aggregation over it. It's source is mainly parquet and no write is done just a serialization from the result.

Maybe I'm using  deprecated api functions or the order of the operations are not generating a good plan...

Can someone give me some idea of any change on the versions that could generate this behavior?


Ing. Ivaldi Andres