Fwd: Array[Double] two time slower then DenseVector

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Fwd: Array[Double] two time slower then DenseVector

David Ignjić

Hello all,
I am currently looking in 1 spark application to squeze little performance and here this code (attached in email)

I looked in difference and in:
org.apache.spark.sql.catalyst.CatalystTypeConverters.ArrayConverter
if its primitive we still use boxing and unboxing version because in code
org.apache.spark.sql.catalyst.util.ArrayData#toArray
we don't use method :  ArrayData .toDoubleArray as its used in VectorUDT.

Now is the question do i need to provide patch or someone can me show it how to get same performance with array as with dense vector.
Or i need to create jira ticket


Thanks



---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

spark.scala (2K) Download Attachment