If there is timestamp type data in DF, Spark 2.3 toPandas is much slower than spark 2.2.

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

If there is timestamp type data in DF, Spark 2.3 toPandas is much slower than spark 2.2.

李斌松
If there is timestamp type data in DF, Spark 2.3 toPandas is much slower than spark 2.2.
Reply | Threaded
Open this post in threaded view
|

Re: If there is timestamp type data in DF, Spark 2.3 toPandas is much slower than spark 2.2.

Irving Duran

I haven't noticed or seen this behavior.  Have you noticed this with by testing the same dataset between versions?

Thank you,

Irving Duran
On 06/06/2018 11:22 PM, 李斌松 wrote:
If there is timestamp type data in DF, Spark 2.3 toPandas is much slower than spark 2.2.


signature.asc (484 bytes) Download Attachment