Shuffle uses Direct Memory Buffer even after setting "spark.shuffle.io.preferDirectBufs = false"

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Shuffle uses Direct Memory Buffer even after setting "spark.shuffle.io.preferDirectBufs = false"

Vaibhav Kulkarni

Hi,


I am using Standalone Spark 2.3 and have a question regarding Shuffle. Going by the documentation, default Shuffle behaviour is to use Direct Memory buffers. But, even after I set the following parameter, I notice Shuffle still uses Direct Memory buffers.


spark.shuffle.io.preferDirectBufs = false


Is this a bug ? How can I disable use of Direct Memory for Shuffle


Thanks,

Vaibhav Kulkarni