How to tune groupBy operations in Spark 2.x?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
SRK
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to tune groupBy operations in Spark 2.x?

SRK
This post has NOT been accepted by the mailing list yet.
Hi,

How to tune the Spark Jobs that use groupBy operations? Earlier I used to use  --conf spark.shuffle.memoryFraction=0.8 --conf  spark.storage.memoryFraction=0.1  to tune my jobs that use groupBy. But, with Spark 2.x this configs seem to have been deprecated.

What would be the appropriate config options to tune the Spark Jobs that use groupBy operations?

Thanks,
Swetha
Loading...