re: yarn resource overcommit: cpu / vcores

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

re: yarn resource overcommit: cpu / vcores

Peter Liu
Hi there,

is there any best practice guideline on yarn resource overcommit with cpu / vcores, such as yarn config options, candidate cases ideal for overcommiting vcores etc.?

this slide below (from 2016) seems to address the memory overcommit topic and hint a "future" topic on cpu overcommit:

any help/hint would be very much appreciated!



I have a system with 80 vcores and a relatively light spark streaming workload. overcomming the vocore resource (here 100) seems to help the average spark batch time. need more understanding on this practice.
Skylake (1 x 900K msg/sec) total batch# (avg) avg batch time in ms (avg) avg user cpu (%) nw read (mb/sec)
70vocres 178.20 8154.69 n/a n/a
80vocres 177.40 7865.44 27.85 222.31
100vcores 177.00 7,209.37 30.02 220.86