Spark on kubernetes memory spike and spark.kubernetes.memoryOverheadFactor not working

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark on kubernetes memory spike and spark.kubernetes.memoryOverheadFactor not working

Maiti, Mousam

Hi Team,

We are using spark on Kubernetes, through spark-on-k8s-operator. Our application deals with multiple updateStateByKey operations. Upon investigation, we found that the spark application consumes a higher volume of memory. As spark-on-k8s-operator doesn’t give the option to segregate spark JVM memory and pod memory, pod memory usage reaches 100 %. Container OS kernel kills the executor because of OOM.

So, we are trying to limit the memory overhead using –

spark.kubernetes.memoryOverheadFactor

0.1

This sets the Memory Overhead Factor that will allocate memory to non-JVM memory, which includes off-heap memory allocations, non-JVM tasks, and various systems processes. For JVM-based jobs this value will default to 0.10 and 0.40 for non-JVM jobs. This is done as non-JVM tasks need more non-JVM heap space and such tasks commonly fail with "Memory Overhead Exceeded" errors. This prempts this error with a higher default.

 

But to no avail. Is there something we are missing? Is there any way around for this problem.

Thanks,

Mousam