In an executor, are the Python worker memory and the MemoryOverhead overlapping?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

In an executor, are the Python worker memory and the MemoryOverhead overlapping?

o_rayer
This post has NOT been accepted by the mailing list yet.
Hi there,

I'm using PySpark on a Hadoop cluster and I could not find the info about the executor memory model with Python.

I know that the Python memory (spark.python.worker.memory) does not overlap the JVM Heap (spark.executor.memory).

However, does the Python Memory overlap the Executor Memory Overhead or not ?

Thank you very much,
Loading...