Spark executor memory & relationship with worker threads

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark executor memory & relationship with worker threads

Sai Prasanna

Hi All,
Does number of worker threads bear any relationship to setting executor memory ?.
I have a 16 GB RAM, with an 8-core processor. I had set SPARK_MEM to 12g and was running locally with default 1 thread.
So this means there can be maximum one executor in one node scheduled at any point of time. If I increase the number of worker threads to say 4, do I need to reduce SPARK_MEM to 3g or I need not ???

Is there any performance difference between running in interactive spark shell to non-interactive standalone spark apart from building time(sbt…) ??? 

THANKS !!