No of cores per executor.

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

No of cores per executor.

Amit Sharma
I have set  5 cores per executor. Is there any formula to determine best combination of executor and cores and memory per core for better performance. Also when I am running local spark instance in my web jar getting better speed than running in cluster.



Thanks
Amit
Reply | Threaded
Open this post in threaded view
|

Re: No of cores per executor.

Samik Raychaudhuri
Hi,
Take a look at this video: [https://www.youtube.com/watch?v=daXEp4HmS-E]. Pretty dense, but might answer some of your questions.
Thanks.
-Samik


On 09-Dec-19 4:12 AM, Amit Sharma wrote:
I have set  5 cores per executor. Is there any formula to determine best combination of executor and cores and memory per core for better performance. Also when I am running local spark instance in my web jar getting better speed than running in cluster.



Thanks
Amit