Re: Multiple vcores per container when running Spark applications in Yarn cluster mode

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Multiple vcores per container when running Spark applications in Yarn cluster mode

Saisai Shao
I guess you're using Capacity Scheduler with DefaultResourceCalculator, which doesn't count cpu cores into resource calculation, this "1" you saw is actually meaningless. If you want to also calculate cpu resource, you should choose DominantResourceCalculator.

Thanks
Jerry

On Sat, Sep 9, 2017 at 6:54 AM, Xiaoye Sun <[hidden email]> wrote:
Hi,

I am using Spark 1.6.1 and Yarn 2.7.4.
I want to submit a Spark application to a Yarn cluster. However, I found that the number of vcores assigned to a container/executor is always 1, even if I set spark.executor.cores=2. I also found the number of tasks an executor runs concurrently is 2. So, it seems that Spark knows that an executor/container has two CPU cores but the request is not correctly sent to Yarn resource scheduler. I am using the org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler on Yarn.

I am wondering that is it possible to assign multiple vcores to a container when a Spark job is submitted to a Yarn cluster in yarn-cluster mode.

Thanks!
Best,
Xiaoye 

Reply | Threaded
Open this post in threaded view
|

Re: Multiple vcores per container when running Spark applications in Yarn cluster mode

sunxiaoye0116
Hi Jerry, 

This solves my problem. 🙏 thanks

On Sun, Sep 10, 2017 at 8:19 PM Saisai Shao <[hidden email]> wrote:
I guess you're using Capacity Scheduler with DefaultResourceCalculator, which doesn't count cpu cores into resource calculation, this "1" you saw is actually meaningless. If you want to also calculate cpu resource, you should choose DominantResourceCalculator.

Thanks
Jerry

On Sat, Sep 9, 2017 at 6:54 AM, Xiaoye Sun <[hidden email]> wrote:
Hi,

I am using Spark 1.6.1 and Yarn 2.7.4.
I want to submit a Spark application to a Yarn cluster. However, I found that the number of vcores assigned to a container/executor is always 1, even if I set spark.executor.cores=2. I also found the number of tasks an executor runs concurrently is 2. So, it seems that Spark knows that an executor/container has two CPU cores but the request is not correctly sent to Yarn resource scheduler. I am using the org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler on Yarn.

I am wondering that is it possible to assign multiple vcores to a container when a Spark job is submitted to a Yarn cluster in yarn-cluster mode.

Thanks!
Best,
Xiaoye 

Reply | Threaded
Open this post in threaded view
|

Re: Multiple vcores per container when running Spark applications in Yarn cluster mode

Gourav Sengupta
Saisai,

thanks a ton :)

Regards,
Gourav

On Mon, Sep 11, 2017 at 11:36 PM, Xiaoye Sun <[hidden email]> wrote:
Hi Jerry, 

This solves my problem. 🙏 thanks

On Sun, Sep 10, 2017 at 8:19 PM Saisai Shao <[hidden email]> wrote:
I guess you're using Capacity Scheduler with DefaultResourceCalculator, which doesn't count cpu cores into resource calculation, this "1" you saw is actually meaningless. If you want to also calculate cpu resource, you should choose DominantResourceCalculator.

Thanks
Jerry

On Sat, Sep 9, 2017 at 6:54 AM, Xiaoye Sun <[hidden email]> wrote:
Hi,

I am using Spark 1.6.1 and Yarn 2.7.4.
I want to submit a Spark application to a Yarn cluster. However, I found that the number of vcores assigned to a container/executor is always 1, even if I set spark.executor.cores=2. I also found the number of tasks an executor runs concurrently is 2. So, it seems that Spark knows that an executor/container has two CPU cores but the request is not correctly sent to Yarn resource scheduler. I am using the org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler on Yarn.

I am wondering that is it possible to assign multiple vcores to a container when a Spark job is submitted to a Yarn cluster in yarn-cluster mode.

Thanks!
Best,
Xiaoye