Spark Cluster over yarn cluster monitoring

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Cluster over yarn cluster monitoring

Chetan Khatri
Hi Users,

I do submit X number of jobs with Airflow to Yarn as a part of workflow for Y customer. I could potentially run workflow for customer Z but I need to check that how much resources are available over the cluster so jobs for next customer should start.

Could you please tell what is the best way to handle this. Currently, I am just checking availableMB > 100 then trigger next Airflow DAG over Yarn.

GET http://rm-http-address:port/ws/v1/cluster/metrics
Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Spark Cluster over yarn cluster monitoring

Chetan Khatri
Could someone please help me to understand better..

On Thu, Oct 17, 2019 at 7:41 PM Chetan Khatri <[hidden email]> wrote:
Hi Users,

I do submit X number of jobs with Airflow to Yarn as a part of workflow for Y customer. I could potentially run workflow for customer Z but I need to check that how much resources are available over the cluster so jobs for next customer should start.

Could you please tell what is the best way to handle this. Currently, I am just checking availableMB > 100 then trigger next Airflow DAG over Yarn.

GET http://rm-http-address:port/ws/v1/cluster/metrics
Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Spark Cluster over yarn cluster monitoring

Jörn Franke
Use yarn queues:


Am 27.10.2019 um 06:41 schrieb Chetan Khatri <[hidden email]>:


Could someone please help me to understand better..

On Thu, Oct 17, 2019 at 7:41 PM Chetan Khatri <[hidden email]> wrote:
Hi Users,

I do submit X number of jobs with Airflow to Yarn as a part of workflow for Y customer. I could potentially run workflow for customer Z but I need to check that how much resources are available over the cluster so jobs for next customer should start.

Could you please tell what is the best way to handle this. Currently, I am just checking availableMB > 100 then trigger next Airflow DAG over Yarn.

GET http://rm-http-address:port/ws/v1/cluster/metrics
Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Spark Cluster over yarn cluster monitoring

Chetan Khatri
Thanks Jörn

On Sun, Oct 27, 2019 at 8:01 AM Jörn Franke <[hidden email]> wrote:
Use yarn queues:


Am 27.10.2019 um 06:41 schrieb Chetan Khatri <[hidden email]>:


Could someone please help me to understand better..

On Thu, Oct 17, 2019 at 7:41 PM Chetan Khatri <[hidden email]> wrote:
Hi Users,

I do submit X number of jobs with Airflow to Yarn as a part of workflow for Y customer. I could potentially run workflow for customer Z but I need to check that how much resources are available over the cluster so jobs for next customer should start.

Could you please tell what is the best way to handle this. Currently, I am just checking availableMB > 100 then trigger next Airflow DAG over Yarn.

GET http://rm-http-address:port/ws/v1/cluster/metrics
Thanks.