How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

shyla deshpande
I get the ERROR
1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers

Is there a way to clean up these directories while the spark streaming application is running?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

shyla deshpande
Is there a way to do this without stopping the streaming application in yarn cluster mode?

On Mon, Dec 17, 2018 at 4:42 PM shyla deshpande <[hidden email]> wrote:
I get the ERROR
1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers

Is there a way to clean up these directories while the spark streaming application is running?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

shyla deshpande
Please point me to any documentation if available. Thanks

On Tue, Dec 18, 2018 at 11:10 AM shyla deshpande <[hidden email]> wrote:
Is there a way to do this without stopping the streaming application in yarn cluster mode?

On Mon, Dec 17, 2018 at 4:42 PM shyla deshpande <[hidden email]> wrote:
I get the ERROR
1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers

Is there a way to clean up these directories while the spark streaming application is running?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

Fawze Abujaber

On Wed, Dec 26, 2018 at 1:05 AM shyla deshpande <[hidden email]> wrote:
Please point me to any documentation if available. Thanks

On Tue, Dec 18, 2018 at 11:10 AM shyla deshpande <[hidden email]> wrote:
Is there a way to do this without stopping the streaming application in yarn cluster mode?

On Mon, Dec 17, 2018 at 4:42 PM shyla deshpande <[hidden email]> wrote:
I get the ERROR
1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers

Is there a way to clean up these directories while the spark streaming application is running?

Thanks


--
Take Care
Fawze Abujaber
Reply | Threaded
Open this post in threaded view
|

Re: How to clean up logs-dirs and local-dirs of running spark streaming in yarn cluster mode

shyla deshpande
Hi Fawze,
Thank you for the link. But that is exactly what I am doing.
I think this is related to yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage setting. 
When the disk utilization exceeds this setting, the node is marked unhealthy. 

Other than increasing the default 90%, is there anything else I can do?

Thanks
-Shyla



On Tue, Dec 25, 2018 at 7:26 PM Fawze Abujaber <[hidden email]> wrote:

On Wed, Dec 26, 2018 at 1:05 AM shyla deshpande <[hidden email]> wrote:
Please point me to any documentation if available. Thanks

On Tue, Dec 18, 2018 at 11:10 AM shyla deshpande <[hidden email]> wrote:
Is there a way to do this without stopping the streaming application in yarn cluster mode?

On Mon, Dec 17, 2018 at 4:42 PM shyla deshpande <[hidden email]> wrote:
I get the ERROR
1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers

Is there a way to clean up these directories while the spark streaming application is running?

Thanks


--
Take Care
Fawze Abujaber