Dynamic allocation not releasing executors after unpersisting all cached data

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Dynamic allocation not releasing executors after unpersisting all cached data

Jeffrey Charles
I'm persisting a dataframe in Zeppelin which has dynamic allocation enabled to get a sense of how much memory the dataframe takes up. After I note the size, I unpersist the dataframe. For some reason, Yarn is not releasing the executors that were added to Zeppelin. If I don't run the persist and unpersist steps, the executors that were added are removed about a minute after the paragraphs complete. Looking at the storage tab in the Spark UI for the Zeppelin job, I don't see anything cached.

Is there any way to get Yarn to automatically remove executors after doing a persist followed by an unpersist if there is no activity on the executor within the configured dynamic allocation timeout (similar to how it works without a persist/unpersist cycle) without having to set spark.dynamicAllocation.cachedExecutorIdleTimeout? The main reason I'd like to avoid setting that configuration is I do not want to the executors being reclaimed if they do have cached data.
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic allocation not releasing executors after unpersisting all cached data

Vadim Semenov-2
Try doing `unpersist(blocking=true)`
On Mon, Jul 9, 2018 at 2:59 PM Jeffrey Charles
<[hidden email]> wrote:
>
> I'm persisting a dataframe in Zeppelin which has dynamic allocation enabled to get a sense of how much memory the dataframe takes up. After I note the size, I unpersist the dataframe. For some reason, Yarn is not releasing the executors that were added to Zeppelin. If I don't run the persist and unpersist steps, the executors that were added are removed about a minute after the paragraphs complete. Looking at the storage tab in the Spark UI for the Zeppelin job, I don't see anything cached.
>
> Is there any way to get Yarn to automatically remove executors after doing a persist followed by an unpersist if there is no activity on the executor within the configured dynamic allocation timeout (similar to how it works without a persist/unpersist cycle) without having to set spark.dynamicAllocation.cachedExecutorIdleTimeout? The main reason I'd like to avoid setting that configuration is I do not want to the executors being reclaimed if they do have cached data.



--
Sent from my iPhone

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Dynamic allocation not releasing executors after unpersisting all cached data

Jeffrey Charles
Thanks for the suggestion. I gave it a try but the executor still isn't being released several minutes after running that.

On Mon, Jul 9, 2018 at 3:51 PM Vadim Semenov <[hidden email]> wrote:
Try doing `unpersist(blocking=true)`
On Mon, Jul 9, 2018 at 2:59 PM Jeffrey Charles
<[hidden email]> wrote:
>
> I'm persisting a dataframe in Zeppelin which has dynamic allocation enabled to get a sense of how much memory the dataframe takes up. After I note the size, I unpersist the dataframe. For some reason, Yarn is not releasing the executors that were added to Zeppelin. If I don't run the persist and unpersist steps, the executors that were added are removed about a minute after the paragraphs complete. Looking at the storage tab in the Spark UI for the Zeppelin job, I don't see anything cached.
>
> Is there any way to get Yarn to automatically remove executors after doing a persist followed by an unpersist if there is no activity on the executor within the configured dynamic allocation timeout (similar to how it works without a persist/unpersist cycle) without having to set spark.dynamicAllocation.cachedExecutorIdleTimeout? The main reason I'd like to avoid setting that configuration is I do not want to the executors being reclaimed if they do have cached data.



--
Sent from my iPhone