Dataset Caching and Unpersisting

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Dataset Caching and Unpersisting

Daniele Foroni
Hi all,

I am having troubles with caching and unpersisting a dataset.
I have a cycle that at each iteration filters my dataset.
I realized that caching every x steps (e.g., 50 steps) gives good performance.

However, after a certain number of caching operations, it seems that the memory used for caching is filled, so I think I should have to unpersist the old cached dataset.

This is my code:

I tried to use an external variable to cache and unpersist it but it doesn’t seem to solve the problem (maybe I used it in the wrong way).
Do you kindly have any suggestion?

Thank you for your support!