Clearing usercache on EMR [pyspark]

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Clearing usercache on EMR [pyspark]

Shuporno Choudhury
Hi everyone,
I am running spark jobs on EMR (using pyspark). I noticed that after running jobs, the size of the usercache (basically the filecache folder) keeps on increasing (with directory names as 1,2,3,4,5,...).
    Directory location: /mnt/yarn/usercache/hadoop/filecache/
Is there a way to avoid creating these directories or automatically clearing the usercache/filecache after a job/periodically?
--
--Thanks,
Shuporno Choudhury
Reply | Threaded
Open this post in threaded view
|

Re: Clearing usercache on EMR [pyspark]

Shuporno Choudhury
Can anyone please help me with this issue?

On Fri, 3 Aug 2018 at 11:27, Shuporno Choudhury <[hidden email]> wrote:
Can anyone please help me with this issue?

On Wed, 1 Aug 2018 at 12:50, Shuporno Choudhury [via Apache Spark User List] <[hidden email]> wrote:
Hi everyone,
I am running spark jobs on EMR (using pyspark). I noticed that after running jobs, the size of the usercache (basically the filecache folder) keeps on increasing (with directory names as 1,2,3,4,5,...).
    Directory location: /mnt/yarn/usercache/hadoop/filecache/
Is there a way to avoid creating these directories or automatically clearing the usercache/filecache after a job/periodically?
--
--Thanks,
Shuporno Choudhury



If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Clearing-usercache-on-EMR-pyspark-tp33096.html
To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML


--
--Thanks,
Shuporno Choudhury


--
--Thanks,
Shuporno Choudhury