Shuffle Files

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Shuffle Files

Usman Ghani
Where on the filesystem does spark write the shuffle files?
Reply | Threaded
Open this post in threaded view
|

Re: Shuffle Files

Aniket Mokashi
From BlockManager code + ShuffleMapTask code, it writes under spark.local.dir or java.io.tmpdir.

val diskBlockManager = new DiskBlockManager(shuffleBlockManager,
    conf.get("spark.local.dir", System.getProperty("java.io.tmpdir")))




On Mon, Mar 3, 2014 at 10:45 PM, Usman Ghani <[hidden email]> wrote:
Where on the filesystem does spark write the shuffle files?



--
"...:::Aniket:::... Quetzalco@tl"