how to change temp directory when spark write data ?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

how to change temp directory when spark write data ?

JF Chen
I have two spark apps writing data to one directory. I notice they share one temp directory, and the spark fist finish writing will clear the temp directory and the slower one may throw "No lease on *** File does not exist" error
So how to specify the temp directory? 
Thanks!

Regard,
Junfeng Chen
Reply | Threaded
Open this post in threaded view
|

Re: how to change temp directory when spark write data ?

Sandip Mehta

tryspark.local.dir property.



On Wed, Dec 5, 2018 at 1:42 PM JF Chen <[hidden email]> wrote:
I have two spark apps writing data to one directory. I notice they share one temp directory, and the spark fist finish writing will clear the temp directory and the slower one may throw "No lease on *** File does not exist" error
So how to specify the temp directory? 
Thanks!

Regard,
Junfeng Chen
Reply | Threaded
Open this post in threaded view
|

Re: how to change temp directory when spark write data ?

JF Chen
Directory to use for "scratch" space in Spark, including map output files and RDDs that get stored on disk. 
It seems a directory on my local disk. But currently the temp directory is on hdfs, under the path of  df.write.parquet(path) 

Regard,
Junfeng Chen


On Wed, Dec 5, 2018 at 6:28 PM Sandip Mehta <[hidden email]> wrote:

tryspark.local.dir property.



On Wed, Dec 5, 2018 at 1:42 PM JF Chen <[hidden email]> wrote:
I have two spark apps writing data to one directory. I notice they share one temp directory, and the spark fist finish writing will clear the temp directory and the slower one may throw "No lease on *** File does not exist" error
So how to specify the temp directory? 
Thanks!

Regard,
Junfeng Chen