Please help: change $SPARK_HOME/work directory for spark applications

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Please help: change $SPARK_HOME/work directory for spark applications

Chen Jin
Hi,

Currently my application jars and logs are stored in $SPARK_HOME/work,
I would like to change it to somewhere having more space. Could anyone
advise me on this? Changing the log dir is straightforward which just
to export SPARK_LOG_DIR, however, there is no environment variable for
SPARK_WORK_DIR.

Thanks a lot,

-chen
Reply | Threaded
Open this post in threaded view
|

Re: Please help: change $SPARK_HOME/work directory for spark applications

Nan Zhu
Hi, Jin 

It’s SPARK_WORKER_DIR

Line 48 WorkerArguments.scala

if (System.getenv("SPARK_WORKER_DIR") != null) {
    workDir = System.getenv("SPARK_WORKER_DIR")
  }

Best,

-- 
Nan Zhu

On Wednesday, January 15, 2014 at 2:03 PM, Chen Jin wrote:

Hi,

Currently my application jars and logs are stored in $SPARK_HOME/work,
I would like to change it to somewhere having more space. Could anyone
advise me on this? Changing the log dir is straightforward which just
to export SPARK_LOG_DIR, however, there is no environment variable for
SPARK_WORK_DIR.

Thanks a lot,

-chen