is it forgotten to document how to set SPARK_WORKER_DIR?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

is it forgotten to document how to set SPARK_WORKER_DIR?

Nan Zhu
Hi, all

I’m trying to change my worker dir to a mounted disk with larger space

but I found that no document telling me how to do this, 

I have to check the source code and found the following lines

 if (System.getenv("SPARK_WORKER_DIR") != null) {
    workDir = System.getenv("SPARK_WORKER_DIR")
  }

is it forgotten to be documented?

Best,

-- 
Nan Zhu
Reply | Threaded
Open this post in threaded view
|

Re: is it forgotten to document how to set SPARK_WORKER_DIR?

Archit Thakur
What do you mean by worker dir?


On Tue, Jan 7, 2014 at 11:43 AM, Nan Zhu <[hidden email]> wrote:
Hi, all

I’m trying to change my worker dir to a mounted disk with larger space

but I found that no document telling me how to do this, 

I have to check the source code and found the following lines

 if (System.getenv("SPARK_WORKER_DIR") != null) {
    workDir = System.getenv("SPARK_WORKER_DIR")
  }

is it forgotten to be documented?

Best,

-- 
Nan Zhu

Reply | Threaded
Open this post in threaded view
|

Re: is it forgotten to document how to set SPARK_WORKER_DIR?

Nan Zhu
In worker machines, you will find $SPARK_HOME/work/

that’s worker dir

-- 
Nan Zhu

On Tuesday, January 7, 2014 at 7:29 AM, Archit Thakur wrote:

What do you mean by worker dir?


On Tue, Jan 7, 2014 at 11:43 AM, Nan Zhu <[hidden email]> wrote:
Hi, all

I’m trying to change my worker dir to a mounted disk with larger space

but I found that no document telling me how to do this, 

I have to check the source code and found the following lines

 if (System.getenv("SPARK_WORKER_DIR") != null) {
    workDir = System.getenv("SPARK_WORKER_DIR")
  }

is it forgotten to be documented?

Best,

-- 
Nan Zhu