Increase executor memory for spark shell

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Increase executor memory for spark shell

KK R
Hi,
Each of the workers are allocated 1GB of memory. When i invoke the spark shell, only 512MB of the memory is getting used. How can i increase the memory of the executor for spark shell?

Thanks,
Kiruba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Increase executor memory for spark shell

Rakesh Nair
Kiruba

You can specify the executor memory while starting up the spark shell. 

Something like this:
MASTER=spark://<spark_hostname>:7077 SPARK_MEM=1g ./spark-shell 

This will set executor memory to 1GB. 


On Fri, Dec 27, 2013 at 12:59 AM, KK R <[hidden email]> wrote:
Hi,
Each of the workers are allocated 1GB of memory. When i invoke the spark shell, only 512MB of the memory is getting used. How can i increase the memory of the executor for spark shell?

Thanks,
Kiruba



--
Regards
Rakesh Nair
Loading...