Spark ignores --master local[*]

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark ignores --master local[*]

raimon.bosch
This post has NOT been accepted by the mailing list yet.
Hi all,

I have a small program in SparkSQL that in my local machine with Mac OS X uses 100% of 4 cpus (when using --master local[*] option). But when I execute the same code with the same option on a virtual machine based on Debian Jessie, it only uses 1 cpu. Looks like it is unable to assign more than 1 cpu, no matter what --master option you send.

I'm using spark-2.1.0-bin-hadoop2.7 with java 8 on both cases.

Is there any known issue about that?

Thanks in advance,
Reply | Threaded
Open this post in threaded view
|

Re: Spark ignores --master local[*]

Vikash Pareek
Your VM might not be having more than 1 core available to run spark job.
Check with *nproc* command to see how many cores available on VM and *top
*command to see how many cores are free.



-----

__Vikash Pareek
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


__Vikash Pareek