Jobs don't run on all the nodes of the cluster

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Jobs don't run on all the nodes of the cluster

mileszhou
This post has NOT been accepted by the mailing list yet.
My cluster has 4 nodes, but jobs never run on one of the nodes (which is not the master). From the logs, I saw that all my nodes had been resolved. The parameters are as follows:

--number_workers=32
--number_cores=6
--worker_memory=24g

Each node has 48g of physical memory and 8 cores. I configured all my nodes as data nodes (slaves), including the name node.

Any idea?
Reply | Threaded
Open this post in threaded view
|

Re: Jobs don't run on all the nodes of the cluster

dataginjaninja
This post has NOT been accepted by the mailing list yet.
+1 We are having a similar issue. All but one node.
Reply | Threaded
Open this post in threaded view
|

Re: Jobs don't run on all the nodes of the cluster

mileszhou
This post has NOT been accepted by the mailing list yet.
The problem has been resolved; my fault, though: The node that didn't run was running out of disk space.