how to best imitate a real cluster with small number of nodes

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

how to best imitate a real cluster with small number of nodes

dachuan
Hi, community,

I have limited number of machines, and with different size of RAM and different cores number. Is there any advice for how to imitate a real cluster with these small number of nodes?

Is the following way ok?

1st machine: 8 cores, 16G memory, 5 spark workers, one core per worker, 3GB mm per worker
2nd machine: 2 cores, 4G memory, 1 spark worker, one core per worker, 3GB mm per worker
3rd machine: 2 cores, 4G memory, 1 spark worker, one core per worker, 3GB mm per worker
4th machine: 4 cores, 16G memory. 3 spark worker, one core per worker, 3GB mm per worker

And one Executor per Worker (since one Worker only has one core). 

If you have any other convenient way to get a cluster with at least 20 large memory nodes, please let me know, and by the way, I have zero experience about Amazon EC2, so I am a little scared about the potential monetary cost, please correct me if I am wrong.

thanks,
dachuan.