Reply: Any best practice for hardware configuration for the masterserver in standalone cluster mode?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Reply: Any best practice for hardware configuration for the masterserver in standalone cluster mode?

XiaoboGu
Thanks for your reply, I am new hand at Spark, does driver mean the server where user applications are commit?



------------------ Original ------------------
Sender: "Mayur Rustagi"<[hidden email]>;
Send time: Tuesday, Dec 31, 2013 9:55 PM
To: "user"<[hidden email]>;
Subject: Re: Any best practice for hardware configuration for the masterserver in standalone cluster mode?

Master server needs to be a little beefy as the driver runs on it. We ran into some issues around scaling due to master servers. You can offload the drivers to workers or other machines then the master server can be smaller. 
Regards
Mayur



On Tue, Dec 31, 2013 at 6:48 PM, guxiaobo1982 <[hidden email]> wrote:
Him

I read the following article regarding to hardware configurations for the worker servers in the standalone cluster mode, but what about the master server?



Regards,

Xiaobo Gu


Reply | Threaded
Open this post in threaded view
|

Re: Reply: Any best practice for hardware configuration for the masterserver in standalone cluster mode?

Mayur Rustagi
Driver is the process that manages the execution across the cluster. So say your application is a "sql query" then the system spawns a shark-cli-driver that uses spark framework, hdfs etc to execute the query and deliver result. All this happens automatically so you dont need to worry about it as a user of spark/shark framework. Just go for a bigger machine with a master.




On Tue, Dec 31, 2013 at 7:01 PM, guxiaobo1982 <[hidden email]> wrote:
Thanks for your reply, I am new hand at Spark, does driver mean the server where user applications are commit?



------------------ Original ------------------
Sender: "Mayur Rustagi"<[hidden email]>;
Send time: Tuesday, Dec 31, 2013 9:55 PM
To: "user"<[hidden email]>;
Subject: Re: Any best practice for hardware configuration for the masterserver in standalone cluster mode?

Master server needs to be a little beefy as the driver runs on it. We ran into some issues around scaling due to master servers. You can offload the drivers to workers or other machines then the master server can be smaller. 
Regards
Mayur

Mayur Rustagi
Ph: <a href="tel:%2B919632149971" value="+919632149971" target="_blank">+919632149971


On Tue, Dec 31, 2013 at 6:48 PM, guxiaobo1982 <[hidden email]> wrote:
Him

I read the following article regarding to hardware configurations for the worker servers in the standalone cluster mode, but what about the master server?



Regards,

Xiaobo Gu