State of spark docker script

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

State of spark docker script

Aureliano Buendia
Hi,


Is the spark docker script now mature enough to substitute spark-ec2 script? Anyone here using the docker script is production?
Reply | Threaded
Open this post in threaded view
|

Re: State of spark docker script

Aaron Davidson
Whoa, wait, the docker scripts are only used for testing purposes right now. They have not been designed with the intention of replacing the spark-ec2 scripts. For instance, there isn't an ssh server running so you can stop and restart the cluster (like sbin/stop-all.sh). Also, we currently mount SPARK_HOME into the containers such that recompiling spark in one location will allow all containers on that machine to run against the updated jars (which is great for testing but maybe not so much for production).

It is possible to get these scripts up to a state where they are generally usable for cluster deployment, but that would take some probably nontrivial effort and as far as I know no one is actively working towards that goal. Also, I'd take a look at the amplab-docker spark scripts, which are more fleshed out: https://github.com/amplab/docker-scripts


On Sun, Mar 9, 2014 at 9:33 AM, Aureliano Buendia <[hidden email]> wrote:
Hi,


Is the spark docker script now mature enough to substitute spark-ec2 script? Anyone here using the docker script is production?

Reply | Threaded
Open this post in threaded view
|

Re: State of spark docker script

Xuefeng Wu
In reply to this post by Aureliano Buendia
Hi Aureliaono,

First, docker is not ready for production, unless you know what are doing and prepared for some risk.

Then, in my opinion , there are so many hard code in spark docker script, you have to modify it for your goal.


Yours, Xuefeng Wu 吴雪峰 敬上

> On 2014年3月10日, at 上午12:33, Aureliano Buendia <[hidden email]> wrote:
>
> Hi,
>
>
> Is the spark docker script now mature enough to substitute spark-ec2 script? Anyone here using the docker script is production?