Problem with Spark on Mesos

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Problem with Spark on Mesos

juanpedromoreno
Hi, I'm using vagrant. I've built a cluster with MESOS and SPARK with the following structure:

- 2 zookeeper nodes
- 2 master nodes  
- 3 slave nodes

In each master node, I've installed MESOS and Spark.
My $SPARK_HOME/conf/spark_env.sh contains:

export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
export SPARK_EXECUTOR_URI=http://dominio.com/spark/spark-0.9.0-bin-cdh4.tgz
export MASTER=zk://192.168.30.101:2181,192.168.30.102:2181

The issue I'm having is that executors, launched by the slaves nodes, always download the spark distribution from http://dominio.com/spark/spark-0.9.0-bin-cdh4.tgz.
Disk usage of slave nodes is continuously increasing, in addition to the introduced latency which is due to downloading and decompressing the tgz file. Is there any way to avoid it?

Thanks in advance.