Problem with Spark on Mesos

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Problem with Spark on Mesos

Hi, I'm using vagrant. I've built a cluster with MESOS and SPARK with the following structure:

- 2 zookeeper nodes
- 2 master nodes  
- 3 slave nodes

In each master node, I've installed MESOS and Spark.
My $SPARK_HOME/conf/ contains:

export MESOS_NATIVE_LIBRARY=/usr/local/lib/
export MASTER=zk://,

The issue I'm having is that executors, launched by the slaves nodes, always download the spark distribution from
Disk usage of slave nodes is continuously increasing, in addition to the introduced latency which is due to downloading and decompressing the tgz file. Is there any way to avoid it?

Thanks in advance.