Using spark and mesos container with host_path volume

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Using spark and mesos container with host_path volume

Antoine DUBOIS
Hello,
I'm trying to mount a local ceph volume to my mesos container.
My cephfs is mounted on all agent at /ceph
I'm using spark 2.4 with hadoop 3.11 and I'm not using Docker to deploy spark.
The only option I could find to mount a volume though is the following (which is also a line I added to spark-defaults.conf in both my client and in the .tgz deployed on mesos):

spark.mesos.executor.docker.volumes /ceph:/mnt:rw

However it seems that the volume is never mounted on mesos containers and using spark-shell on my mesos cluster I'm unable to access any thing in /mnt.
I have activated the volume/host_path on all mesos agent and I don't really know what could I do now.

Is there a way to do what I want with mesos containizer or do I have to use docker (a without docker solution would be so lovely).

Thank you in advance for any hint or help.

Antoine



smime.p7s (2K) Download Attachment