How do you set POSIX rlimit on mesos

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How do you set POSIX rlimit on mesos

camper42
We are running spark on mesos, and encounter too many open files error.

I have already set the hard limit of nofile at /etc/security/limits.conf
`* hard nofile 65536`
and increase the soft nofile limit in the spark-env.sh, but not work
`ulimit -n 65536`

stderr when executor start: `spark-env.sh: line 6: ulimit: open files: cannot modify limit: Operation not permitted`

mesos doc says that posix rlimit specified as part of a taks's ContainerInfo, so increase rlimit with ulimit command in the spark-env.sh not work?

I think that start mesos agent with a higher nofile limit will solve the problem, but maybe it should set with spark properties?

--
camper42 (Fengyu Cao)
Douban, Inc.

Mobile: +86 15691996359
E-mail:  [hidden email]