How do you set POSIX rlimit on mesos

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

How do you set POSIX rlimit on mesos

We are running spark on mesos, and encounter too many open files error.

I have already set the hard limit of nofile at /etc/security/limits.conf
`* hard nofile 65536`
and increase the soft nofile limit in the, but not work
`ulimit -n 65536`

stderr when executor start: ` line 6: ulimit: open files: cannot modify limit: Operation not permitted`

mesos doc says that posix rlimit specified as part of a taks's ContainerInfo, so increase rlimit with ulimit command in the not work?

I think that start mesos agent with a higher nofile limit will solve the problem, but maybe it should set with spark properties?

camper42 (Fengyu Cao)
Douban, Inc.

Mobile: +86 15691996359
E-mail:  [hidden email]