Clarify "please raise ulimit -n" message
My logfiles often contained messages like:
- Error creating network socket: Too many open files in system
This happens when socket layer calls return ENFILE or EMFILE. Raising kern.maxfiles and kern.maxfilesperproc makes these messages go away, as expected. However, the following message kept happening:
- Failing because we have 11062 connections already. Please raise your ulimit -n. [8422 similar message(s) suppressed in last 21600 seconds]
I spent a long time looking into file descriptor limits because the value (11062) was suspiciously close to the maximum number of file descriptors per process (11095), which I just raised earlier. Then I discovered #define ERRNO_IS_ACCEPT_RESOURCE_LIMIT(e) in src/common/compat.h and that I had to look for ENOMEM and ENOBUFS in addition to ENFILE and EMFILE.
Reading through the socket code in the kernel, I found that FreeBSD has a default maximum accept() backlog of 128 connections. When more than that number of TCPs are in the syncache, accept() will fail with one of ENOMEM or ENOBUFS. I haven't spent the time to figure out which (it's a maze of twisty passages between kern/uipc_socket and netinet/tcp_usrreq.c -- neither of those are in my active brain cache).
The actual "solution" to the problem (allow tor to accept more connections, and thus make the message go away) was to raise kern.ipc.somaxconn.
The instruction to raise ulimit -n is definitely wrong for FreeBSD. Or at least only part of the story, and certainly cause for confusion. Perhaps the message should point to some generic documentation suggesting system limits/knobs to twiddle, rather than assuming that all the world is Linux and ulimit -n will work in every case.
Trac:
Username: philip