Linux: how to monitor the nofile limit

In a previous post I explained how to measure the number of processes that are generated when a fork() or clone() call checks the nproc limit. There is another limit in /etc/limits.conf – or in /etc/limits.d – that is displayed by ‘ulimit -n’. It’s the number of open files – ‘nofile’ – and here again we need to know what kind of files are counted.

nofile

‘nofile’ is another limit that may not be easy to monitor, because if you just count the ‘lsof’ output you will include a lot of lines which are not file descriptors. So how can we count the number of files descriptors in a process?

lsof

‘lsof’ is a utility that show all the open files. Let’s take an example:

I get the pid of my pmon process:

And I list the open files for that process

I’ve removed hundreds of lines with FD=mem and size=4M. I’m in AMM with memory_target=800M and SGA is implemented in /dev/shm granules. With lsof, we see all of them. And with a large memory_target we can have thousands of them (even if granule becomes 16M when memory_target is larger than 1GB). But don’t worry, they don’t count in the ‘nofile’ limit. Only ‘real’ file descriptors are counted – those with a numeric FD.

So, if you want to know the processes that are near the limit, you can use the following:

The idea is to filter the output of lsof and use awk to keep only the numeric file descriptors, and aggregate per process. Then, we sort them and show the highest counts. Here the Postfix master process has 89 files open. Then log writer follows.

You can get the same information from /proc filesystem where files handles are in /proc//fd:

for p in /proc/[0-9]* ; do echo $(ls $p/fd | wc -l) $(cat $p/cmdline) ; done | sort -n | tail

15 ora_dmon_DEMO
16 ora_dbw0_DEMO
16 ora_mmon_DEMO
16 ora_rsm0_DEMO
16 /app/oracle/product/12.1/bin/tnslsnrLISTENER-inherit
17 automount--pid-file/var/run/autofs.pid
17 dbus-daemon--system
20 rpc.mountd
21 ora_lgwr_DEMO
89 /usr/libexec/postfix/master

Same result, much quicker and more information about the process. This is the way I prefer, but remember that if you want to see all processes, you should be logged as root.

The proof

As I did for nproc, I have written a small C program that open files (passed as arguments) for a few seconds, so that I’m sure I’m counting the right things.

And I encourage to do the same on a test system and let me know if your result differs. Here is the source: openfiles.zip

First, I set my nofile limit to only 10

ulimit -n 10

Then, let’s open 7 files. In addition with stdin, stdout and stderr we will have 10 file handles:

I was able to open those 7 files. Then I check lsof:

We see our 10 file handles and this proves that only numeric FD are counted when checking the nofile limit of 10. You see stdin, stdout, stderr as FD 0,1,2 and then my 7 files opened in read only.

Let’s try to open one more file:

Here the limit is reached and the open() call returns error 24 (ENFILE) because we reached the nofile=10.

Threads

When counting the processes for the nproc limit, we have seen that threads must be counted as processes. For the nofile limit we don’t need to detail the threads because all threads share the file descriptor table.

Recommended values

Currently this is what is set on Oracle linux 6 for 11gR2 (in /etc/security/limits.conf):

For 12c, these are set in /etc/security/limits.d/oracle-rdbms-server-12cR1-preinstall.conf which overrides /etc/security/limits.conf:

Do you think it’s a bit low? Just for information, here is what is set in the ODA X4-2:

oracle soft nofile 131072

In any case, it is a good idea to check if you are reaching the limit and the above scripts on lsof or /proc should help for that.