Just another Scott Alan Miller Sites site

At the best of times, working with ulimits (limits) in Linux can be confusing and complicated.  Sometimes it can be seriously challenging to determine if a limit is being imposed by the system, a user, a script or inheritance from a calling process.  One of the trickier problems to identify is that of a “fork bomb protection” limit being exceeded.  As long as you know to look for it, it is pretty easy, but if you don’t know to look for this overriding limit it can be pretty confusing to identify why processes are failing.

Typically you will encounter this limit when a user receives repeated “Resource temporarily unavailable” errors such as these below:

$ ps aux
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

Or attempting to run a process might prompt “cannot fork [Resource temporarily unavailable]“  Our first guess is likely to inspect the /etc/security/limits.conf file and make sure that the user in question is given enough “open files” and possibly “max user processes.”  In many cases, simply raising one or the other of these limits may solve the problem.  If you collect the output of ulimit from the user session in question you can determine what the limits are that are directly affecting the process calls.

What you may find is that no matter what you do, the “max user processes” or “-u” option in ulimit never goes above 1,024.  Is that strange because you raised this limit in the limits.conf file?  It is indeed.  Before we decide that that is the issue, let’s see how many processes the user in question is using.  Remember that Linux does not have a threading model but only has processes and lightweight processes (LWPs) that behave much like threads in other operating systems.  The “max user processes” value uses the count of all processes and LWPs combined.  So to see how many processes the user is using we can check:

# ps -eLF | grep ^baduser | wc -l
1024

If that number returned, where “baduser” is the username of the offending user account, is at or near 1,024 we can be pretty certain that something has gone wrong and the user has been forking processes rapidly bringing it to the system protection limit.  This is an extremely high number, even for very busy servers.  It’s not an impossible number, just an abnormally high one.

Red Hat Enterprise Linux, and by extension its derivatives like CentOS, have an extra layer of limits applied specifically to the number of processes that an individual user can call to protect against malicious or accidental fork bombs.  This additional limit file is located at /etc/security/limits.d/90-nproc.conf.  The purpose of this file is to override any attempt to raise the “max user processes” limit above 1,024 in the normal limits.conf file to protect users from themselves.

Should you need to raise system-wide user limits, you will need to raise the cap in the 90-nproc.conf file and then additionally raise (or limit) them for individual users in the normal limits.conf file.  That Red Hat makes such a point of making this hard to raise accidentally should trigger red flags that you probably do not really want to raise this limit.  But if you need to do so, that is where the additional limit is kept.

In Linux it is very easy to configure the use of virtual memory for the kernel.  This behaviour is known as “swappiness” as is controlled through a controllable kernel parameter, vm.swappiness.  It is easiest to think of this setting as a slider that goes from zero to one hundred.  The lower the number, the less the kernel will attempt to swap, the high the value the more aggressively the operating system will attempt to utilize swap space.

On most systems, such as Red Hat Enterprise Linux 6 and Ubuntu 12.04, the default setting for swappiness is “60″.  For traditional, physical servers this is a very good starting value and will work well for most purposes.  This value is generally balanced with a blend of throughput versus latency with the assumption that the physical storage of the server is relatively lightly used.  For systems such as desktops or batch processing servers moving this value higher might be logical to increase latency delays while also increasing overall system throughput. On a latency sensitive system or a system where disk IO is at a premium, lowering this number is generally advisable.

On a virtualized system, a good starting point is a value of “10″.  This is because, with rare exception, virtualized systems have shared disk IO and it is important for the overall system to reduce disk utilization when possible.

To determine the swappiness setting of your running system simply run this command:

cat /proc/sys/vm/swappiness

To modify the system setting for swappiness, add or modify this line in /etc/sysctl.conf:

vm.swappiness = 10

This change will require that sysctl reload or that the system is restarted.  This change is persistent.  If you want to test a temporary change that will disappear upon reboot, try this comand instead to alter the /proc filesystem directly:

echo 10 > /proc/sys/vm/swappiness

This will change system behaviour immediately.  Remember to profile your system before and after a change such as this so that you can determine if a change is beneficial or not for your situation.

When a kernel panic is received in Linux we can either do nothing or we can reboot.  Typically in a production environment we would want to reboot and hopefully return the system to a working state as quickly as possible.  In a non-production environment where we are more interested in troubleshooting than in bringing services back online as quickly as we can we are more likely to want to not reboot so that we can investigate from the console while the box is still panicked.

The parameter for this behavior is stored in /proc/sys/kernel/panic.  The settings are simple.  A value of zero for “never reboot, just wait for intervention” and any positive integer indicating the number of seconds to wait after detecting a kernel panic before rebooting.

Let’s view our current setting:

# cat /proc/sys/kernel/panic
0

So in this case we are going to just “sit there” should the kernel panic.  Now we will modify this value so that the system will automatically restart after thirty seconds once the kernel has had a panic.

# echo "30" > /proc/sys/kernel/panic

# cat /proc/sys/kernel/panic
30

This is perfect for testing but the change is not permanent and will revert to default when the system restarts.  To make the change permanent we need to add an entry to the /etc/sysctl.conf file.

# echo "kernel.panic = \"30\""  >> /etc/sysctl.conf