Processes

Revision / Modified: Jan. 17, 2001
Author: Tom Berger

Original documents:
http://www.mandrakeuser.org/docs/admin/aproc.html http://www.mandrakeuser.org/docs/admin/aproc2.html

Introducing Processes

Did you ever had some program 'hanging' on you? Indeed 'hanging' so badly that it gobbled up all system resources, slowing down all operations to a crawl? And didn't you wish there were just a way to kill that damn thing? Were you ever interested in knowing how much system resources a program uses?

Well, you have come to right place, then :-). What are processes? Basically everything going on on your system: your web browser is a process, your web server - if you happen to run one - is also a process. A process boots your machine and another shuts it down. Every tiny script starts a process, albeit usually a very short one. "A process" says the textbook, "is an instance of a program in execution."

Processes can be started, stopped, killed, given priorities, scheduled. You do many of these actions already without knowing about it: when you open a program, you start a process, by closing the program, you kill the process. It's not some fancy Linux thing either, every modern operation system has them, because processes allow for multi-tasking: by keeping a unified list of what is going on, the operating system can run a schedule, thus providing its user with the illusion of being able to do many things at once.

Usually the operating system,i.e. the kernel, manages processes for you, you only need to interfere if you have special preferences (e.g. want to have a process to get more resources) or if a process goes awry and starts eating all the available system resources.

Process Monitors

The traditional Unix/Linux tools for viewing and handling processes are the console programs 'top' and 'ps' (part of the 'procps' RPM) as well as a bunch of command line programs like 'kill' or 'nice'.

There are graphical interfaces for these programs, however, which you might find more convenient and easier to use: 'ksysguard' (package 'kdeadmin', menu: Applications - Monitoring - KDE System Guard), 'kpm' (package 'kdeutils', menu: Applications - Monitoring - Process Management) and 'gtop' (package 'gtop', menu: Applications - Monitoring - GTop). 'ksysguard' even allows you to watch and control processes on other machines.

Since all these tools use the same information from the '/proc' directory (more on that later), they all display processes and their attributes in similar fashion.
If you open them, you'll find a dynamic (apart from 'ps') column display with a row for each process. Columns are:

Manipulating processes via the graphical utilities usually just involves 'highlighting' the process you want to handle and using the right mouse button click context menu. 'top' is controlled via shortcut keys, read man top for a list.

Notice that there's a standard key combination for killing graphical applications, <ALT> <CTRL> <ESC>. This will change the mouse pointer to a 'death' symbol. Click on an application window and that application will be killed. You still want to check if the processes of that application had been killed, too, though.

Processes In Detail

The father of all processes is 'init' (initializer). The pstree command shows this quite impressively.
'init' manages the runlevels, it controls which processes are started and stopped at which runlevels and it is responsible for bringing the system to these runlevels. If you shutdown your system, you essentially tell 'init' to bring it to runlevel 0. It's the same if you type shutdown -h now or init 0.

Information on current processes is stored in the '/proc' file system. The '/proc' file system consists of kernel data changed in real-time. It is therefore not a 'real' file system like the others: the data contained take up no place on the hard-disk and you can't write to it. But you can extract and change the information contained therein (e.g. using the 'cat' command).

Listing '/proc', you'll find a lot of directories which names only consist of numbers. These numbers are the 'process IDs' (PIDs). If you list such a directory, you'll find files like 'cmdline', which contains the command line the process was started with, or 'stat', which contains number codes on the current process.
For example:

# cat /proc/1/cmdline
init

Which means: 'The process with the PID 1 was started with the command 'init''. 'init' will always be the first process and thus is the only process with a permanent PID. All other processes will have increasing PIDs, the maximum PID being 32768 (although the maximum number of processes on Linux is only limited by the amount of physical memory available).

The children of 'init' can have child processes themselves. If you start a program by typing its name on a command line, the program will be a child of the program which handles the command line. In most cases, this will be 'bash' (the 'Bourne-Again-SHell').
One important thing to know is that child processes 'inherit' some assets of their fathers. If you run a process with 'root' privileges, a child of this process will also have these privileges.

Now you might ask: 'Since init runs with 'root' privileges, and it is the father of all processes, does that mean that all processes and programs on Linux run with root privileges?'

It's obvious they don't do that. Servers like the font server, web servers, mail servers run on their own user account. The font server for example runs on the 'xfs' account, the Apache web server on the 'apache' user account. Servers which don't come with their own user account use the standard 'nobody' user account. These are not 'real' accounts: nobody can login via these so-called system accounts.
The reason for this is security: servers can be accessed from outside the machine. If one of them contains a software flaw which allows attackers to execute commands on your machine as the user who is running the server, it would be a disaster if that user would be 'root'. Notice that this isn't a system mechanism, but a standard feature of Linux servers. You should refrain at all cost from using servers which do not include it.

Another mechanism is the 'login' command. Have a look at the output of 'pstree' again. You will see that every process and program you started after boot is a child of the 'login' process. 'login' switches the permissions of these child processes depending on how you log into the system. If you log in as 'root', all children will have 'root' privileges, if you log in as a user, the children will run with the permissions (the 'UserID', UID) of this user. The latter is preferable for the system's security and stability. If you need 'root' privileges, you can always log into the 'root' account from your user account by using 'su' or 'sudo'.

setUID

As you've already seen, processes can run with a UserID different from the user who started them. Servers are started by 'root' but drop this privilege as soon as possible and run on their own system account. In Unix-speak, one says they are running 'setuid account' (e.g. Apache runs 'setuid apache').

Of course, you can also do that the other way round by providing processes started by a user root privileges.
The most prominent example is the graphical subsystem, 'X'. 'X' needs 'root' privileges to run. However, the children of X do not run as 'root'. How come?
'X' is started via a so-called wrapper, '/usr/X11/bin/Xwrapper'. If you look at the permission bits of this program, you will see it's 'setuid root': -rws--x--x. This wrapper handles the permissions part when starting X, so that X itself doesn't need to be 'setuid root'. It runs as 'root' but its children don't inherit its UID.

Priority

Processes run either in user mode or in kernel mode. The operating system and its core services run in the privileged kernel mode, whereas 'normal' applications run in user mode, this way badly behaving user processes do not interfere with system operation. Some user processes however need access to privileged kernel functions. For this they switch to kernel mode, execute the needed function and switch back to user mode.

All user processes get the same share on the processor schedule. But you can change that with the nice -n value process command. Negative values increase the priority. They can only be applied by 'root', whereas positive values can be applied by any user.
nice -n value process starts a process with a given priority, the command renice priority PID changes the priority of a running process.

Processes And Threads

A program may open several processes if needed. This however isn't very efficient: each process would need its own address (memory) space and switching between these would cost a considerable amount of system resources. This is where 'threads' come in. Threads are started by a process, but they remain within the original address space used by the process. Switching between threads consumes much less resources than between processes. Applications which use this technique are called 'multi-threaded'.
Programming thread-enabled applications cleanly is very hard which justifies the use of threads only in very complex pieces of software, like the Mozilla browser.

As a system administrator, you only have to know about them that process monitors on Linux can not discriminate a thread from a process. Point in case is the Mozilla browser. If you start it, the process monitors will show multiple entries for it:

tom      11290 13.4  2.3 33080 21124 ?       S    11:47   0:02 /usr/lib/mozilla/
tom      11296  0.0  2.3 33080 21124 ?       S    11:47   0:00 /usr/lib/mozilla/
tom      11297  0.0  2.3 33080 21124 ?       S    11:47   0:00 /usr/lib/mozilla/
tom      11298  0.0  2.3 33080 21124 ?       S    11:47   0:00 /usr/lib/mozilla/
tom      11299  0.7  2.3 33080 21124 ?       S    11:47   0:00 /usr/lib/mozilla/

This does not mean that the Mozilla browser has started five processes, but that's one process with four threads, all in the same address space. How can one tell? Well, you don't get any real proof, since the monitor doesn't know either, but there are hints like the same memory consumption numbers, the same command name and the same start time. Only the first entry is valid when looking at system resource usage.

For more on Linux process management, read the Linux Gazette article Processes on Linux and Windows NT by Glen Flower.

Process Accounting

Process accounting enables you to keep detailed accounting information for the system resources used, their allocation among users, and system monitoring.
(Enabling Process Accounting on Linux HOWTO)

You will need to install the 'psacct' package from the first Mandrake Linux CD which will in turn install a system service of the same name. Documentation (and an amusing bit of Unix lore) can then be found in info accounting.
Both 'Webmin' and 'Linuxconf' offer graphical modules to configure and monitor process accounting.

Estimating Process Resource Usage

Estimating the resource usage, especially the memory consumption of processes is by far more complicated than it looks like at a first glance.
How much resources a process needs, depends on many factors, most of them varying. One is the amount of physical RAM in the system. If there is much free RAM available, more caching will be performed and thus more memory consumed. However this doesn't really count as resource usage, since this cached memory is available in case some other process needs it. Up to 20% of system memory can be used for buffering and caching. This dramatically improves disk accesses, so you don't 'loose' anything when memory is assigned to this task. Only unused memory is wasted ;-).

Graphical applications depend on certain collections of code libraries for displaying their interface (buttons, menus etc). Common collections are GTK+ (GIMP, GNOME), Qt (KDE), Tcl/Tk and others. If you use applications of many different collections, all these libraries have to be loaded into system memory, thus using more resources. An application written with Qt for KDE will use more system resources in GNOME than on KDE. If you start a second Qt application, however, it will use about as much resources as on KDE, since the first application already did load the needed interface libraries. If you close both applications, not all resources will be freed up again, instead the loaded libraries will be cached (if the amount of free RAM permits it).

The numbers for each process presented by process monitors tend to mislead and are easily misinterpreted. On the previous page you've already read about threads. But even if the process doesn't spawn threads, it is in most cases almost impossible to determine how much system memory an application really consumes.

Let me give a simple example:

  1. top: Mem: 900040K av, 411864K used, 488176K free, 1048K shrd, 58592K buff, 166816K cac
  2. Starting the 'bluefish' HTML editor.
  3. top entry for bluefish: 4716 (size) 4716 (real) 3580 (shared).
    top entry for system: Mem: 900040K av, 414268K used, 485772K free, 1432K shrd, 58620K buff, 167592K cac

According to the process table entry, 'bluefish' uses 4716 KB of real memory and 3580 KB of shared memory. The system however only reports an increase of 2404 KB general memory usage, of these being 400 KB more in shared memory, 30 KB more in buffers and 600 KB more in cache.
Regardless of how you add or subtract these numbers, you will never get the numbers from the process entry and the system numbers in sync.

I rather trust the system numbers, but even they can be misinterpreted quite easily:

42 processes: 41 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 0.9% user, 1.3% system, 0.0% nice, 97.6% idle
Mem:   257676K av, 252940K used,   4736K free, 202852K shrd,   7464K buff
Swap:  130748K av,    256K used, 130492K free,               197620K cached

What usually confuses people here is the cache and buffer management of Linux. Let's have a look at it:

Mem:  257676K av, 252940K used,   4736K free, 202852K shrd,   7464K buff
Swap: 130748K av,    256K used, 130492K free                197620K cached

'Mem' shows some 252 MB RAM installed. In fact it's 256 MB. What happened to the remaining 4 MB? Simple: they are used for 'shadowing' the system's BIOS and contain the GNU/Linux kernel.
The next fields seem to indicate that the system is on the brink of collapse: almost all of the memory seems to be used up. Is that so? No, it isn't. Look at the last entries of those lines. These tell you that about 200 MB of system memory are used for caching and buffering. This memory is available for every application which needs it.
The 'free' command line tool gives a more comprehensible overview:

$ free
        total         used     free [...]
Mem:   257676       253624     4052 [...]
-/+ buffers/cache:   50360   207316
Swap:  130748          256   130492

Have a look at the third line:

-/+ buffers/cache: 50360 207316

displaying again the actual amount of memory in use by applications (50 MB) and free (202 MB). The rest is used for caching and buffering, i.e. to make your system faster.

To estimate the resource consumption of a process, it's easiest to run 'free' before, during and after running the process. The figures will vary because of the already mentioned reasons, so you should do this several times to get an average.

Setting Resource Limits

PAM (Pluggable Authentication Modules, man 8 pam) offers a mechanism to set limits to resource usage, setting limits via shell mechanisms ('ulimit' in bash, 'limit' in csh etc) is deprecated.
You configure these limits as 'root' in '/etc/security/limits.conf'. Among other things you can set the maximum number of processes per user or group, process priorities, maximum CPU time and more.
Parameters and examples are supplied in the configuration file.

More information can be found in the appropriate chapter of the L-PAMSAG.

Dealing With Bad Processes

If a process behaves badly, the kernel sends it a certain signal (man 7 signal) to terminate it.
The most infamous signal is SIGNAL 11 (alias SIGSEGV, alias 'segmentation fault'). The process has tried to access a memory segment which hasn't been allocated to it.
Reasons for a segfault might be a programming error, a wrong library version or a hardware failure. The default action of a process when receiving a Signal 11, is to terminate itself and write the contents of the system memory to a 'core' file (in Unix slang, the process 'dumps the core'). This core file can be useful to a programmer for debugging. Notice that Mandrake Linux by default disables the creation of core files.

Things get a bit hairy however when the process is that bad that it ignores this signal and starts to hoard all the available processor time on the system, thus slowing down ('starving out') all the other processes. The kernel marks this process as 'uninterruptible' and its status in the process table will be switched from 'R' or 'S' to 'D'. If you see a process entry with a 'D' in its STATUS row, it's definitely time for you to do something.

Almost all signals can be handled by processes which means that they can also be ignored by them. There is however one signal which can't be ignored by any process and that's SIGNAL 9 (SIGKILL), the appropriately named 'kill signal'. If you send that signal to a process, you will quite literally kill it by taking away all its resources and removing it from the process table. You can then safely go on in your daily work. All unsaved data of that process will be lost, too, though.

Most system monitors offer you a possibility to kill a process via the right mouse click context menu (in console 'top', press the <k> key). There is one problem though: processes run with the permissions of the user who started them. A user can only kill processes he himself has started, only 'root' can kill all processes. Your system monitoring program will and should usually run under your user account which means you can't kill processes from other users or 'root' via it.

If a process started by root or another user misbehaves, you have to switch to the 'root' account on a console using the su command. You can then either kill the process by its process ID (PID) or by its name.
If you don't know the PID of the process yet, you can find out with

# ps aux --sort %cpu

which lists the most CPU hungry process last, the PID being the leftmost number. You can then go on to actually kill the process with the 'kill' command:

# kill -s 9 PID

You can also kill a process by name. Notice that this can be dangerous, since you either might accidentally kill processes you didn't intend to kill (or none at all). The command is

# killall -s 9 name

If you ever happen to come across a non-Linux Unix system, make sure to read the man page for 'killall' on that system. Some Unixes take this command literally which can get very ugly ...

A rather harmless type of a bad process is the so-called 'Zombie' (STATUS: Z, 'defunct'). If you've read the previous page, you know that processes can have 'children', i.e. spawn new processes. If the parent process is terminated normally, it will send a termination signal to all its children signals, too. If however the parent is terminated abnormally, it might not have gotten around to send its children the termination signal. Without their parent process these processes turn into 'living dead': they still appear in the process table, but they don't use any resources and there is no way of 'reaching' these processes. Unlike their namesakes, process zombies don't do any harm. They will vanish from the process table upon the next reboot.


Legal: This text is covered by the GNU Free Documentation License. Standard disclaimers of warranty apply. Copyright LSTB (Tom Berger) and Mandrakesoft 1999-2002.