#7: Job control and background processes

Posted by | Comments (7) | Trackbacks (4)

Linux is a multitasking operating system, so you can execute several programs at once. It is also possible to make use of this on the console. Normally when you run a program, you have to wait until the program has finished to get your shell prompt back. Therefore, during the execution of a program the shell is “locked”. But it isn't unresponsive as you can see when you press CTRL+C, which sends a SIGINT to the running process. Hence, the shell is not really “locked” but rather “occupied” and you can perform operations while the program is being executed.

Besides Ctrl+C, there is another keyboard shortcut you should definitely know about and that is Ctrl+Z. Ctrl+Z sends a SIGTSTP to the currently running process. Unlike SIGSTOP, this signal can be handled by the process and does not cause the process to stop necessarily, but in most cases, when you press Ctrl+Z, you'll see something like that:

$ ^Z
[1]+  Stopped                 foobar

This indicates that the process foobar has just been stopped and is not executed anymore. But it is not terminated. Stopped processes are still alive and marked with a T in the STAT column of ps. You can continue a stopped process at any time by entering %N, where N is the number in square brackets from above (here it is 1, so %1 would continue the process). This number is called the job ID, which is assigned to each stopped process by the shell's job control.

You may use this job ID to continue the process, but you don't have to do this in the foreground. You may also continue executing it in the background, so you can use the shell while the process is being executed concurrently. To send a stopped process to the background, run

bg N

Again, N is the job ID (if you don't pass any, the job ID of the last stopped process will be used). Background processes keep their job ID so you can still interact with them even though they don't occupy the shell (note: you still may see STDOUT of the backgrounded process but if you hit enter you see that your shell prompt is still there). To see all processes being stopped or running in the background, use jobs. E.g.

$ jobs
[1]+  Running                 dd if=/dev/zero of=/dev/null &

This shows you that there is a process running in the background, continuously copying zeros from /dev/zero to /dev/null (this is supposed to run forever and heat up your CPU, so don't run this command longer than needed ;-)). You might also have noticed the ampersand & at the end of the command name. This is a shell operator to send processes into background. Thus, instead of running a command, tapping Ctrl+Z and then invoking bg you could simply use the ampersand operator:

dd if=/dev/zero of=/dev/null &

This process would directly go into background. You can check that with jobs.

It is also possible to bring backgrounded processes back to foreground. The command for this is fg. As well as bg, the command fg takes the job number as argument. If no argument is passed, the last backgrounded or stopped process will be brought to foreground. In doing so, stopped processes will continue execution. fg is completely equivalent to %. Both bring stopped or backgrounded processes back to foreground, and both may be used with or without an explicit job ID.

fg and bg are the shell's default commands for job control but, as mentioned before, they just use signals. Job control is not really a Linux thing. The kernel itself only provides multitasking and process groups and handles signals, but managing job IDs and foreground/background processes is the shell's business (therefore, job IDs are not available across multiple shell instances). The signal for stopping processes is SIGTSTP or the uncatchable SIGSTOP and the signal for continuing them (in background) is SIGCONT. Let's try this. Run this command (or any other endless running or long lasting command):

$ sleep 900 &
[1] 3456

This will sleep for the next 15 minutes peacefully in the background. The output of this command is the job ID in brackets and the PID of this background process. Now control the execution state of this command with jobs:

$ jobs
[1]+  Running                 sleep 900 &

If jobs outputs something like that you've done all right. To stop this process now, either bring it back to foreground and hit Ctrl+Z or send a SIGTSTP manually:

$ kill -TSTP 3456

[1]+  Stopped                 sleep 900

To continue the process, we send a SIGCONT but this time we don't pass the PID but the job ID to kill:

kill -CONT %1

Control with jobs:

[1]+  Running                 sleep 900 &

Great, process is running again!

Jobs, i.e., via job control stopped and backgrounded processes, depend on the opened shell instance since they are handled in process groups being children of the current shell process. When the shell is terminated by sending it a SIGHUP, this signal will be resent to all subprocesses. Login shells also send a SIGHUP to their children when leaving with the exit or logout command if huponexit is set. Because of this, jobs probably don't survive closing the shell. But there is one command to avoid this behavior: nohup. nohup (which stands for no hangup) let's you make sure that a process does not terminate when closing the shell window. To run our sleep process in the background with nohup, type

nohup sleep 900 &

You can now close your terminal without terminating our process. All output of our program will be written to a file nohup.out within the current working directory (but since our program is not snoring there's no output here). What nohup does, is actually no magic. It just changes the signal disposition attribute of this process for SIGHUPs to Ign, so hangup signals are ignored.

One thing which is good to know is that when a job has finished, its job information will still be kept in memory until you run jobs or close the terminal/tty. If there is a finished job, jobs would output something like that:

$ jobs
[1]+  Done                    sleep 1

That's basically it, but I have one great warning in the end: be careful with background processes and sudo! If you use the ampersand operator to send a process into background, you'll see nothing more than nothing and jobs shows you why:

$ jobs
[1]+  Stopped                 sudo foobar

Your process is stopped because sudo is waiting for your password. You have to bring this process back to foreground for entering it. However, the more crucial issue is that due to the shell-dependency terminating “sudoed” background processes with SIGINT, SIGTERM or even SIGKILL by addressing their job ID is not possible. You would get an error

bash: kill: (4567) - Operation not permitted

This is because the sudo process is owned by root; thus, you have to kill this process with root privileges and that closes the circle: you can't address job IDs from within temporary sudo environments. You need a real shell environment for that but sudo -s or su would open a completely new shell instance, so you can't address the parent shell's job IDs either. Hence, you have to kill those processes as root by directly addressing their PIDs. To examine the corresponding PIDs, either use ps, jobs -l or jobs -p to only show PIDs. The result of jobs -p can directly be passed to kill and would therefore kill all jobs of the current shell instance. For instance:

$ sudo kill `jobs -p`
[2]+  Exit 143                sudo foo
[1]+  Exit 143                sudo bar

Note: if sudo is still waiting for your password at this time, you have to bring it to the foreground first or you must send a harsh SIGKILL (infamous signal No. 9). SIGINT or SIGTERM would not terminate sudo in that case because those signals can be ignored.

You see, using sudo in combination with your shell's job control functions is sometimes very delicate. One quick tip here: sudo itself provides a parameter -b, which sends the process into background. This is probably the better choice here since it makes things clearer and sudo also doesn't wait forever unnoticed in the background if it waits four your password. Just keep that in mind!

Read more about job control:


robo47 sent a Trackback on : (permalink)

RT @reflinux: #Advent series “24 Short #Linux #Hints”, day 7: #Job control and background #processes http://bit.ly/fTu8Df

robo47 sent a Trackback on : (permalink)

RT @reflinux: Corrected Advent calendar article #14: Ctrl+Z sends a SIGTSTP , not an uncatchable SIGSTOP . http://bit.ly/fTu8Df


There have been 7 comments submitted yet. Add one as well!
George wrote on : (permalink)

Nitpick time!

CTRL-Z doesn’t issue “SIGSTOP”, but rather “SIGTSTP” – a “STOP” issued from the TTY. The difference is you can set your own handler for TSTP (handy if you want to play nice with the terminal in a curses program, or just don’t want people suspending your program)

I’m kind of confused by the whole HUP thing with background jobs. Apparently when I run a background job and then kill the terminal window, the job keeps running. I ran a program of mine that monitors the signals it receives and it appears to never get a HUP. I only get HUP if it’s a foreground job in bash, or if I use “xterm -e signaltest”… And I can’t remember this ever being an issue with background jobs. I run it in the background, log out, it keeps running… Is there some bash shopt that forces it to pass on SIGHUP?

Janek Bevendorff
Janek Bevendorff wrote on : (permalink)

Hi George, thanks for your comment!

I was pretty sure that Ctrl+Z sends a SIGSTOP, but it seems that you are right. I’ll do some research here and correct in then. Thank you very much!

Concerning the SIGHUP thing: SIGHUP is not a kill signal. If you log out, all processes started from your shell receive a SIGHUP, but it’s up to themselves, what they do with it. Some exit, some reload their configs, some ignore it. If your program ignores SIGHUP, you can leave out nohup, but then the output would not be logged to a file, so you had to do that by yourself with output redirection.

I’ve edited the SIGHUP section of the article a bit to make clear that SIGHUP does not directly kill all processes.

EDIT #2:
Corrected SIGSTOP to SIGTSTP and added a short description of the difference between these two signals.
Again, thank you for that!

George wrote on : (permalink)

That’s the thing, though: most of the programs I tried just don’t do anything special with SIGHUP. My signaltest program catches SIGHUP and writes out a message – when I ran the program in the background and killed the TTY, the signaltest apparently didn’t get a SIGHUP (the program’s stderr was redirected to a file so I could see the “signal received” messages…)
I don’t think “sleep” does anything special with SIGHUP – and I wrote and tested another program that writes text to stdout periodically – neither program terminated when I ran them in the background and then killed the TTY… Since the program doesn’t do anything special with signals, SIGHUP should have killed it. Based on that and the signaltest results it seems background jobs don’t get HUP’ed when the tty dies… Unless I’ve got something wrong, or unless it’s something that varies according to the shell…
Of course, that’s just what happened with bash and dash. zsh seems to have killed the background jobs when I exited (first ctrl-d gave a warning, second ended the shell and killed the background jobs) via SIGHUP. ksh and mksh both kept the background jobs alive…
So again, I don’t know if it’s a result of some bit of shell configuration done by the distro or what – but so far only one of the shells I’ve tried passes SIGHUP on to the background jobs on exit.

Janek Bevendorff
Janek Bevendorff wrote on : (permalink)

Then huponexit is disabled in bash, which seems to be the default behavior in current bash releases. I wonder if this has always been the default behavior.

To set huponexit run

shopt -s huponexit

for your current shell instance. Background processes should also get a SIGCONT first to make sure they notice the signal.
I think I should add some more information to this part. nohup seems to be quite obsolete in many cases.

Janek Bevendorff
Janek Bevendorff wrote on : (permalink)

Ah, got it!
If you send a SIGHUP to bash, it will resend it to all processes and then exit. If you just kill bash (e.g. close your terminal window or send SIGKILL), the processes will stay untouched.
With huponexit set, login (!) shells will also send SIGHUP to processes when exiting the shell with logout or exit, respectively.
See also here
Edited that in the article.

George wrote on : (permalink)

Ah, OK, that makes sense. I thought killing an xterm would’ve generated SIGHUP – but I guess since xterm exec’s its child process itself, it really has no reason to rely on the TTY’s hangup mechanism to kill the child process…

Killing a window in screen will generate a HUP, so that’s apparently a good way of testing these cases. (Of course you could also just kill -HUP the shell process – but I wanted to test these things in a way that issued the signal via the controlling TTY… I’m pretty sure screen does that…)

Write a comment:

HTML-Tags will be converted to Entities.
Textile-formatting allowed
Standard emoticons like :-) and ;-) are converted to images.
Design and Code Copyright © 2010-2017 Janek Bevendorff Content on this site is published under the terms of the GNU Free Documentation License (GFDL). You may redistribute content only in compliance with these terms. tweetbackcheck