#17: Process substitution

Posted by | Comments (2) | Trackbacks (2)

You probably know command substitution. Command substitution executes a command and uses its output as a variable on the shell. The operator for this is either the backtick operator

`command`

or the (newer and recommended) parenthesis operator with a dollar sign:

$(command)

Command substitution is very common and I use it often for things like rebuilding my input drivers after xorg updates:

emerge -av1 `eix -I --only-names x11-drivers/`

or alternatively

emerge -av1 $(eix -I --only-names x11-drivers/)

There are hundreds of thousands of different use cases for this. But did you also know that there is a counterpart of command substitution? This counterpart is called process substitution. Unlike command substitution this does not use the output of a command as a variable, but writes it to a FIFO in /etc/fd/ or receives its STDIN from there (I wrote about FIFOs earlier this month). Instead of using the output directly, only the filename of this FIFO/named pipe is used. The operators for process substitution are

<(command)

and

>(command)

The first one writes its STDOUT to the FIFO and can be used as input file for another program. The second one receives its STDIN from the FIFO. Thus, when you write to the file, the command in parentheses receives the written data as STDIN. Note that there is no space after < and >!

You can see how process substitution works by running

echo <(true)
/dev/fd/63

The STDOUT of <(true) is written to the FIFO /etc/fd/63 and then the command itself has been substituted with the filename. The same happens to the second operator:

echo >(true)
/dev/fd/63

But how do we use this? Process substitution is very helpful if you have a command which only accepts files and does not work with STDIN/STDOUT. For example, if you want to compare the output of two commands with diff, you could write their output into files and then compare them with diff, but there is a much easier way to do this:

diff <(echo 'foo') <(echo 'bar')

The output is

1c1
< foo
---
> bar

Simple but handy. You could also use this with grep. Instead of piping STDOUT you could also make grep reading from a FIFO:

grep foobar <(ps auxw)

Since grep can operate on STDIN as well as on files this leads to the same result as

ps auxw | grep foobar

Of course the piping variant is much simpler, but this is just an example to demonstrate how process substitution works.

As described above, the second process substitution operator does not provide its STDOUT via FIFO but receives its STDIN from it. For instance, a complex and preposterous way of printing “foobar” to the screen would be:

echo 'foobar' > >(cat)

This redirects the output of the echo command to a file, which is our FIFO. Then the command in parentheses receives the written data as STDIN. A more meaningful example would be to pipe STDOUT of a command to several programs at once. For instance:

echo 'foobar' | tee >(command1) >(command2) | command3

In this example tee would write to the FIFOs for command1 and command2, which get the data as STDIN. Then all the STDOUT of tee, including STDOUT of command1 and command2, is passed to command3 since it's piped to tee and all its arguments. If that's not what you want you have to use process substitution for command3 as well.

Another example could be to compress files generated by a program, but this does not work in all cases. For instance, I had some trouble with mencoder, perhaps because it checks the size of the output file. If a program verifies what it has written or does some calculation based on the output file, writing to FIFOs might not work.

There are not so many cases in which the >(command) notation is expedient, but <(command) is very handy in several cases. But keep in mind that process substitution is very Bash specific. Only a few other shells have implemented it as well (such as ZSH).

Read more about process substitution:

Trackbacks

Manko10 sent a Trackback on : (permalink)

RT @reflinux: #Advent series “24 Short #Linux #Hints”, day 17: #Process substitution http://bit.ly/eC8dzE

robo47 sent a Trackback on : (permalink)

RT @reflinux: #Advent series “24 Short #Linux #Hints”, day 17: #Process substitution http://bit.ly/eC8dzE

Comments

There have been 2 comments submitted yet. Add one as well!
Basti
Basti wrote on : (permalink)

Nice article and a indeed very nice method to diff! keep on writing – learned a lot smile

best regards

Basti

george
george wrote on : (permalink)

Another fun little variant:

exec 5< <(command)

This runs (command) in the background, creates a FIFO, and binds that FIFO to file descriptor 5 in the shell. Then you can read data out at your leisure…

Unfortunately this (as far as I can tell) doesn’t get you to the point of having a nice implementation of coprocesses in bash… There’s no way to get both an input and output pipe to a process with this method unless you resort to something like a named pipe.

The whole /dev/fd thing is kind of funky in general – filesystem entries corresponding to file descriptors that are already open in the current process…

Write a comment:

HTML-Tags will be converted to Entities.
Textile-formatting allowed
Standard emoticons like :-) and ;-) are converted to images.
Design and Code Copyright © 2010-2018 Janek Bevendorff Content on this site is published under the terms of the GNU Free Documentation License (GFDL). You may redistribute content only in compliance with these terms. tweetbackcheck