Linux/BASH - Redirection concepts for scripting

This article has the intention to take the reader a step further in their BASH programming abilities explaining concepts that the author thinks are the foundation for developing complex scripts. The concepts presented here are all about redirection. 

The topics covered by this article are the following, redirecting STDIN, STDOUT, and STDERR. The differences between Here strings (<<<) and Here Documents (<<). Piping with anonymous pipes and named pipes. 

Within each topic, I'll provide a little example in which we can execute the previously explained concepts and commands for better understanding. 

Let's put the work on it. 

1.- Redirecting Command Input/Output

Inside our favorite Unix-like terminal, every time we type a command are bound to it two default outputs plus one default input, the three ones are called file descriptors. The input: stdin (the keyboard), the first output: stdout (the screen), and the second output: stderr (error messages output to the screen). 

Each time we open a terminal belongs a Process Id (PID) to it, when we run a new command, it's associated with a new child PID forked from the main one. Using linux the set of file descriptors open in a process can be accessed under the path /proc/$PID/fd/, where $PID is the process identifier.

In the image below, we recall the Main PID with the variable "$$", it gives us a value of 311. This PID is associated with our main process. After we generate a script called: process_id.sh, every time it's called, it generates a new forked PID, and three file descriptors belonging to such PID. For instance, the first forked PID is 465 and has three /proc/ directories. The first one: /proc/465/fd/0, refers to the default input, called standard input. The second one: /proc/465/fd/1, refers to the first default output, called standard output. The Third one: /proc/465/fd/2, refers to the second default output, called: standard error.

Notice that the three file descriptors are linked to the route: /dev/pts/1; this is due to such route refers to our pseudo-terminal (the same name to refer to our current terminal). Furthermore, and most importantly, the forked (child) process inherits their path from the father (main) process. In other words, if no new path for any standard file descriptor is explicitly declared with any redirection operator, the child process inherits the father's standard file descriptors.

Each three file descriptors (one default input, plus two default output) have their own corresponding number, ranging from 0 to 2. Also, they have a short name, usually referred to it by the Linux community instead of their full name. The table below shows the three file descriptors description.

The fourth column, called "Default Path," means from which specific path the standard file descriptors are taking their corresponding values. By default, the STDIN file descriptor is receiving values from our current terminal, which is a synonym for: /dev/pts/1. Therefore, both STDOUT and STDERR are also sending their output to our current terminal. 

1.1.- Redirecting the Standard Input (0)

I want to begin this redirecting file descriptors section talking about the Linux concept of FILE. Under Linux, everything is treated as a FILE, which means that everything in the system from processes, files, directories, sockets, pipes, links are represented by a file descriptor, having their owner and permissions. As Linus Torvalds said, referring to Linux, "Everything is a stream of bytes." 

When we see the man pages for two of our most used commands: $ls and $cat, one of their optional parameters is a FILE. Moreover, the DESCRIPTION section for the cat command says that $cat, concatenate FILE(s) to the Standard Output (this is going to have sense in the next section).

That being said, now remember, when we call a function/command, such a new command generates a subshell with a new process ID. Such a new subshell inherits the three standard file descriptors, and by default, each one has the same path, which is our current terminal (/dev/pts/1). 

Employing the $grep command, we can read in their man page that "If no FILEs are specified, or if the FILE "-" is given, grep searches standard input."

Therefore, when we call the $grep "fox" < $PWD/text.txt command, $grep can see that there aren't any FILEs explicitly declared, so it takes STDIN as input, which is already being redirected with the stdin redirection operator: "<". Then, $grep forks a new subshell with a new PID. Next, it takes the redirected $PWD/text.txt FILE and reads it line by line, searching through all the text: "The quick brown fox jumps over the lazy dog" inside of it. Last, and due to we didn't redirect both file descriptors outputs, there are inherited from their parent process; therefore, the $grep results are displayed in our current terminal. All this process is being represented in the image below. 

This redirection process using the stdin redirection operator (<) sets the contents (the keyword here) of some FILE to the STDIN file descriptor. 

We need to be plenty aware that even if we are redirecting STDIN, their contents aren't going to be on consideration unless the command we are using is set to read from it. In other words, $grep searches inside the contents of FILE, unless we don't explicitly declare a FILE, in that case, searches on STDIN. In the image below, we employ first the: "$grep "fox" < $PWD/text.txt" command, which highlights the word "fox" from the "The quick brown fox jumps over the lazy dog" string. Second, we declare a FILE for the $grep command and also redirect STDIN, with the following command: "$grep "fox" ./other_text.txt < $PWD/text.txt". This second command highlights the word "fox" but now from the "These are the contents of a new text with a new fox" string, ignoring the contents of STDIN.

1.2.- Redirecting the Standard Output (1)

By default, the two main Outputs, standard output (STDOUT) and standard error (STDERR), are redirected to our current terminal (computer screen); therefore, both have the initial path value of: /dev/pts/1.  We can redirect STDOUT to any FILE using the greater than operator: >. Let's say, for example, we need to store in a regular file the output for a find command which searches since our current directory (.) any regular file ending with the ".txt" extension. Calling the command: "find . -name '*.txt'", $find prints to our current terminal all the regular files that end with ".txt".

 If we want to store this $find output to a regular file, we can redirect STDOUT employing the '>' operator and save their contents to a new regular file. This new file has the STDOUT contents and is stored with the path and filename we declared. Notice that this time when we call the: "find . -name '*.txt' > $HOME/find_STDOUT.txt" command, the terminal doesn't print anything, because STDOUT is being redirected to a new file called: "$HOME/find_STDOUT.txt".

This STDOUT redirection process is represented in the image below. There we can see that STDIN and STDERR are both inherited to the forked process. But STDOUT takes the FILE of: "$HOME/find_STDOUT.txt"

1.3.- Redirecting the Standard Error Output (2)

We can redirect STDERR using the "2>" redirection operator. By default, STDERR has the /dev/pts/1 path, which means that any error that our command interpreter encounters, will be redirected to our current terminal. Like STDOUT, we can redirect STDERR to any regular FILE, although, STDERR is commonly redirected to /dev/null, which is a special Linux directory. In this directory, any FILE that is redirected to it is deleted.

Let's take the same $find command from the 1.3 section: "find . -name '*.txt'". This time I created a restricted directory called: /RestrictedDir, therefore when $find tries to search on it, it'll print a Permission Denied error on STDERR (our terminal)

Moreover, we can redirect the STDERR output to a regular file, using the "2>" operator. Redirecting STDERR and calling $find, prints STDOUT to our terminal but the Permission Denied error is redirected to $HOME/find_STDERR.txt

This STDERR redirection process is represented in the image below. There we can see that STDIN and STDOUT are both inherited to the forked process. But STDERR takes the FILE of: "$HOME/find_STDERR.txt". Moreover, due to STDOUT is inherited from the main process, the $find output is printed to our current terminal.

A word of advice, for STDOUT and STDERR, we can redirect their contents to a new file, employing the standard operator, >, and 2>, respectively. These operators delete if exists, or create a new file. If we add a '>' greater than symbol to both redirection operators, resulting in >>, and 2>>, such action appends their contents to the redirected regular file if exists, otherwise, creates a new file. In other words, using > and 2> create a new file. Using >> and 2>> appends their contents to a file. This append redirection option is handy for creating log files or debugging. 

1.4. Here Documents/Strings redirection to STDIN

When dealing when STDIN redirection often we encounter these two operators: << and <<<. The first one, <<, is called Here Documents, the second one, <<<, is called Here Strings. The next two subsections talk about both redirection types. 

1.4.1 Here Documents

The Here Documents, also called heredocs, are a special kind of STDIN redirection. This STDIN redirection is enabled with the "<<" operator, followed by an opening/ending string, which by convention is "EOF", although it can be any string. Therefore, all the contents that follow EOF are redirected to STDIN until the text reaches the final EOF.

Remembering, in section 1.1, when we deal with STDIN redirection, we can redirect the contents of FILE to STDIN. The Here document redirection does the same, but with the benefit to see what is being redirected to STDIN.

Let's take the same example from section 1.1, using the $grep "fox", command, to search on the "The quick brown fox jumps over the lazy dog" string. In the image below, due to we are using the Here Document redirection to STDIN, all the text that we enter is being redirected to STDIN. This redirection won't stop until the terminal reads the EOF string. Finally, the $grep command prints only the line with the "fox" coincidence. 

The Here Documents has its primary utility with non-interactive scripts. In other words, when writing a BASH/KSH script and we need the input for a command inside that script, instead of a separated FILE.

1.4.2 Here Strings

The Here Strings, are similar to Here Documents, a special kind of STDIN redirection. Here Strings, work with the "<<<" operator, and it differs from Heredocs because you can redirect a string on the fly and without the necessity to explicitly declare an opening/ending string. 

Here strings are useful when we want to feed a command with the contents of a variable, which is determined commonly through a subshell. Let's say the following example. We need to have a count of all rows of the last txt modified file from our current directory. This file can change every time, so we can't store it raw inside some variable. We need this variable to be continuously evaluated to determine which is the last modified file. We can obtain and save inside the VAR_LAST_FILE variable the last modified file with the command: $ls -aht --full-time *.txt | head -n 1 | cut -c 77-, if you don't recognize the pipe '|' symbol, don't worry, I'll explain it in the next section of this article. 

After storing the last modified file from our current directory into VAR_LAST_FILE, we can supply it as a Here String into the $wc -l command. Notice that using the VAR_LAST_FILE variable has the same effect as doing the process substitution directly. 

If suddenly you have the necessity to echoing a variable and then pipe it to another command, please be elegant and confuse others using Here Strings. Look at the difference in the image below.  

2.- Anonymous pipes '|' 

One of the principal utilities of the three standard file descriptors appears when using anonymous pipes. We can redirect the output from a command A to the input to a command B, thanks to the anonymous pipes using the '|' operator. This process is called piping.

This redirection process is represented in the image below, here STDOUT from the $commandA is being redirected to STDIN from the $commandB. Remember, when we call a new command it forks a new PID due to process substitution, using the '|' operator forks a new PID and therefore inherits the remain two file descriptors STDOUT and STDERR, from the main process. 

Let's say, for example, we have this "duplicated_file.txt" file, in which we have seven rows containing one number from one to three. These numbers are duplicated. We want to calculate how much each number is being duplicated, and we want the output sorted in descending order. This process can be achieved easily with piping. First, we sort the contents of the duplicated text file; then, we retain the unique values with a total of appearances. Finally, we sort the output again, but now doing a numeric sort and in printing the output in descending order.

This process of piping is called pipe redirection, or typical piping. 

2.1- Named Pipes - FIFOs

Up to now, all redirection methods have one direction. STDIN and Here Strings/Documents read contents from a FILE. STDOUT and STDERR can write their contents to a FILE. Pipe redirection can join STDOUT with STDIN. No one of them can read and write at the same time. The Named Pipes, also called FIFO (first-in first-out) pipes, solves this issue enabling the possibility to read and write data with the same object. Commonly used to transfer data between processes, one process writes to the FIFO pipe, and the other reads from it. 

You can create a named pipe with the $mkfifo command, followed by the name of the named pipe. This named pipe, even if it's a part of the system, has a size of zero, and it's labeled with the letter 'p.' The command: $mkfifo PlumberPipe in the image below, creates a named pipe in the /AdvancedBash directory

After created the PlumberPipe pipe, you can redirect any content to it. This stored content is saved until it's read from another command. In the example below, the echo string is redirected to PlumberPipe, and the process is running in the background through the '&' operator. When we read the contents of PlumberPipe with the $cat command, it prints "This content is redirected to the PlumberPipe pipe" and ends the 591 Process, which was started with the $echo command. 

Here you can read a more advanced example using named pipes with the Netcat command. 

3.- Conclusions

Linux has three primary standard file descriptors, STDIN (our keyboard), STDOUT (our current terminal), and STDERR (also our current terminal), which are embedded for any Linux command. Employing the redirection operators, we can redirect any standard file descriptor. Saving STDOUT/STDERR to a FILE, or read contents from a FILE and send it to STDIN. Additionally to the traditional redirection operators, we can redirect content to STDIN through Here Strings for dynamic redirection, and Here Documents for static scripting. The anonymous pipes are a handy way to redirect STDOUT from a command A and pass it to STDIN from a command B. The named pipes are useful when dealing with a bidirectional redirection.

Thanks for reading!

I'll leave this fantastic ASCII Mario® for the strong ones that made it until the end.