Stream Redirection

Under normal circumstances every Unix program has three streams opened for it when it starts up, one for input, one for output, and one for printing diagnostic or error messages. These are typically attached to the user's terminal (see tty(4)) but might instead refer to files or other devices, depending on what the parent process chose to set up.

-Taken from  the BSD Library Functions Manual for STDIN(3)

The input stream is referred to as Standard Input or stdin, the ouput stream as Standard Output or stdout and the Error stream as Standard Error or stderr.  As a systems admins, we often have need for input/output redirection, to many reasons for which to discuss within the context of this article.  However, we will highlight a few real-world use-cases for input/output redirection to which you'll be able to reference as a guide within the context of your own environment for further use.

Standard Input or STDIN

STDIN is typically taken from your keyboard. However, using input redirection, we can redirect the contents of a file as the input for a command.  This is quite simply accomplished by appending a "0< FILENAME " to the end of a give command:

For example, we see in the following diagram a typical STDIN scenario where a prompt seeking STDIN, asks the user to provide input via the keyboard in order to remove the file some_file.txt.

Using this example, we can redirect STDIN providing an answer file so that the command completes automatically without prompting for confirmation via the keyboard.  In the following example, we have created a text file named command_list that contains one line with the word "yes".  We again run the command rm but this time redirecting STDIN to our answer file, command_list.

The results are exactly as we expected.  The command completed successfully, removing the file some_file.txt while receiving it's input to the confirmation prompt from our answer file, command_list.

We can further simply the syntax by omitting the "0" with the same result:

Here is another example where we display the contents of a file:

We can also make use of a delimiter by the use of a '0<< DELIMITER' (the zero can be omitted to simplify the syntax.)  For example:

In the above example, we used the delimiter MY_DELIMITER to provide a multi-line input stream to wall to announce scheduled maintenance to our users.  As you can see, the input stream begins with the first delimiter MY_DELIMITER and terminates with the second MY_DELIMITER, everything in-between is sent as STDIN to wall.

Standard Output or STDOUT

STDOUT can be thought of as the output container produced by a given process.  Most programs will usually default their STDOUT to a screen buffer device such as the system console or a terminal like your X terminal.  So for example, when running the command ls, the output that is produces is sent to our connected X terminal:

So in the example above, we see a simple directory listing of our working directory.  The highlighted output from 'ls' was sent to STDOUT which happened to be our X terminal.

We can now redirect the output of ls to a file by the use of our output modifier 1>.  When using a output modifier, if the target file already exists, its contents will be overwritten.  If the target does not exist, a new file will be created.

As you can see, the output which had previously be sent to our screen was redirected to a newly created file called file_list.txt. Like input redirection, we can drop the 1 from our syntax to simply the command.

Rerunning a command with the > will only overwrite the contents of what had previously been stored in the target file:

Appending STDOUT

Appending of STDOUT to a target can be a very valuable tool to you for logging and other such utilities on a system.  In the above example, with each iteration of the command echo, the targeted output's contents were replaced with the commands most recent output.  You can append the output to STDOUT by the use of >> as the output modifier.  For example:

In the above, we began by zeroing out the file using the overwrite output modifier, '>'.  We then append, '>>' to the text "First Time:" followed by a directory listing, followed by the text "Second Time:" followed by a directory listing.  I've highlighted the output to help illustrate this example.

Although the above was a trivial example, in the following diagram, we've created a simple file monitoring program that makes use of the date command to log when a file trigger_file appears on the system. Lets take a look:

Still a very basic example, but you can now begin to see how powerful stream redirection can be.

Standard Error or STDERR

STDERR can also be thought of as a output container, but different from STDOUT in that its contents are of errors from the running process.  Also like STDOUT, STDERR usually defaults it's output to a screen buffer device like the console or a X terminal. Redirecting STDERR can also be useful for a variety of tasks; we will not go into all of them, but will highlight a couple examples to give you an idea.

When running find on the root of your file-system as a non-privileged user, your console will often times become polluted with Permission denied errors making it very difficult to sift through the output distinguishing the search string from the errors:

We know that we don't have permissions to all of these folders, but for the ones we do, we would like find to search in them but less all the permission denied errors.  We can do this by the use of the STDERR output modifier 2>.  Using the STDERR output modifier, we can now choose to output all the errors to a text file (for further analysis) or we can just discard them altogether.  In the first example, we'll send the errors to a text file:

As you can see, running find this time produced a much more clean output that only shows the exact hits of our search criteria.  If we like, we can now go back and exam the contents of error_log.txt to see if there was anything of further interest.  We could have also just as easily discarded the output as seen in our next example:

Here, we sent all the errors to the black hole, /dev/null.  This can be useful when you know the command you are running will generate errors but you do not wish to see them.

Appending STDERR

Just like STDOUT, we can use the STDERR append modifier 2>> if we would like the output of our command to be appended to a file rather than overwriting the contents.  For example:

From the above example, we see our first command ls completes with no output sent to the screen and STDERR sent to error_log.txt.  The second command, find, sends the results of the search to our screen (STDOUT) and any errors (STDERR) to error_log.txt.  Upon further inspection of error_log.txt, we see both commands threw errors but because we redirected STDERR, we never saw them until we inspected the contents of error_log.txt.

Combined Output

The final part of our stream redirection discussion focuses on combined out.  Leveraging what we have learned thus far, we may now combine both the STDOUT and STDERR streams into one file. A typical use case for this is system logging.  We often see jobs scheduled through crond or at that redirect their output to STDOUT and/or STDERR for historical records of the transpired events.  This can prove to be a very helpful tool when trying to understand why last night's batch process failed to run properly.

In this example, we look at a basic syntax to combine both STDOUT and STDERR into a single file:

Notice the use of our append modifiers >> for both STDOUT and STDERR.  Had we forgotten to use the append modifier, we would have lost the output from STDOUT.  When combining the redirection for STDOUT and STDERR, STDOUT is always written first followed by STDERR.  So depending on if you used the append or overwrite modifier for STDERR, your results will be quite different.  I'll demonstrate with a couple examples.

STDOUT  =  write,  STDERR = append:

STDOUT = write,  STDERR = write:

STDOUT = append,  STDERR = write:

So what we have learned, in order to preserve the output of both STDOUT and STDERR in a combined redirection, the STDERR modifier must be set to append or the contents of STDOUT will be lost.

Now although the above examples have helped us better understand the roll of the STDOUT and STDERR modifiers, we had to type a lot more than what is necessary.  Lets lose some keystrokes and simplify our syntax.  We'll start by using a combined output that will overwrite the target file's contents:

Our syntax from above with use of the single > followed by the file-name and then a 2>&1 tells our interpreter to combine both STDOUT and STDERR, and overwrite the contents of our target file with their output.  For our final example, we combine both STDOUT and STDERR, but this time, appending to the contents of our target file:

As expected, the contents of the file were not lost as we used the append modifier, >>, to combine the output into the existing file.

This concludes our article on Stream Redirection, we hope you found this helpful!

Add this page to your favorite website

Comments

Add New Search

Write comment
Name:
Email:	do not notifynotify
Website:
Title:
UBBCode:	-color-aquablackbluefuchsiagraygreenlimemaroonnavyolivepurpleredsilvertealwhiteyellow -size-tinysmallmediumlargehuge
	Please input the anti-spam code that you can read in the image.

!joomlacomment 4.0 Copyright (C) 2009 Compojoom.com . All rights reserved."