|

Linux Redirection and Piping

Linux Redirection and Piping

Linux Redirection is a very interesting topic. When we talk about redirection, we are referring to I/O or input/output redirection. It is possible to redirect the input and output of various commands to and from files, or to other commands in a process known as piping. By learning all of the standard and common commands in Linux, you’ll be able to start chaining them together to create powerful new commands or pipelines. In this episode, we’ll tackle several ways to approach this technique and examine some new ways to handle input and output data. Let’s jump in.

Linux Redirection


stdin stdout stderr

Before we get started, we need to establish a few things with regard to the terms listed above. In Linux, the syntax can be a bit cryptic, with most things either represented by single characters or an abbreviated version of characters. Just by looking at these terms however, we can often figure out what they are intended to mean. stdin represents Standard Input, and it is what a program takes in so to speak. stdout stands for, you guessed it, Standard Output, and this is what the program spits out as it runs. Finally we have stderr which is when a program complains about problems, and who doesn’t know someone like that?

stdin

The first thing to know is that standard input is identified by the less than < character. The syntax is typically program < input. The easiest way to see how this works is to simply complete an example.

vagrant@homestead:~/Code/Laravel$ ls
app bootstrap composer.lock phpunit.xml readme.md vendor
artisan composer.json CONTRIBUTING.md public server.php
vagrant@homestead:~/Code/Laravel$ wc < phpunit.xml
18 24 567
vagrant@homestead:~/Code/Laravel$

In this snippet we simply list the files in our Laravel directory and then use the word count program and feed it information using standard input. The result is output to the terminal and what those numbers mean is 18 words, 24 lines, and 567 characters. Now, notice that the name of the file is not present in that output. That is because wc is not aware of the file it was provided. In fact what happened is the contents of that file where read in through standard input to wc, and wc simply outputs information about that stream so to speak to the terminal.

stdout and stderr

Standard output is identified by the greater than > symbol. Breaking news: We've been using stdout all along, we just never think of stdout in the way that it is actually working. All stdout means is that the output of a program is being presented to the terminal. We can examine how this works with the simple ls command. An example is in order!

vagrant@homestead:~/Code/Laravel$ ls
app bootstrap composer.lock phpunit.xml readme.md vendor
artisan composer.json CONTRIBUTING.md public server.php
vagrant@homestead:~/Code/Laravel$ ls > files.txt
vagrant@homestead:~/Code/Laravel$ ls
app bootstrap composer.lock files.txt public server.php
artisan composer.json CONTRIBUTING.md phpunit.xml readme.md vendor
vagrant@homestead:~/Code/Laravel$ cat files.txt
app
artisan
bootstrap
composer.json
composer.lock
CONTRIBUTING.md
files.txt
phpunit.xml
public
readme.md
server.php
vendor
vagrant@homestead:~/Code/Laravel$

In this example, we first simple type the ls command and observe the files in our directory. Easy stuff. Next, we use redirection to send standard output to a text file named files.txt. Once again we run the ls command and now we can see that there is a new file in our directory named files.txt. Finally, we examine the contents of that file using the cat command, and what we find is that the output of the ls command was placed into this file when we had run ls > files.txt. Cool!

Something to note with stdout is that there are some operators to be aware of. Those are 1>, 2> and >>. 1> is basically means standard output. So when you use >, it is kind of an implied 1>. The 2> represents a standard error. So for example if you tried to list the contents of a directory that does not exist, you see some errors on the screen. You can redirect that standard error to a file just like we did with standard output. Finally, the >> operator simply means to append to the file. When we redirect, all contents in the file get overwritten. With >> we can append. Let's see these concepts in action.

vagrant@homestead:~/Code/Laravel$ ls haha
ls: cannot access haha: No such file or directory
vagrant@homestead:~/Code/Laravel$ ls haha 2> error.txt
vagrant@homestead:~/Code/Laravel$ cat error.txt
ls: cannot access haha: No such file or directory
vagrant@homestead:~/Code/Laravel$ ls katyperry 2>> error.txt
vagrant@homestead:~/Code/Laravel$ cat error.txt
ls: cannot access haha: No such file or directory
ls: cannot access katyperry: No such file or directory
vagrant@homestead:~/Code/Laravel$

Let's walk through what just happened. First off, we try to list the contents of something that does not exist which is haha. As expected, we see a nice error to let us know the misstep we just made. Next, we run the same command but add 2> error.txt. You'll note that no error message came to the screen when we did that. That does not mean there wasn't a problem however. We simply instructed Linux to send the standard error to a file named error.txt. When we view that text file, we now see the error. Next, we try to list the contents of katyperry, and we know that this will fail, because katyperry does not exist in this file system. By using the 2>> operator with error.txt, we instruct Linux to append any errors to the existing error.txt file. A quick examination of error.txt does in fact now contain both the prior error we created, and the new one when we tried to ls katyperry. Great work.

Introduction to Linux Piping

The vertical line or pipe symbol | is the operator that allows us to begin piping. Now what this means is that you can combine commands in order to send standard output from one command to standard input on another. You'll often find this used with grep, but the possibilities are limitless. Let's check out a quick example using the ls and the grep command together.

vagrant@homestead:~/Code/Laravel$ ls
app bootstrap composer.lock error.txt phpunit.xml readme.md vendor
artisan composer.json CONTRIBUTING.md files.txt public server.php
vagrant@homestead:~/Code/Laravel$ ls | grep md
CONTRIBUTING.md
readme.md
vagrant@homestead:~/Code/Laravel$

This is pretty slick. As you can see, first we just ran the ls command to see what is in our directory. Now let's say you want to look for files that only have the letters md in them. Well, we can run the command ls | grep md and note that we get the two files that have md in them. Those would be CONTRIBUTING.md and readme.md. So what happened here is the output of ls, which is all of the files that exist in our Laravel directory, were sent as input to the grep command where grep was instructed to search for the letters md.

xargs

Not all commands accept information on standard input. For example the echo command is one such example. There is a work around however. Linux has a handy little command called xargs, and what it does is to act as a transcoder of sorts that accepts standard input and then formats that data as arguments so to speak for the next command in the chain. Let's demonstrate this whole concept now.

vagrant@homestead:~/Code/Laravel$ ls | echo

vagrant@homestead:~/Code/Laravel$ ls | xargs echo
app artisan bootstrap composer.json composer.lock CONTRIBUTING.md
error.txt files.txt phpunit.xml public readme.md server.php vendor
vagrant@homestead:~/Code/Laravel$

What we see here is that the first instance of using ls and echo together via the pipe is that we get a blank line. This is because the echo command expects some data to actually echo out to the terminal. By including xargs after the pipe however, we can see that echo now successfully outputs the data it received via the output of ls by way of xargs. Hopefully that makes sense. When in doubt, just fire up your terminal and practice these techniques a few times.

tee

The tee command is pretty slick because it acts like a T. Meaning, picture feeding data into the bottom of a capital letter T, and then having that data get blasted out of both the left and right spigots of the T. That is sort of what tee does for us. Let's see an example.

vagrant@homestead:~/Code/Laravel$ ls | tee teeout.txt
app
artisan
bootstrap
composer.json
composer.lock
CONTRIBUTING.md
error.txt
phpunit.xml
public
readme.md
server.php
teeout.txt
vendor
vagrant@homestead:~/Code/Laravel$ cat teeout.txt
app
artisan
bootstrap
composer.json
composer.lock
CONTRIBUTING.md
error.txt
phpunit.xml
public
readme.md
server.php
teeout.txt
vendor
vagrant@homestead:~/Code/Laravel$

We can see in this example that the command ls | tee teeout.txt did two things. It sent the output to the terminal, as we can see by the file names listed, and it also sent that output to a file named teeout.txt. To confirm this, we run cat teeout.txt and voila, we can see the contents of that file containing the same output that was sent to the terminal. Pretty slick!

Linux Redirection and Piping Summary

This was a fun episode to learn about redirection and piping in Linux. With these concepts we became familiar with the formal ideas of stdin, stdout, and stderr. Really these are just fancy acronyms for things you've been working with all along. Pipes help us control data flow between different programs, allowing us to pipe output from one command to input on another. Finally, we learned that not all commands are able to accept standard input, and in that scenario you may be able to use xargs to overcome this.