I/O redirection

I/O redirection and piping
Previous | Next

Topics

Redirecting to a file
Appending to a file
noclobber to prevent accidentally overwriting a file
Protection attributes
stdout and stderr
Pipes
Command substitution
Inline data
Inline data in scripts
See also

Redirecting to a file

You can redirect or pipe i/o in much the way you might under cmd.exe. Here’s a simple example redirecting stdout from the word count of the famous “Hello, world” program. cat just copies from any files you tell it or, by default, from stdin to stdout.

74 E% cd hello 75 D% ls hello.c hello.exe 76 D% cat hello.c #include <stdio.h> main () { printf("Hello, world.\n"); } 77 D% wc hello.c >hello.wc 78 D% cat <hello.wc 5 8 72 hello.c

wc tells us that hello.c has 5 lines, containing 8 words, totaling 72 characters.

If the file you write to with > exists, it’s first truncated to zero length (discarding the old contents); if the file doesn’t exist, it’s created. With <, it’s an error if the file doesn’t exist.

Appending to a file

Data can be appended to a file with the >> operator:

79 D% echo that^'s all folks >>hello.wc 80 D% cat hello.wc 5 8 72 hello.c that's all folks 81 D% _

When you append with >>, if the file exists, data is written onto the end; if it doesn’t exist, it’s created. (The single quote character has special meaning to the shell on the command line; the special meaning is turned off by the shell’s escape character, ^.)

noclobber to prevent accidentally overwriting a file

Not everyone is comfortable with letting the shell glibly toss away an existing file if you type > when you meant >> or lose it somewhere if you mistype an existing filename with >>. The noclobber variable lets you tell the shell you want this to be caught, so you can decide if this was really what you meant.

If you set noclobber, you have to type >! to redirect to an existing file:

81 D% set noclobber = 1 82 D% echo trash this file > hello.c csh: Couldn't open 'hello.c' as a redirected standard output.

Come to think of it, let’s not overwrite that file.

Similarly if you want to append to something that doesn’t already exist:

83 D% echo appended data >> newdata csh: Couldn't open 'newdata' as a redirected standard output. 84 D% echo appended data >>! newdata 85 D% cat newdata appended data 86 D% rm newdata

Protection attributes

If a file has any of the special protection attributes, hidden, read-only or system, set, you cannot overwrite it by redirecting i/o to it. Even when you type !, you still can’t. Before you can redirect to it, you must clear all these attribute bits.

87 D% ls -l zork -SHAR Feb 23 13:16 0 zork 88 D% echo new zork data >! zork csh: Couldn't open 'zork' as a redirected standard output. 89 D% chmod -R zork 90 D% echo new zork data >! zork csh: Couldn't open 'zork' as a redirected standard output. 91 D% chmod -SH zork 92 D% ls -l zork ---A- Feb 23 13:16 0 zork 93 D% echo new zork data > zork 94 D% _

stdout and stderr

Redirecting both stdout and stderr together is done by adding an ampersand. For example, using echo -2 to deliberately write to stderr and parentheses for a simple grouping:

94 D% (echo -2 error; echo standard) > zork error 95 D% cat zork standard 96 D% (echo -2 error; echo standard) >& zork 97 D% cat zork error standard 98 D% _

Separately redirecting stderr and stdout to different files is a little tricky: first you redirect them both, then redirect stdout by itself. Here’s an example running the C compiler with stdout to log and stderr going to errors.

98 D% cl hello.c >& errors > log

You can type as many i/o redirections in a row as you like. The shell evaluates them one after another. If you redirect to a new file, then redirect to something else, the effect is just like touch’ing the file.

Pipes

Pipes are a way of connecting a series of activities together so that the output of one is read as input to the next. Each of the activities runs asynchronously and concurrently with the others. Data is passed completely in memory and is very fast.

The syntax is similar to i/o redirection in its use of the & character. To pipe just stdout, use | by itself:

99 D% ls -L | more

To pipe both stdout and stderr together, use |&:

100 D% cl hello\hello.c |& more

The leftmost part of the pipeline is evaluated directly by the shell’s current thread. The successive right parts are evaluated by child threads. (This is so that piping a command that lists status information on the current thread through a filter like more operates sensibly.) Each part of the pipeline can be an arbitrarily complex statement, perhaps even run in a separate window.

Pipes are much faster and more responsive than with vanilla Windows due to improved buffering and scheduling technology. A long pipeline finishes much faster. Also, when you type Ctrl-C to interrupt, it comes back immediately without a lot of nuisance messages.

Command substitution

A particularly novel way of piping statements together is to use the output of one as command line arguments of another. This is called command substitution and you indicate it by typing backquotes, `...`, or double backquotes, ``...``, (literally, two backquotes) around a command.

101 D% ls +a . hello zork .. memos 102 D% echo `ls +a` . hello zork .. memos 103 D% _

When command substitution is done, all the extra “white space” (space characters, tabs and newlines) is squeezed out and any ANSI escape sequences that might have turned on highlighting or color, etc., are also deleted. You just get the list of words the backquoted command wrote to stdout. In this example, the order of the files is a bit scrambled when the line ends are removed; the -1 (numeric one) single column option can fix this. (Try it again using ls +a1 inside the backquotes.)

With double backquotes, ANSI escape sequences are still deleted, but each line is taken as a separate word. No additional parsing for words is done. This is particularly useful when dealing with pathnames that may contain spaces. Here’s an example using ls to give a detailed listing of itself:

103 D% whereis ls D:\Program Files\Hamilton C shell 2012 x64\Bin\ls.exe 104 D% ls -l ``whereis ls`` ---A-- Jul 22 08:00 155968 d:\Program Files\Hamilton C shell 2012 x64\Bin\ls.exe

If there are any variable substitutions inside the backquotes, they’re done by the child, not the parent. This lets you easily embed for loops and other programming constructs into the command substitution.

Inside backquotes, only the backquote character needs to be escaped to avoid having it processed by the parent thread.

Inline data

A novel variation on i/o redirection is inline data, also called “here” documents: literal text you want the shell to feed a command as stdin. Here’s an example:

105 D% cat <<eof 106 D? (this is the inline data) 107 D? eof (this is the inline data) 108 D% _

The << operator is followed by a string the shell is asked to look for to mark the end of the inline data. The end-of-data string can be virtually anything you like, including wildcard characters, dollar signs, etc.; their normal meaning is turned off and they’re treated as ordinary literal characters. Only quote or escape characters have any special meaning, which is to turn off substitutions in the inline text (as we’ll discuss in a moment). Continuation lines as the shell collects the inline data get a different prompt, controlled by the prompt2 variable. Once the data has been collected in memory, it’s written through a pipe to the command.

One very convenient use of inline data is when you want to quickly search for any one of a number of important words in a large library. E.g., to scan for some specific strings in a set of C files:

108 D% fgrep -ns <<xxx ~\sh\*.c 109 D? Open 110 D? Close 111 D? Read 112 D? Write 113 D? xxx : search results

In situations where the inline data is being created inside a larger structure, the data is assumed to start on the first line following a break between statements. For example, inside a for loop:

114 D% for i = 1 to 3 do 115 D? cat <<eof; echo i = $i 116 D? (this is the inline data) 117 D? eof 118 D? end (this is the inline data)_ i = 1 (this is the inline data)_ i = 2 (this is the inline data)_ i = 3

If you want to put several inline i/o redirections on the same line, type the associated inline data sections, each with its own terminating string, in the same left-to-right order in which they appeared.

So far, we’ve just shown examples involving static text. But it’s also possible to ask the shell to do command and variable substitutions on the inline text:

119 D% cat << *** 120 D? The ^$home directory is $home. 121 D? Today's date is `date`. 122 D? *** The $home directory is d:\Nicki Today's date is Fri Jul 22 2011 8:57:22.03. 123 D% _

Although substitutions and escape characters inside the here document are processed, quotes (both single and double) are not.

The C shell implements here documents by spawning a child thread to do any substitutions and write the results into a pipe feeding the current thread as it continues to evaluate the statement. If the here document contains references to shared variables, they’ll be evaluated by that other thread. And unless they’re local variables, the values will not be snapshotted when the here document thread is created. If the current thread (or any other thread) continues to make changes to a variable after the here document thread is spawned but before it evaluates the variable, the here document will contain the new, not the old value.

Command and variable substitution and escape processing inside a here document is turned off if any part of the end-of-data string following the << is quoted with ', " or ` or escaped:

123 D% cat <<^*** 124 D? The ^$home directory is $home. 125 D? Today's date is `date`. 126 D? *** The ^$home directory is $home Today's date is `date`. 127 D% _

Inline data in scripts

Inline data can be especially useful if you’re writing a script file or passing commands to the shell through a pipe. In either of these cases, the low-level ReadFiles to the Windows kernel cannot be depended on to stop at the end of a line because pipes and files are considered block-oriented rather than line-oriented like the keyboard. If too many characters are read, there’s no simple way to back up. For this reason, it’s not realistic to write a script where a child process is supposed to inherit stdin pointed into the script file. In a script file, this is not reliable:

: : csh echo hello exit : :

The file descriptor the child process inherits will likely not be pointing at the “echo hello” when it exits, the parent will likely not find it pointed just past the “exit”. This type of script should be written as:

: : csh <<eof echo hello exit eof : :