Redirection is the shell’s one genuinely powerful idea: every program speaks three numbered streams, and you rewire them from outside. Once the numbering is clear, the syntax stops being incantation.
Topic 1: The Three Standard Streams
Every process starts with three file descriptors already open:
| FD | Name | Default | Purpose |
|---|---|---|---|
0 | stdin | keyboard | Input |
1 | stdout | terminal | Normal output — data |
2 | stderr | terminal | Errors and diagnostics — not data |
The split exists so you can pipe data onward while errors still reach a human. A program that writes its progress messages to stdout has broken that contract, and its output cannot be piped cleanly.
cmd > out.txt # stdout to file (1> is implied)
cmd 2> err.txt # stderr to file
cmd < in.txt # stdin from file
cmd >> out.txt # APPEND rather than truncate
cmd > /dev/null # discard stdout
cmd 2> /dev/null # discard stderr -- use sparingly
Topic 2: The Ordering Rule
This is the part that looks arbitrary until you see the mechanism.
cmd > file 2>&1 # BOTH to file ✓
cmd 2>&1 > file # stderr to TERMINAL, stdout to file ✗
2>&1 means “make fd 2 point wherever fd 1 currently points”. It is a snapshot, not a link. Redirections are processed left to right:
cmd > file 2>&1
────────┬─────
1. fd1 → file
2. fd2 → wherever fd1 points → file ✓ both in the file
cmd 2>&1 > file
─────┬────────
1. fd2 → wherever fd1 points → the terminal
2. fd1 → file ✗ fd2 still on the terminal
Bash offers a shorthand that avoids the trap entirely:
cmd &> file # both, bash-only
cmd &>> file # both, appending
cmd > file 2>&1 # both, POSIX -- use this in /bin/sh scripts
Swapping streams:
Occasionally you want errors on stdout and data on stderr — for example to grep the errors:
cmd 3>&1 1>&2 2>&3 3>&-
Read it as: save fd1 in fd3, point fd1 at fd2, point fd2 at the saved fd3, close fd3. It is the classic three-way swap, using fd 3 as the temporary.
Topic 3: Pipes and the Subshell Problem
A pipe connects one process’s stdout to the next’s stdin. Both run concurrently, not one after the other.
grep ERROR app.log | awk '{print $4}' | sort | uniq -c
The trap that costs people an afternoon:
count=0
find /var/log -name '*.log' | while read -r f; do
(( count++ ))
done
echo "$count" # prints 0
Every element of a pipeline runs in a subshell. The while loop increments a copy of count, and that copy dies when the subshell exits. Three ways out:
# 1. Process substitution -- the loop runs in the CURRENT shell
while read -r f; do (( count++ )); done < <(find /var/log -name '*.log')
# 2. A here-string, when the data is already in a variable
while read -r f; do (( count++ )); done <<< "$file_list"
# 3. lastpipe -- bash-only, requires job control off
shopt -s lastpipe
find ... | while read -r f; do (( count++ )); done
Option 1 is the general answer and worth making a habit.
Exit status of a pipeline:
By default a pipeline’s exit status is that of the last command, so a failure upstream is invisible:
false | true; echo $? # 0 -- the failure vanished
set -o pipefail
false | true; echo $? # 1 -- now it propagates
echo "${PIPESTATUS[@]}" # per-command statuses: "1 0"
PIPESTATUS is an array holding every stage’s exit code, and it is the only way to know which stage failed. Capture it immediately — the next command overwrites it.
Topic 4: Here-Documents
A here-doc feeds a literal block to a command’s stdin. It is how scripts generate config files, SQL, and multi-line payloads without a pile of echo calls.
cat > /etc/myapp/config.yml <<EOF
server:
host: ${HOSTNAME}
port: 8080
env: ${ENVIRONMENT}
EOF
Quoting the delimiter changes everything:
cat <<EOF # UNQUOTED: expands $vars, $(cmds), backticks
Path is $PATH
EOF
cat <<'EOF' # QUOTED: completely literal, expands NOTHING
Path is $PATH
EOF
Quote the delimiter whenever you are generating a script, a systemd unit, or anything else containing $ that must survive verbatim. Forgetting this is how a generated cron file ends up with the shell’s $PATH baked in instead of the literal string.
Indentation with <<-:
deploy() {
cat <<-EOF
This block is indented with TABS
and the leading tabs are stripped.
EOF
}
<<- strips leading tab characters only — not spaces. It exists so here-docs can be indented to match surrounding code, and it is finicky enough that many people simply do not indent them.
Here-strings:
grep "pattern" <<< "$variable"
read -ra parts <<< "$csv_line"
jq . <<< "$json_response"
<<< is a one-line here-doc. It is the tidy way to feed a variable to a command that wants stdin, and it avoids echo "$var" | with its extra process and its trailing-newline surprises.
Topic 5: Process Substitution
<(cmd) runs a command and presents its output as a filename. This is the tool for commands that insist on files rather than stdin.
diff <(sort file1) <(sort file2) # compare without temp files
comm -13 <(sort a.txt) <(sort b.txt) # lines only in b
join <(sort -k1 users) <(sort -k1 roles)
while read -r line; do ... done < <(cmd) # no subshell -- see Topic 3
Under the hood the shell creates /dev/fd/63 or a FIFO and passes that path. You can see it:
echo <(echo hi) # prints something like /dev/fd/63
The writing form >(cmd) exists too, and tee is where it pays off:
build 2>&1 | tee >(grep -i error > errors.log) > build.log
One pass, two destinations, filtered differently.
Note on portability: process substitution is not POSIX. It works in bash, zsh and ksh, but not in dash — which is what /bin/sh is on Debian and Ubuntu. A script with #!/bin/sh and <(...) fails on exactly the systems where it matters most.
Topic 6: Command Substitution and Its Cost
now=$(date +%Y-%m-%d)
count=$(grep -c ERROR app.log)
files=$(ls) # ← still wrong, for the reasons in the quoting lesson
Prefer $( ) over backticks: it nests, and quoting inside it behaves normally.
Two behaviours to know:
- Trailing newlines are stripped. All of them. Usually convenient, occasionally not — to preserve them, append a sentinel:
out=$(cmd; echo x); out=${out%x}. - It forks. Each
$( )is a fork and an exec. In a tight loop that is the dominant cost:
# 10,000 forks
for f in *; do
name=$(basename "$f")
done
# zero forks -- parameter expansion, from the quoting lesson
for f in *; do
name=${f##*/}
done
On a directory of ten thousand files the second version is roughly two orders of magnitude faster. This is the single highest-value optimisation in shell scripting: when a builtin can do it, do not fork.
Try it yourself: Time both loops over a large directory with time. Then run diff <(ls /etc) <(ls /usr/share) and confirm no temp files were created.
Common mistake: Using cmd > file 2>&1 inside a loop that appends. The > truncates on every iteration, so you keep only the last one. Use >> for accumulation, or redirect the whole loop once: for ...; do ...; done > file.