Unit 5: Shell Features

I. Orientation: The Shell as Command Interpreter

The shell is the program that reads the lines you type at a terminal, interprets any special characters in them, and hands the resulting command to the operating-system kernel for execution (the Bourne shell sh dates from 1977; bash, the GNU Bourne-Again Shell, from 1989). Before a program ever runs, the shell performs a fixed sequence of transformations on the command line — and pattern matching, redirection, and pipes are three of those transformations.

Defining properties the later sections rely on:

• Word splitting first, then expansion: the shell breaks the line into words, then expands each word; the program you invoke never sees the special characters, only the results.
• The shell, not the program, expands wildcards: ls *.txt reaches ls as an already-expanded list of filenames. This is called globbing.
• Three standard streams: every process opens stdin (file descriptor 0), stdout (fd 1) and stderr (fd 2) by default, all connected to the terminal.
• Everything is a file: terminals, disk files and inter-process channels all present the same read/write interface, which is why the same operators redirect to any of them.
• Order of processing on one line: redirections and pipe setup are established before the command executes, so a program starts life already wired to its inputs and outputs.

II. Pattern Matching using Wildcards

Filename generation (globbing) by the shell

Wildcards are unquoted metacharacters that the shell expands against the set of existing filenames in a directory, replacing the pattern with an alphabetically sorted list of matches.

A. The core wildcard characters

The point is that each wildcard matches a different shape of text within a filename.

• * — any string: matches zero or more of any character.
  • ls *.c → every file ending in .c (e.g. main.c, io.c).
  • ls a* → every name beginning with a.
• ? — any single character: matches exactly one character.
  • ls file?.txt matches file1.txt, fileA.txt but not file10.txt or file.txt.
• [...] — character class: matches one character from the enclosed set.
  • ls [abc]* → names starting with a, b or c.
  • Ranges: [0-9] any digit, [a-z] any lowercase letter; ls log[0-9].txt matches log0.txt…log9.txt.
  • Negation: [!abc] (or [^abc]) matches one character that is not a, b or c.

B. Rules and edge cases of globbing

The point is that globbing has boundaries that distinguish it from full regular expressions.

• Hidden files excluded: a leading . is not matched by * or ?; ls * skips .bashrc. You must write .* explicitly.
• No match → literal pattern: in default bash, if ls *.xyz matches nothing, the word *.xyz is passed unchanged to the command.
• Glob ≠ regex: * here means "any string", whereas in a regular expression * means "zero or more of the preceding item". These are different languages.
• Quoting disables expansion: ls "*.c" or ls \*.c searches for a file literally named *.c.

C. Worked example — targeting a subset

$ ls
data.log  err1.log  err2.log  err10.log  notes.txt
$ ls err?.log
err1.log  err2.log
$ ls err[0-9]*.log
err1.log  err2.log  err10.log

• err?.log requires exactly one character, so err10.log (two digits) is excluded.
• err[0-9]*.log requires one digit then any string, so all three err logs match.

III. Input/Output Redirection

Reconnecting the standard streams to files

Redirection replaces a process's default terminal connection for stdin, stdout or stderr with a file, using operators the shell interprets before running the command.

A. Output redirection

The purpose is to capture a command's stdout in a file instead of the screen.

• > — overwrite: ls > list.txt sends stdout to list.txt, truncating it to empty first.
• >> — append: date >> log.txt adds to the end of log.txt, preserving existing content.
• Full form uses the descriptor: > is shorthand for 1>; ls 1> list.txt is identical.
• Creation: if the target file does not exist, both > and >> create it.

B. Input redirection

The purpose is to feed a file into a command's stdin in place of keyboard input.

• < — read from file: sort < names.txt gives sort its input from the file; equivalent to 0< names.txt.
• Contrast with an argument: wc -l < f.txt prints only the count (input came anonymously via stdin); wc -l f.txt prints the count and the filename, because f.txt is an argument the program can see.
• Here-document <<: feeds inline text as stdin until a delimiter line.
  
  BASH

    cat << END
    line one
    line two
    END

  
  Everything up to END becomes the stdin of cat.

C. Error redirection and stream combination

The point is that stdout (fd 1) and stderr (fd 2) are separate and can be routed independently.

1. Separating the streams: find / -name x 2> errors.txt sends only error messages to the file, leaving normal results on screen.
2. Merging the streams: command > out.txt 2>&1 sends stdout to out.txt, then makes stderr point to the same place as stdout.
   • 2>&1 means "make fd 2 a duplicate of fd 1". Order matters: it must follow the > redirection, because it copies wherever fd 1 currently points.
   • Discarding output: command 2> /dev/null throws errors away; /dev/null is the "bit bucket" that discards everything written to it.

IV. Pipes and Pipe Chaining

Connecting the output of one process to the input of the next

A pipe (|) is a shell construct that connects the stdout of one command directly to the stdin of another, so data flows between concurrently running processes without any intermediate file.

A. The pipe operator

The point is that | wires two commands together in memory.

• Syntax: command1 | command2 — stdout of command1 becomes stdin of command2.
• No temporary file: the pipe is an in-kernel buffer; contrast the two-step ls > tmp; sort tmp with the single ls | sort.
• Concurrent execution: both commands run at the same time; command2 consumes data as command1 produces it, rather than waiting for it to finish.
• stderr not carried: only stdout flows through |; error messages still go to the terminal unless separately redirected.

B. Pipe chaining (pipelines)

The purpose is to build a data-processing assembly line by joining several filters, each transforming the stream in turn.

• General form: cmd1 | cmd2 | cmd3 | ..., evaluated left to right, each stage's output feeding the next stage's input.
• Filters are programs designed for this: grep (select lines by pattern), sort (order lines), uniq (collapse adjacent duplicates), wc (count), head/tail (first/last lines), cut (select fields).
• Worked example — most frequent words:
  
  BASH

    cat report.txt | tr ' ' '\n' | sort | uniq -c | sort -nr | head -3

  • tr ' ' '\n' puts each word on its own line.
  • sort groups identical words together so uniq can act.
  • uniq -c prefixes each unique line with its count.
  • sort -nr orders numerically (-n), highest first (-r).
  • head -3 keeps the top three lines.

C. Combining pipes with redirection

The point is that pipes and redirection compose on a single command line.

• Capturing a pipeline's result: ps aux | grep ssh > matches.txt runs the pipeline, then redirects its final stdout to a file.
• tee — split the stream: ls | tee files.txt | wc -l writes the listing to files.txt and passes it on to wc -l, so you both save and process the data.
• Ordering rule: the shell sets up pipes first and per-command redirections second, so 2>&1 inside a pipe stage affects only that stage.

D. Applications and limitations

The point is that pipelines are powerful for stream data but have inherent constraints.

• Strength — composability: small single-purpose tools combine into complex operations without new programs (the Unix philosophy).
• Strength — memory efficiency: large data streams flow through without ever being stored whole on disk.
• Limitation — one-directional, text-oriented: a pipe carries a byte stream downstream only; the receiving command cannot send data back.
• Limitation — exit status: by default a pipeline's status is that of the last command, so a failure in an earlier stage can be masked.
• Limitation — no random access: a filter sees the stream sequentially and cannot seek backwards, unlike a program reading a redirected file.