Unit 6: Filters and Regular Expressions

I. Orientation: The Filter Model

A filter is a command that reads a stream of text from standard input (stdin), transforms it, and writes the result to standard output (stdout), leaving the original file untouched. Filters embody the Unix philosophy (Bell Labs, 1970s): small tools that do one thing well and combine through pipes.

• Three standard streams: every process has stdin (fd 0), stdout (fd 1), stderr (fd 2); filters default to reading fd 0 and writing fd 1.
• Line orientation: filters treat input as records separated by the newline \n; most operate line by line.
• Non-destructive: output goes to the terminal or a redirect (>, >>); the source file is unchanged unless redirection overwrites it.
• Composability: the pipe | connects one command's stdout to the next command's stdin, forming a pipeline.
• Exit status: a filter returns 0 on success; grep uniquely returns 1 when no line matches, driving conditional logic.

II. Pattern Matching — grep and Extended grep

Both search input for lines matching a pattern and print the matching lines; they differ only in the regex dialect they accept.

A. grep

grep (Global Regular Expression Print) prints lines that match a basic regular expression (BRE).

• Syntax: grep [options] PATTERN [file...] — e.g. grep "error" log.txt prints every line containing error.
• Case control: -i ignores case, so grep -i unix matches Unix and UNIX.
• Inversion: -v prints non-matching lines: grep -v "^#" conf drops comment lines.
• Counting and locating: -c prints only the count of matching lines; -n prefixes each match with its line number.
• Word and whole-line: -w matches whole words only; -x requires the whole line to match.
• Recursion: -r descends directories: grep -r "TODO" ./src.
• BRE quirk: +, ?, {, |, (, ) are literal unless backslash-escaped (\+, \{2\}).

B. Extended grep

egrep, or the equivalent grep -E, interprets extended regular expressions (ERE), where the metacharacters work without escaping.

• Alternation: grep -E "cat|dog" matches either word; in plain grep this needs \|.
• Quantifiers unescaped: grep -E "ab+c" matches one-or-more b; a{2,4} matches 2–4 as directly.
• Grouping: grep -E "(ab)+" repeats the group ab.
• Contrast 1 (BRE): compact but requires escaping every advanced operator — safer for simple literals.
• Contrast 2 (ERE): verbose metacharacters read naturally — preferred for alternation and grouping.

III. Field and Line Selection — cut, head, tail

These filters slice the stream by column position or by position at the file's ends.

A. cut

cut extracts columns from each line, either by character position or by delimited field.

• By character: cut -c1-5 file prints characters 1 through 5 of every line.
• By field: -f selects fields and -d sets the delimiter — cut -d: -f1 /etc/passwd prints usernames (field before the first :).
• Multiple fields: cut -d, -f1,3 data.csv keeps columns 1 and 3.
• Limitation: the delimiter is a single character and repeated delimiters are not collapsed, so uneven whitespace defeats cut -d' ' (use tr or awk instead).

B. head

head prints the beginning of the stream.

• Default: head file prints the first 10 lines.
• Custom count: head -n 3 file (or head -3 file) prints the first 3 lines.
• Bytes: head -c 20 file prints the first 20 bytes.
• Pipeline use: sort data | head -n 5 yields the five smallest values.

C. tail

tail prints the end of the stream and can follow a growing file.

• Default: tail file prints the last 10 lines.
• Custom count: tail -n 2 file prints the last 2 lines.
• From a point: tail -n +5 file prints from line 5 to the end (note the +).
• Follow mode: tail -f app.log streams new lines as they are appended — the standard way to watch live logs.

IV. Ordering and Counting — sort, wc, uniq

This group reorders records and reports quantities.

A. sort

sort orders lines and, with keys, orders by chosen fields.

• Default: lexicographic ascending order using the locale collation.
• Numeric: -n sorts by numeric value so 10 follows 9, not precedes it; -r reverses.
• Key selection: -k picks a field and -t sets the separator — sort -t: -k3 -n /etc/passwd sorts by numeric UID.
• De-duplication: -u removes duplicate lines while sorting.
• Prerequisite role: sort must precede uniq, because uniq only collapses adjacent duplicates.

B. wc

wc (word count) reports the size of the stream.

• Three counts: default output is lines words bytes, e.g. wc file → 12 84 560 file.
• Individual flags: -l lines only, -w words only, -c bytes, -m characters.
• Idiom: ls | wc -l counts entries; grep -c is preferred over grep ... | wc -l for counting matches.

C. uniq

uniq collapses or reports consecutive identical lines.

• Adjacency rule: only neighbouring duplicates are affected, hence the usual sort | uniq.
• Counting: -c prefixes each line with its occurrence count.
• Filtering: -d prints only duplicated lines; -u prints only unique (non-repeated) lines.
• Worked example:
  
  BASH

    sort words.txt | uniq -c | sort -rn

  
  This produces a frequency table sorted from most to least common word.

V. Character Translation — tr

tr translates, squeezes, or deletes individual characters read from stdin; it takes no filename argument and works only through redirection or a pipe.

A. tr

• Translate sets: tr 'a-z' 'A-Z' maps lowercase to uppercase character by character.
• Delete: tr -d '0-9' removes all digits from the stream.
• Squeeze: -s compresses runs of a character — tr -s ' ' turns multiple spaces into one, repairing input for cut.
• Complement: -c targets characters not in the set — tr -cd 'a-zA-Z\n' keeps only letters and newlines.
• Classes: POSIX names such as [:alpha:], [:digit:], [:space:] may replace explicit ranges.

VI. Filters with Pipes

A pipeline chains filters so that each stage refines the previous stage's output, avoiding intermediate files.

• Mechanism: cmd1 | cmd2 — the shell connects cmd1's stdout to cmd2's stdin and runs both concurrently.
• Left-to-right refinement: data flows one direction; each command is a transformation applied in sequence.
• Design order: filter early to shrink the stream (grep first), then reshape (cut), then order (sort), then summarise (uniq -c, wc).
• Worked example — top three IP addresses in a log:
  
  BASH

    cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -n 3

  • cut isolates the IP field, sort groups equal IPs, uniq -c counts each, the second sort -rn ranks by count, head keeps the top three.
• Redirection vs pipe: > sends output to a file; | sends it to another command — pipes never touch disk, so they are faster for chained work.

VII. Regular Expression Fundamentals

A regular expression (regex) is a pattern language describing sets of strings; filters like grep, sed, and tr's cousins match input against it.

A. Literals and Metacharacters

• Literals: ordinary characters match themselves — cat matches the substring cat.
• Metacharacters: . ^ $ * [ ] \ carry special meaning and must be escaped with \ to match literally.
• Any character: . matches exactly one character of any kind except newline.

B. Anchors

• Line start: ^ anchors the match to the beginning — ^From matches lines starting with From.
• Line end: $ anchors to the end — done$ matches lines ending in done.
• Empty-line test: ^$ matches a blank line.

C. Character Classes

• Bracket set: [aeiou] matches any one listed character; [0-9] matches a digit by range.
• Negation: [^0-9] matches any character that is not a digit.
• POSIX classes: [[:upper:]], [[:alnum:]] name portable sets independent of locale ranges.

D. Quantifiers

• Zero or more: * repeats the preceding item any number of times — ab* matches a, ab, abbb.
• One or more / optional (ERE): + requires at least one, ? makes the item optional.
• Bounded (ERE): {2,4} matches between two and four repetitions.

E. Grouping and Alternation

• Group: parentheses bind a subpattern — ERE (ab)+, BRE \(ab\)\+.
• Alternation: | offers choices — ERE red|green|blue.
• Greedy match: quantifiers consume as much as possible, so <.*> on <a><b> matches the whole span, not just <a>.