A filter is a program that takes its input from the standard input (stdin), performs some transformation or operation on the data, and sends the result to the standard output (stdout).

How filters process data:

• They read data line by line (or stream) from stdin.
• They process/transform the data according to their function.
• They write the processed output to stdout, which can be redirected or piped.

Key characteristics:

• Filters can be connected using pipes (|) to build powerful command chains.
• They follow the Unix philosophy: do one thing and do it well.

Three common examples:

• grep — filters lines matching a pattern.
• sort — arranges lines in order.
• wc — counts lines, words, and characters.

Example:
bash
cat file.txt | grep "error" | sort

Here grep and sort act as filters in a pipeline.

grep (Global Regular Expression Print) searches for a pattern in one or more files and prints the matching lines.

Syntax:
bash
grep [options] pattern [file...]

Important options:

• -i : Ignore case while matching.
  bash
  grep -i "hello" file.txt

• -c : Count the number of matching lines.
  bash
  grep -c "error" log.txt

• -n : Display matching lines with their line numbers.
  bash
  grep -n "main" program.c

• -v : Invert match; display lines that do not match.
  bash
  grep -v "debug" log.txt

• -l : List only the names of files containing the match.
  bash
  grep -l "TODO" *.txt

• -w : Match whole words only.
  bash
  grep -w "is" file.txt

Note: grep returns matching lines to stdout, making it ideal for use in pipelines.

These three variants differ in the type of pattern matching they support:

Extended grep (egrep or grep -E):

• Supports metacharacters +, ?, |, {}, () without escaping.
• Use it when you need complex patterns such as alternation or grouping.

Example:
bash

Extended grep — match lines containing 'cat' or 'dog'

egrep "cat|dog" animals.txt

Equivalent basic grep needs escaping

grep "cat|dog" animals.txt

When to use Extended grep: Whenever the search requires alternation (|), one-or-more (+), optional (?), or grouping () — Extended grep makes patterns cleaner and more readable.

The cut command extracts selected portions (columns or characters) from each line of input.

Syntax:
bash
cut [options] [file]

Important options:

• -c : Cut by character position.
  bash
  cut -c1-5 file.txt # first 5 characters of each line

• -f : Cut by field number.

• -d : Specify the field delimiter (default is TAB).
  bash
  cut -d":" -f1 /etc/passwd # extract usernames

Extracting character ranges:

• cut -c1-5 → characters 1 to 5
• cut -c3- → from character 3 to end
• cut -c-4 → from start to character 4

Extracting columns (fields):
bash
cut -d"," -f2,4 data.csv # extract 2nd and 4th fields

Example: To get the shell of each user:
bash
cut -d":" -f1,7 /etc/passwd

Note: cut works well when data is neatly delimited but cannot handle multiple/variable spaces as a single delimiter (use awk for that).

head displays the beginning portion of a file, while tail displays the ending portion.

head command:
bash
head [options] [file]

• By default shows the first 10 lines.

• -n : specify number of lines.
  bash
  head -n 5 file.txt # first 5 lines

• -c : specify number of bytes.

tail command:
bash
tail [options] [file]

• By default shows the last 10 lines.

• -n : specify number of lines.
  bash
  tail -n 15 file.txt # last 15 lines

• +n : start from line n to end.
  bash
  tail -n +5 file.txt # from line 5 onward

Monitoring a growing log file:
The -f (follow) option keeps the file open and displays new lines as they are appended:
bash
tail -f /var/log/syslog

This is extremely useful for real-time log monitoring. Combine with grep:
bash
tail -f access.log | grep "404"

The sort command arranges the lines of text in a specified order (default is ASCII/alphabetical ascending).

Syntax:
bash
sort [options] [file]

Important options:

• -r : Reverse (descending) order.
  bash
  sort -r names.txt

• -n : Numeric sort (treats values as numbers).
  bash
  sort -n marks.txt

• -k : Sort by a specific key/field.
  bash
  sort -k2 data.txt # sort by 2nd field

• -t : Specify field separator.
  bash
  sort -t":" -k3 -n /etc/passwd # sort by UID numerically

• -u : Remove duplicate lines after sorting.

• -f : Case-insensitive sort.

Examples:
bash

Numeric sort

sort -n numbers.txt

Reverse numeric sort

sort -nr numbers.txt

Sort CSV by 3rd column numerically

sort -t"," -k3 -n students.csv

Note: Without -n, sort compares numbers as strings, so 10 would come before 2.

The wc (word count) command counts the number of lines, words, and characters/bytes in its input.

Syntax:
bash
wc [options] [file]

Default output (three numbers): lines words characters filename
bash
wc file.txt

12 45 310 file.txt

Important options:

• -l : Count only lines.
  bash
  wc -l file.txt

• -w : Count only words.
  bash
  wc -w file.txt

• -c : Count bytes.

• -m : Count characters.

• -L : Length of the longest line.

Usage in pipelines:
bash

Count number of users on the system

cat /etc/passwd | wc -l

Count files in a directory

ls | wc -l

Count matching lines

grep "error" log.txt | wc -l

Note: When multiple files are given, wc prints counts per file and a total line at the end.

The tr (translate) command translates, deletes, or squeezes characters read from standard input. It works only on stdin (does not accept a filename directly).

Syntax:
bash
tr [options] SET1 [SET2]

1. Translating characters:
bash

Convert lowercase to uppercase

tr 'a-z' 'A-Z' < file.txt

echo "hello" | tr 'a-z' 'A-Z' # HELLO

2. Deleting characters (-d):
bash

Remove all digits

echo "abc123" | tr -d '0-9' # abc

3. Squeezing repeated characters (-s):
bash

Compress multiple spaces into one

echo "hello world" | tr -s ' ' # hello world

4. Complement set (-c):
bash

Keep only alphabets, replace rest with newline

tr -cs 'a-zA-Z' '\n' < file.txt

Common uses:

• Case conversion.
• Removing unwanted characters (e.g., carriage returns tr -d '\r').
• Squeezing whitespace.

Note: tr operates on characters, not words or patterns.

The uniq command filters out adjacent duplicate lines from sorted input.

Syntax:
bash
uniq [options] [file]

Why sort before uniq:

• uniq only removes consecutive duplicate lines.
• If duplicates are scattered throughout the file, they will not be detected.
• Therefore, we sort the data first so identical lines become adjacent.
  bash
  sort file.txt | uniq

Important options:

• -c : Prefix each line with its count of occurrences.
  bash
  sort file.txt | uniq -c

• -d : Display only duplicate lines.
  bash
  sort file.txt | uniq -d

• -u : Display only unique (non-repeated) lines.
  bash
  sort file.txt | uniq -u

• -i : Case-insensitive comparison.

Example — frequency count of words:
bash
cat words.txt | sort | uniq -c | sort -nr

This lists each word with its count, sorted by most frequent.

A pipe (|) is a mechanism that connects the standard output of one command to the standard input of another command, allowing data to flow through a chain of commands.

Concept:
bash
command1 | command2 | command3

• Output of command1 becomes input of command2, and so on.
• No temporary files are needed; data flows in memory.

Why pipes are powerful:

• Each filter does one job well; combining them solves complex problems.
• Follows the Unix philosophy of building small, composable tools.

Examples:

1. Count logged-in users:
bash
who | wc -l

2. Find the top 5 largest files:
bash
ls -l | sort -k5 -nr | head -5

3. Find most frequent words:
bash
cat file.txt | tr ' ' '\n' | sort | uniq -c | sort -nr | head

4. Unique error messages:
bash
grep "error" log.txt | sort | uniq

Note: Pipes greatly reduce the need for intermediate files and enable one-line solutions to complex text-processing tasks.

A Regular Expression (regex) is a sequence of characters that defines a search pattern used for matching, searching, and manipulating text.

Fundamental building blocks:

• Literals: Ordinary characters that match themselves. e.g., cat matches the string cat.

• Metacharacters: Special characters with meaning:

  • . — matches any single character. c.t matches cat, cot, cut.
  • * — matches zero or more of the preceding character. ab* matches a, ab, abb.
  • ^ — anchors match to the start of a line. ^The matches lines starting with The.
  • $ — anchors match to the end of a line. end$ matches lines ending with end.

• Character classes [ ]: Match any one character inside brackets.

  • [aeiou] matches any vowel.
  • [0-9] matches any digit.
  • [^0-9] matches any non-digit.

Example:
bash
grep "^[A-Z].*.$" file.txt

Matches lines that start with a capital letter and end with a period.

Uses: Text searching (grep), stream editing (sed), and pattern-based programming.

Anchors and quantifiers are key components of regular expressions that control where and how many times a pattern matches.

Anchors (position matching):

• ^ — Start of line. ^abc matches lines beginning with abc.
• $ — End of line. xyz$ matches lines ending with xyz.
• \b — Word boundary (in ERE/PCRE). \bcat\b matches the whole word cat.

Quantifiers (repetition):

• * — Zero or more of the preceding element. bo* matches b, bo, boo.
• + — One or more (ERE). bo+ matches bo, boo but not b.
• ? — Zero or one (optional). colou?r matches color and colour.
• {n} — Exactly n times. a{3} matches aaa.
• {n,} — At least n times. a{2,} matches aa, aaa, ...
• {n,m} — Between n and m times. a{2,4} matches aa to aaaa.

Examples:
bash

Lines that are empty

grep "^$" file.txt

Match a word with 5 to 8 letters (ERE)

egrep "[a-z]{5,8}" file.txt

Note: +, ?, and {} require escaping in Basic Regular Expressions but work directly in Extended (egrep/grep -E).

Basic Regular Expressions (BRE) and Extended Regular Expressions (ERE) are two standards for defining patterns; they differ mainly in how metacharacters are treated.

Key differences:

• In BRE, characters like +, ?, |, (), {} are treated as literals unless escaped with a backslash.
• In ERE, these are treated as metacharacters by default.

Examples:
bash

BRE — match 'cat' or 'dog'

grep "cat|dog" file.txt

ERE — same match

grep -E "cat|dog" file.txt

Conclusion: ERE makes complex patterns more readable, which is why egrep/grep -E is preferred for advanced matching.

A character class matches any one character from a defined set. It is enclosed in square brackets [ ].

Basic character classes:

• [abc] — matches a, b, or c.
• [a-z] — matches any lowercase letter (range).
• [A-Za-z0-9] — matches any alphanumeric character.
• [^0-9] — negated class; matches any character that is NOT a digit.

POSIX character classes (portable, locale-aware named classes used inside [[ ]]):

Examples:
bash

Match lines containing a digit

grep "[[:digit:]]" file.txt

Match words with only alphabets

egrep "^[[:alpha:]]+$" file.txt

Advantage of POSIX classes: They are locale-independent and more readable than raw ranges.

grep combined with regular expressions is a powerful tool for pattern-based text searching. Below are practical real-world examples.

1. Find lines starting with a specific word:
bash
grep "^Error" logfile.txt

2. Find blank lines:
bash
grep "^$" file.txt

3. Match email-like patterns (ERE):
bash
grep -E "[a-zA-Z0-9._]+@[a-zA-Z]+.[a-z]+" contacts.txt

4. Find IP-address-like patterns:
bash
grep -E "[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}" access.log

5. Search for words with exactly 3 letters:
bash
grep -E "\b[a-zA-Z]{3}\b" file.txt

6. Find lines containing digits:
bash
grep "[0-9]" data.txt

7. Case-insensitive whole-word search:
bash
grep -iw "linux" notes.txt

Explanation: Regular expressions let grep go beyond fixed strings, enabling matching of structured data like emails, IPs, dates, and codes. Combining with options (-i, -w, -n, -E) increases flexibility.

These tasks demonstrate the power of combining filters using pipes.

(a) Top 3 users consuming the most disk space:
bash
du -sh /home/* | sort -rh | head -3

• du -sh /home/* → shows size of each user's directory.
• sort -rh → sorts human-readable sizes in reverse (largest first).
• head -3 → takes the top 3 entries.

(b) Count the number of unique words in a file:
bash
tr ' ' '\n' < file.txt | sort | uniq | wc -l

• tr ' ' '\n' → converts each word onto a separate line.
• sort → makes duplicate words adjacent.
• uniq → removes duplicates.
• wc -l → counts the unique words.

(c) 5 most frequently occurring lines in a log file:
bash
sort log.txt | uniq -c | sort -nr | head -5

• sort → groups identical lines together.
• uniq -c → counts occurrences of each line.
• sort -nr → sorts by count (descending).
• head -5 → shows the top 5.

Conclusion: Each pipeline chains simple filters to solve a non-trivial task efficiently without writing a program.

In Unix/Linux, every process is associated with three default I/O streams, each identified by a file descriptor:

1. Standard Input (stdin, 0): The source of data for a command; by default the keyboard.

2. Standard Output (stdout, 1): Where normal output is written; by default the screen.

3. Standard Error (stderr, 2): Where error messages are written; kept separate from stdout so errors don't mix with valid data.

How filters use these streams:

• A filter reads from stdin, processes data, and writes to stdout.
• Example: sort reads lines from stdin and outputs sorted lines to stdout.

How pipes use these streams:

• A pipe | connects stdout of one command to stdin of the next.
  bash
  grep "error" log.txt | sort | uniq

Redirection examples:
bash
command > out.txt # redirect stdout
command 2> err.txt # redirect stderr
command < in.txt # redirect stdin
command > all.txt 2>&1 # merge stderr into stdout

Note: Separating stderr from stdout allows error messages to be handled independently in pipelines.

A word frequency report lists each distinct word along with the number of times it appears, usually sorted by frequency. This can be built entirely with filters and pipes.

Complete pipeline:
bash
cat file.txt | tr -s ' ' '\n' | tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr

Stage-by-stage explanation:

• cat file.txt — Sends the file contents to stdout.

• tr -s ' ' '\n' — Translates spaces into newlines and squeezes multiple spaces, placing each word on its own line.

• tr 'A-Z' 'a-z' — Converts everything to lowercase so The and the are treated as the same word.

• sort — Sorts the words alphabetically so identical words become adjacent (required for uniq).

• uniq -c — Removes duplicates and prefixes each word with its count.

• sort -nr — Sorts numerically in reverse, placing the most frequent words at the top.

Sample output:

 15 the
  9 and
  7 is

Counting total unique words:
bash
tr -s ' ' '\n' < file.txt | sort | uniq | wc -l

Here wc -l counts how many unique words exist.

Conclusion: This demonstrates how small filters, chained together, solve a real text-analysis problem elegantly.

In regular expressions, some characters have special meaning (metacharacters). To match them as literal characters, they must be escaped using a backslash (\).

Common metacharacters that need escaping to be literal:
. * ^ $ [ ] \ + ? ( ) { } |

Why escaping is important:

• Without escaping, a metacharacter is interpreted by its special function, not as itself.
• Example: . means any character. To match an actual dot (period), you must write \..

Examples:
bash

Match a literal dot (e.g., a file extension)

grep "file.txt" list.txt

Without escaping, this matches 'fileXtxt', 'file1txt', etc.

grep "file.txt" list.txt

Matching a literal $ or *:
bash
grep "$100" prices.txt # matches the string $100
grep "5 * 3" math.txt # matches 5 * 3

Escaping in BRE vs ERE (reversed meaning):

• In BRE, \+ means one or more (metacharacter), while + is literal.
• In ERE, + means one or more, while \+ is literal.

Conclusion: Escaping controls whether a character behaves as a metacharacter or a literal, and misusing it is a common source of pattern-matching errors.

Given a colon-delimited file students.txt with fields RollNo:Name:Marks:Branch, we use cut, sort, and grep with pipes.

Sample data:

101:Aman:88:CSE
102:Bhavna:76:ECE
103:Chetan:91:CSE

(a) Extract only names (field 2):
bash
cut -d":" -f2 students.txt

• -d":" sets the delimiter to colon.
• -f2 selects the 2nd field (Name).

(b) Display records sorted by marks (field 3) in descending order:
bash
sort -t":" -k3 -nr students.txt

• -t":" sets colon as the field separator.
• -k3 sorts on the 3rd field (Marks).
• -n numeric sort, -r reverse (descending).

(c) Count students in the CSE branch:
bash
grep -c ":CSE$" students.txt

• -c counts matching lines.
• :CSE$ matches records ending with :CSE (branch field), avoiding false matches.

Alternative using pipes:
bash
cut -d":" -f4 students.txt | grep -c "CSE"

Conclusion: By combining field extraction (cut), sorting (sort with keys), and pattern matching (grep), we can perform database-like queries directly on text files.