Unit 5: Shell Features - Subjective Questions

Pattern Matching refers to the shell's ability to match filenames against special characters (called wildcards or metacharacters) so that a single command can operate on multiple files at once.

Why wildcards are useful:

• Efficiency: A single command can act on dozens of files instead of typing each name.
• Flexibility: You can select files based on partial names, extensions, or character ranges.
• Automation: Scripts can process groups of files dynamically.

Common Wildcards:

• * — matches zero or more characters.
• ? — matches exactly one character.
• [...] — matches any one character from the enclosed set.

Example:

ls *.txt        # lists all files ending in .txt
rm file?.log     # removes file1.log, fileA.log, etc.

The shell performs this matching (called globbing) before passing the expanded names to the command.

The * (asterisk) Wildcard:

• Matches zero or more characters (any string, including an empty one).
• Does not match a leading dot in hidden files by default.

Example:

ls a*      # matches a, ab, abc, apple, etc.
ls *.c     # matches all files ending in .c

The ? (question mark) Wildcard:

• Matches exactly one character.

Example:

ls file?.txt   # matches file1.txt, fileA.txt (single char)
ls ??.sh       # matches any 2-character name ending in .sh

Key Difference:

Thus ? is more restrictive since it requires a character in that exact position, while * is flexible in length.

Character Classes ([...]):

The square brackets match any one character from the enclosed set.

1. Simple Set:

ls file[123].txt   # matches file1.txt, file2.txt, file3.txt

2. Ranges: A hyphen defines a range.

ls [a-z]*      # files starting with any lowercase letter
ls file[0-9]   # file0 to file9
ls [A-Za-z]*   # files starting with any letter

3. Negation: A ! or ^ at the start negates the set.

ls [!0-9]*     # files NOT starting with a digit

4. POSIX Character Classes:

ls [[:digit:]]*   # files starting with a digit
ls [[:upper:]]*   # files starting with an uppercase letter

Key Points:

• Matches exactly one character position.
• Ranges depend on the locale's collating order.
• Very useful for selecting files by a specific character group.

Although both use special symbols, wildcards (globbing) and regular expressions (regex) serve different purposes and behave differently.

Example of difference:

• Wildcard: ls a* → filenames starting with a.
• Regex: grep 'a*' file → matches lines with zero or more as.

Summary: Wildcards operate on filenames and are expanded by the shell; regex operates on text content and is interpreted by individual programs.

I/O Redirection is a shell feature that changes the default sources and destinations of a command's input and output streams, allowing them to be read from or written to files instead of the terminal.

The Three Standard File Descriptors:

• Standard Input (stdin) — descriptor 0: Default source of input, normally the keyboard.
• Standard Output (stdout) — descriptor 1: Default destination of normal output, normally the terminal screen.
• Standard Error (stderr) — descriptor 2: Destination for error messages, also normally the terminal.

Why redirection is useful:

• Save command output to a file.
• Supply input from a file instead of typing.
• Separate normal output from error messages.

Example:

sort < data.txt        # stdin from file
ls > list.txt          # stdout to file
gcc prog.c 2> err.txt   # stderr to file

Each descriptor can be independently redirected, giving fine control over data flow.

Output redirection sends the standard output of a command to a file instead of the terminal.

1. The > Operator (Overwrite):

• Redirects stdout to a file.
• Creates the file if it does not exist.
• Overwrites (truncates) the file if it already exists.

ls > files.txt      # creates or replaces files.txt
date > log.txt      # log.txt now contains only the date

2. The >> Operator (Append):

• Redirects stdout to a file.
• Creates the file if it does not exist.
• Appends to the end of the file if it exists (existing content preserved).

date >> log.txt     # adds date to end of log.txt
echo "Done" >> log.txt

Key Difference:

Caution: > can cause data loss if used on an important existing file.

Input Redirection (<):

Redirects the standard input of a command so it reads from a file instead of the keyboard.

wc -l < data.txt     # counts lines by reading data.txt as stdin
sort < names.txt     # sorts contents of names.txt

Here the command does not receive the filename as an argument — the shell connects the file to stdin.

Here-Document (<<):

Allows you to supply multiple lines of inline input directly within a script or command until a delimiter word is reached.

cat << END
Hello World
This is a here-document.
END

Everything between << END and the END marker is fed as stdin.

Difference:

Here-String (<<<): A related form feeds a single string:

wc -w <<< "count these words"

Standard Error (stderr) uses file descriptor 2. By default it appears on the terminal even when stdout is redirected.

Redirecting stderr alone:

gcc prog.c 2> errors.txt     # only error messages go to file
command 2>> errors.txt        # append errors

Redirecting stderr to a null device (discard):

command 2> /dev/null

Redirecting BOTH stdout and stderr to the same file:

Method 1 — traditional (order matters):

command > out.txt 2>&1

Here 2>&1 means redirect descriptor 2 to wherever descriptor 1 currently points (the file). The order is important: stdout must be redirected first.

Method 2 — Bash shorthand:

command &> out.txt      # both stdout and stderr
command &>> out.txt     # append both

Common Mistake:

command 2>&1 > out.txt   # WRONG order: stderr still goes to terminal

This is because 2>&1 is applied before stdout is redirected to the file.

Pipe (|):

A pipe is a shell mechanism that connects the standard output of one command directly to the standard input of another, allowing commands to be chained so that data flows from one to the next without intermediate temporary files.

Syntax:

command1 | command2

Working:

• The shell creates an in-memory buffer (the pipe).
• command1's stdout is connected to this buffer.
• command2's stdin reads from this buffer.
• Both commands run concurrently.

Example:

ls -l | grep ".txt"

Here the long listing produced by ls -l is fed to grep, which filters lines containing .txt.

Another Example:

cat file.txt | wc -l    # count lines in file.txt

Advantages:

• No temporary files needed.
• Efficient use of memory.
• Encourages the Unix philosophy of small, single-purpose tools working together.

Pipe Chaining connects several commands using multiple | operators so that output flows through a sequence of processing stages, each transforming the data before passing it on.

Syntax:

cmd1 | cmd2 | cmd3 | cmd4

Data Flow:

• cmd1 output → cmd2 input
• cmd2 output → cmd3 input
• cmd3 output → cmd4 input
• All commands run simultaneously, forming a data-processing pipeline.

Example:

cat /etc/passwd | grep "/bin/bash" | cut -d: -f1 | sort | wc -l

Stage-by-stage explanation:

• cat /etc/passwd — outputs the whole file.
• grep "/bin/bash" — keeps lines of bash users.
• cut -d: -f1 — extracts the username field.
• sort — sorts the usernames.
• wc -l — counts the number of such users.

Advantages:

• Complex tasks built from simple tools.
• Readable, modular command construction.
• No intermediate files required.

Both pipes and redirection control data flow, but they connect to different endpoints.

When to use Redirection:

• To save output for later use.
• To read input from an existing file.
• To log errors or output.

When to use Pipes:

• To process output of one command with another immediately.
• To build multi-stage data pipelines.
• When no permanent storage is needed.

Combined Example:

sort < names.txt | uniq > result.txt

Here redirection reads input and writes final output, while the pipe processes data in between.

The tee Command:

tee reads from standard input and writes simultaneously to standard output and to one or more files. Its name comes from the T-shaped plumbing joint that splits a flow.

Syntax:

command | tee file.txt

Working:

• Data coming through the pipe is written to file.txt.
• The same data is also passed to the terminal (or the next command).

Example 1 — Save and display:

ls -l | tee listing.txt

Shows the listing on screen and saves it to listing.txt.

Example 2 — Append mode (-a):

date | tee -a log.txt

Example 3 — In the middle of a pipeline:

cat data.txt | tee raw.txt | grep "error" | wc -l

Here tee saves the raw data while the pipeline continues processing.

Uses:

• Logging intermediate results.
• Debugging pipelines.
• Writing to a file and screen at once.

Filename Expansion (Globbing):

This is the process by which the shell replaces wildcard patterns with the matching filenames before the command actually runs.

Steps the Shell Follows:

1. Scan the command line for wildcard characters (*, ?, [...]).
2. Search the relevant directory for filenames matching the pattern.
3. Replace the pattern with the sorted list of matching filenames.
4. Pass the expanded list as arguments to the command.
5. If no match is found, most shells leave the pattern unchanged (in Bash by default).

Example:

rm *.tmp

• Shell finds a.tmp b.tmp c.tmp.
• Command becomes rm a.tmp b.tmp c.tmp.

Important Notes:

• The command never sees the wildcards — it only sees expanded names.
• Hidden files (starting with .) are not matched by * unless explicitly stated.
• Quoting a pattern (e.g., "*.txt") prevents expansion.
• The nullglob option makes unmatched patterns expand to nothing.

Sometimes we want the shell to treat wildcard characters literally instead of expanding them.

Why disable expansion?

• To pass a pattern to a program (e.g., find, grep) that will interpret it itself.
• To use special characters as literal text.
• To prevent unexpected matches.

Methods to Prevent Expansion:

1. Single Quotes (' '): Strongest — nothing inside is expanded.

echo '*.txt'      # prints *.txt literally

2. Double Quotes (" "): Prevents globbing (but allows variable expansion).

echo "*.txt"      # prints *.txt

3. Backslash (\): Escapes a single character.

echo \*           # prints *

4. set -f / set -o noglob: Disables globbing for the whole session.

set -f
ls *.txt          # * treated literally
set +f            # re-enable

Example use with find:

find . -name "*.c"   # quotes let find interpret the pattern itself

Quoting ensures the pattern reaches find intact rather than being expanded by the shell.

1. List all .log files:

ls *.log

The * wildcard matches any name ending in .log.

2. Count how many .c files exist:

ls *.c | wc -l

• ls *.c lists all C files (one per line via pipe).
• wc -l counts the lines = number of files.

3. Save a sorted, unique list of users:

cut -d: -f1 /etc/passwd | sort | uniq > users.txt

• cut extracts usernames.
• sort orders them.
• uniq removes duplicates.
• > saves the result to users.txt.

4. Find the 5 largest files in a directory:

ls -lS | head -6

• ls -lS sorts files by size (largest first).
• head -6 shows the top lines (1 header + 5 files).

Alternative using du:

du -ah . | sort -rh | head -5

These examples combine wildcards, pipes, and redirection to solve real tasks.

Internal Working of a Pipe:

When you run cmd1 | cmd2, the shell performs several low-level operations to connect the two processes.

Steps Performed by the Shell:

1. Create a pipe using the pipe() system call. This returns two file descriptors:
   • A read end (fd[0])
   • A write end (fd[1])
2. Fork two child processes (one per command).
3. For cmd1: connect its stdout (fd 1) to the write end of the pipe using dup2().
4. For cmd2: connect its stdin (fd 0) to the read end of the pipe.
5. Close unused pipe ends in each process.
6. Execute each command via exec().
7. Both commands run concurrently; data flows through a kernel buffer.

Diagram-style Flow:

+--------+ stdout [ PIPE BUFFER ] stdin +--------+
| cmd1 | ---------> | kernel memory | ------> | cmd2 |
+--------+ +---------------+ +--------+

Key Points:

• The pipe buffer is maintained in kernel memory, typically a few kilobytes.
• If the buffer fills, cmd1 blocks until cmd2 reads (flow control).
• If cmd2 finishes early, cmd1 may receive a SIGPIPE.
• No disk file is created, making pipes fast and memory-efficient.

This mechanism embodies the Unix philosophy of composing small tools into powerful pipelines.

These operators control where output and error streams are directed.

Detailed Explanation:

• > — Normal output only; existing content is lost.

date > log.txt

• >> — Normal output appended; preserves old data.

date >> log.txt

• 2> — Only error messages (descriptor 2).

gcc a.c 2> errors.txt

• &> — Bash shortcut for both streams.

make &> build.log

• 2>&1 — Send stderr wherever stdout points. Order matters:

cmd > file.txt 2>&1   # both to file
cmd 2>&1 > file.txt   # stderr to terminal (wrong order)

Summary: 1 = stdout, 2 = stderr; > overwrites, >> appends, and & groups both streams.

Brace Expansion ({...}):

Brace expansion generates a set of strings by combining a fixed prefix/suffix with the comma-separated (or range) items inside the braces. It happens before filename expansion and does not require files to exist.

Examples:

echo file{1,2,3}.txt     # file1.txt file2.txt file3.txt
echo {a..e}              # a b c d e (sequence)
echo {1..5}              # 1 2 3 4 5
mkdir project/{src,bin,doc}   # creates three directories

Character Class ([...]):

This is a wildcard that matches a single existing filename character.

ls file[123].txt   # matches only EXISTING file1.txt etc.

Key Differences:

Summary: Brace expansion creates text combinations, while character classes filter existing filenames during globbing.

Redirection and pipes are powerful but error-prone. Below are frequent pitfalls and their solutions.

1. Accidental Overwrite with >:

sort data.txt > data.txt   # WRONG: truncates file before reading

• The shell empties data.txt first, losing the data.
• Fix: Use a temp file or sort -o data.txt data.txt.

2. Wrong order of 2>&1:

cmd 2>&1 > out.txt   # stderr still goes to terminal

• Fix: Put 2>&1 after the stdout redirect: cmd > out.txt 2>&1.

3. Piping to commands that don't read stdin:

ls | cd /tmp   # cd ignores piped input

• Fix: Use xargs when a command expects arguments: ls | xargs rm.

4. Using > when append >> is needed:

• Overwrites logs. Fix: Use >> for logging.

5. Unquoted filenames with spaces:

• Wildcards may break on spaces. Fix: Quote variables and use find ... -print0 | xargs -0.

6. Broken pipe (SIGPIPE): Occurs when the reader exits early (e.g., | head). Usually harmless.

Best Practices:

• Never redirect a file to itself.
• Remember descriptor order.
• Use tee to inspect intermediate data.
• Quote patterns to control expansion.

The Pipeline:

cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -3

Stage-by-Stage Explanation:

1. cat access.log — Outputs the entire log file to the pipeline. (Could also be replaced by awk '{print $1}' access.log directly.)

2. awk '{print $1}' — Extracts the first field of each line, which in a standard access log is the client IP address.

3. sort — Sorts the IP addresses so that identical IPs become adjacent. This is required before counting duplicates.

4. uniq -c — Collapses adjacent duplicate lines and prefixes each with a count of occurrences.

5. sort -rn — Sorts numerically (-n) in reverse (-r) so the highest counts come first.

6. head -3 — Displays only the top 3 lines (the three most frequent IPs).

Sample Output:

152 192.168.1.10
98 10.0.0.5
45 172.16.0.2

Why this works well:

• Each tool does one job (Unix philosophy).
• No temporary files are created.
• The pipeline is easily modified, e.g., change head -3 to head -10 for the top 10.

This demonstrates the power of pipe chaining combined with text-processing utilities.