Unit 6: File Handling, Data Loading, and Visualization

I. Foundations of File-Based Data Processing

File handling connects a Python program to persistent data stored outside working memory. A typical workflow opens or obtains a file, decodes and loads its contents, selects useful observations, and communicates patterns through visualizations.

• File object: open() returns an object through which a program reads, writes, or navigates a file.
• Path: A path identifies a file-system location; it may be relative, such as data/sales.csv, or absolute, such as /home/user/data/sales.csv.
• Access mode: Modes such as "r", "w", "a", and "b" determine permitted operations and whether data is treated as text or bytes.
• Encoding: Text files store encoded bytes; encoding="utf-8" provides an explicit and widely supported decoding convention.
• Resource management: Files and network responses should be closed promptly, preferably through a with statement.
• Data pipeline:
  1. Acquire data from a local file, command line, or Internet resource.
  2. Load it into Python objects or a tabular structure.
  3. Clean, select, and filter observations.
  4. Visualize relevant variables.
• Core libraries:
  • pathlib, os, and sys support file-system and program interaction.
  • csv handles delimited text.
  • pandas loads and manipulates tables.
  • matplotlib and seaborn create visualizations.

II. File Input and Output — Persistent Text Data

A. Reading files

Reading a file transfers stored content into Python in a form the program can process.

• Opening: open(path, mode, encoding) commonly uses mode "r" for text input.

file = open("notes.txt", "r", encoding="utf-8")
text = file.read()
file.close()

• Reading methods:
  • read() returns all remaining content as one string.
  • read(n) reads at most n characters in text mode.
  • readline() returns one line, usually retaining "\n".
  • readlines() returns a list of lines.
• Iteration: Iterating over the file is memory-efficient because it processes one line at a time.

with open("notes.txt", encoding="utf-8") as file:
    for line in file:
        print(line.rstrip())

• File position: tell() reports the current stream position, while seek(offset) moves it where the stream supports seeking.
• Common failures: A missing path raises FileNotFoundError; invalid decoding can raise UnicodeDecodeError.

B. Writing files

Writing converts Python values to text or bytes and stores them in a file.

• Write modes:
  1. Replace: "w" creates a file or truncates an existing file.
  2. Append: "a" adds data after existing content.
  3. Exclusive creation: "x" raises FileExistsError if the target exists.
• Methods: write(string) writes one string and returns its character count; writelines(iterable) writes strings without automatically adding separators.

lines = ["Alice,82\n", "Ben,91\n"]
with open("scores.txt", "w", encoding="utf-8") as file:
    file.writelines(lines)

• Value conversion: Non-string values require conversion, as in file.write(str(total)).
• Buffering: Data may remain buffered until flush() or close(); normal context-manager exit closes and flushes the file.
• Safety: Writing to a temporary file and then replacing the destination reduces the chance of leaving a partially written result.

C. with statement

The with statement manages a resource’s setup and cleanup through the context-management protocol.

• Automatic cleanup: A file’s __enter__() supplies the resource, and __exit__() closes it when the block ends.
• Exception safety: Closure occurs even if code inside the block raises an exception.

with open("report.txt", "w", encoding="utf-8") as report:
    report.write("Completed\n")

• Scope: The variable remains defined afterward, but the file is closed; subsequent I/O raises ValueError.
• Multiple resources: Several context managers can appear in one statement.

with open("input.txt", encoding="utf-8") as src, \
     open("copy.txt", "w", encoding="utf-8") as dst:
    dst.write(src.read())

III. Files, Programs, and External Resources

A. Interacting with file systems

File-system interaction includes constructing paths, inspecting entries, creating directories, and moving or deleting files.

• Path objects: pathlib.Path provides platform-aware operations instead of manual slash concatenation.

from pathlib import Path

folder = Path("data")
path = folder / "sales.csv"
folder.mkdir(parents=True, exist_ok=True)
print(path.exists(), path.suffix)

• Inspection: exists(), is_file(), and is_dir() test path state; stat() exposes metadata such as byte size.
• Discovery: iterdir() lists direct children, while glob("*.csv") and rglob("*.csv") match files by pattern.
• Modification: rename() or replace() moves a path, and unlink() removes a file.
• Working directory: Path.cwd() returns the current directory, which is the base for relative paths.
• Portability: Path objects adapt separators to Windows, macOS, and Linux conventions.

B. Binary data

Binary mode reads and writes raw bytes rather than encoded text.

• Modes: "rb", "wb", and "ab" combine byte-oriented access with read, write, or append behavior.
• Byte values: A bytes object is immutable and contains integers from 0 through 255; bytearray is mutable.

payload = bytes([80, 89, 84, 72, 79, 78])
with open("tag.bin", "wb") as file:
    file.write(payload)

• Text conversion: "Python".encode("utf-8") produces bytes, while data.decode("utf-8") reconstructs text.
• Structured binary values: The struct module packs numbers according to a declared format; for example, struct.pack("<I", 300) stores an unsigned 32-bit integer in little-endian order.
• Applications: Images, audio, compressed archives, executables, and serialized records require binary-safe handling.
• Caution: Binary formats need specifications; arbitrary byte boundaries do not necessarily correspond to complete records.

C. Command-line arguments and files

Command-line arguments allow filenames and processing options to be supplied when a program starts.

• Basic access: sys.argv[0] is the script name, and later elements are argument strings.

# Run as: python count.py notes.txt
import sys
from pathlib import Path

path = Path(sys.argv[1])
print(len(path.read_text(encoding="utf-8").splitlines()))

• Validation: The program should check argument counts and file existence before processing.
• Structured parsing: argparse.ArgumentParser supports named options, required values, type conversion, usage text, and errors.
• Standard streams: sys.stdin, sys.stdout, and sys.stderr behave like files and support shell redirection.
• Separation of concerns: A filename identifies the resource, whereas opening mode and encoding define how the program accesses it.

D. Getting files from the Internet

Internet files are obtained by sending a request, validating the response, and storing or directly processing its bytes.

• Standard-library retrieval: urllib.request.urlopen() returns a context-managed response.

from urllib.request import urlopen

url = "https://example.com/data.csv"
with urlopen(url, timeout=10) as response:
    data = response.read()

with open("data.csv", "wb") as file:
    file.write(data)

• Binary preservation: Downloaded content should normally be saved with "wb" so no text conversion alters its bytes.
• Response information: Status, headers, and Content-Type help determine whether the expected resource was returned.
• Reliability: Programs should handle timeouts, connection errors, redirects, and unavailable resources.
• Security: Use trusted HTTPS sources, avoid blindly executing downloads, and validate file type, size, and content.

IV. Structured Data Loading and Selection

A. Comma-separated values files

CSV represents rows as records and fields as delimited values, although quoting rules make manual splitting unreliable.

• CSV rules: A field may contain a delimiter, quotation mark, or newline when correctly quoted.
• Reader: csv.reader() returns each row as a list of strings; csv.DictReader() maps header names to values.
• Writer: csv.writer() handles quoting and delimiters according to a selected dialect.

import csv

with open("scores.csv", newline="", encoding="utf-8") as file:
    for row in csv.DictReader(file):
        print(row["name"], int(row["score"]))

• Newline convention: Opening CSV files with newline="" lets the csv module correctly manage platform-specific line endings.
• Types: CSV has no inherent numeric type, so "91" must be converted with int() or inferred by a loading library.
• Variants: Tab-separated data can use delimiter="\t"; other files may require a custom quote character or dialect.

B. Loading data

Loading data converts an external representation into structures suitable for analysis.

• Pandas tables: pandas.read_csv() produces a DataFrame, whose rows are observations and columns are variables.

import pandas as pd

df = pd.read_csv(
    "sales.csv",
    usecols=["region", "units", "price"],
    dtype={"region": "string"}
)

• Other formats: read_excel(), read_json(), and read_parquet() load common spreadsheet, JSON, and columnar data.
• Parsing controls: sep, header, names, dtype, na_values, usecols, and parse_dates clarify how source fields should be interpreted.
• Inspection: df.head(), df.shape, df.info(), and df.describe() reveal samples, dimensions, types, missing values, and numeric summaries.
• Data quality: Incorrect delimiters, mixed types, duplicate rows, and missing entries should be identified before analysis.
• Scale: chunksize=10000 can process a large CSV in blocks rather than loading it entirely into memory.

C. Selecting and filtering

Selection chooses labels or positions, while filtering retains observations that satisfy conditions.

1. Selection:
   • Columns: df["price"] returns a Series; df[["region", "price"]] returns a DataFrame.
   • Labels: df.loc[rows, columns] selects by index and column labels.
   • Positions: df.iloc[rows, columns] selects by zero-based integer positions.
2. Filtering:
   • Boolean mask: Each row receives True or False.
   • Combined conditions: Use &, |, and ~, with each comparison parenthesized.

selected = df.loc[
    (df["units"] >= 10) & (df["region"] == "North"),
    ["units", "price"]
]

• Missing values: isna() and notna() create masks; comparisons with missing values do not behave like ordinary equality tests.
• Ordering: sort_values("price", ascending=False) arranges selected records without changing the selection criterion.

V. Statistical Visualization

A. Visualizing data with Matplotlib and Seaborn

Visualization maps data variables to graphical properties such as position, length, color, and size.

• Matplotlib: matplotlib.pyplot provides detailed figure and axes control.
• Seaborn: Seaborn builds statistical graphics on Matplotlib and works naturally with tidy DataFrame columns.
• Basic workflow:

import matplotlib.pyplot as plt
import seaborn as sns

sns.set_theme(style="whitegrid")
fig, ax = plt.subplots(figsize=(7, 4))
# Plotting command uses ax here.
ax.set(title="Monthly Sales", xlabel="Month", ylabel="Units")
plt.tight_layout()
plt.show()

• Figure and axes: The figure is the complete canvas; an Axes object is an individual plotting area.
• Communication: Titles, units, legends, readable scales, and restrained color choices make a chart interpretable.
• Saving: fig.savefig("chart.png", dpi=300, bbox_inches="tight") exports a high-resolution image.

B. Line plots

A line plot connects ordered observations and is especially suitable for trends over time.

• Mapping: The horizontal axis normally represents an ordered variable; the vertical axis represents a measured quantity.
• Construction: ax.plot(months, sales, marker="o", label="Sales") displays both connecting lines and observations.
• Interpretation: Slope indicates direction and rate of change; peaks and troughs identify local extremes.
• Requirement: Values should be sorted by the horizontal variable before plotting.
• Limitation: Connecting unrelated categories falsely suggests continuity; a bar chart is preferable for nominal categories.
• Multiple series: Distinct colors or line styles can compare groups, but excessive lines create clutter.

C. Bar charts

A bar chart compares numeric magnitudes across discrete categories.

• Encoding: Bar length begins from a common baseline, enabling accurate category comparisons.
• Construction: sns.barplot(data=df, x="region", y="sales", estimator="mean") summarizes sales by region.
• Aggregation: Seaborn’s barplot estimates a statistic, commonly the mean; pre-aggregated totals can use ax.bar(categories, totals).
• Orientation: Horizontal bars improve readability when category labels are long.
• Baseline: A zero baseline should normally be used because truncation exaggerates differences.
• Distinction: Bars represent separate categories and therefore usually have gaps; histogram bins represent adjacent numeric intervals.

D. Histograms

A histogram displays the distribution of one numeric variable by counting observations within intervals.

• Bins: Each bar covers a numeric range, and its height represents frequency, density, or another aggregate.
• Construction: sns.histplot(data=df, x="price", bins=20, kde=True) adds a histogram and optional density estimate.
• Interpretation: Shape can reveal center, spread, skewness, multiple modes, gaps, and possible outliers.
• Bin choice: Too few bins hide structure; too many emphasize random variation.
• Density: With stat="density", total bar area is normalized to approximately 1, rather than total bar height.
• Limitation: Exact individual values are not preserved visually because observations are grouped into intervals.

E. Scatter plots

A scatter plot displays paired numeric observations to investigate relationships between two variables.

• Mapping: Each point represents one row, with coordinates (x, y) determined by two measured variables.
• Construction: sns.scatterplot(data=df, x="advertising", y="sales", hue="region") adds group information through color.
• Interpretation: Direction, strength, curvature, clusters, and outliers describe the observed association.
• Additional encodings: hue, style, and size can represent further variables, but too many encodings reduce clarity.
• Overplotting: Transparency such as alpha=0.5, smaller markers, or sampling can expose dense overlapping points.
• Causation warning: A visible association does not prove that changes in the horizontal variable cause changes in the vertical variable.