Unit 6: File Handling, Data Loading, and Visualization - Subjective Questions

Reading a text file involves opening the file, accessing its contents, and closing it after use.

• The open() function opens a file:
  file = open("data.txt", "r")
• The mode "r" opens the file for reading. It is the default mode.
• read() returns the entire file as one string:
  content = file.read()
• readline() reads one line at a time:
  line = file.readline()
• readlines() returns all lines as a list:
  lines = file.readlines()
• The file should be closed using file.close() to release system resources.

Example:
file = open("data.txt", "r")
content = file.read()
print(content)
file.close()

For safer programming, the with statement should be used because it closes the file automatically.

Python provides several file modes for controlling how a file is accessed:

• "r": Opens an existing file for reading. It raises an error if the file does not exist.
• "w": Opens a file for writing. It creates a new file or overwrites an existing file.
• "a": Opens a file for appending data at the end without deleting existing content.
• "x": Creates a new file and raises an error if the file already exists.
• "r+": Opens an existing file for both reading and writing.
• "b": Specifies binary mode, such as "rb" or "wb".
• "t": Specifies text mode, which is the default.

Modes can be combined. For example, "wb" writes binary data, while "a+" allows reading and appending. Selecting the correct mode prevents accidental data loss and ensures that the file is accessed appropriately.

Python uses the write() method to store text in a file.

Example using write mode:
with open("output.txt", "w") as file:
file.write("Python programming\n")

The "w" mode creates the file if it does not exist. If the file already exists, its previous contents are erased before writing.

Example using append mode:
with open("output.txt", "a") as file:
file.write("File handling is useful.\n")

The "a" mode preserves existing content and adds new content at the end. Multiple strings can be written using writelines():
file.writelines(["First line\n", "Second line\n"])

Therefore, write mode is suitable for creating or replacing a complete file, while append mode is suitable for adding logs, records, or new entries without losing existing data.

The with statement provides a context manager for safely working with files.

Example:
with open("notes.txt", "r") as file:
data = file.read()
print(data)

When execution leaves the with block, Python automatically closes the file, even if an exception occurs.

Advantages:

• Automatic closure: The file is closed without explicitly calling close().
• Resource safety: Operating-system file resources are released promptly.
• Exception handling: The file is still closed if an error occurs inside the block.
• Readable code: The structure clearly shows the scope in which the file is being used.
• Reduced programming errors: It prevents forgetting to close a file.

Thus, the with statement is the recommended method for reading and writing files in Python.

Python interacts with the file system through modules such as os and pathlib.

• A relative path is interpreted from the current working directory, such as data/input.txt.
• An absolute path gives the complete location of a file.
• os.getcwd() returns the current working directory.
• os.listdir() lists files and folders in a directory.
• os.path.exists(path) checks whether a path exists.
• os.makedirs(path, exist_ok=True) creates directories.
• os.remove(path) deletes a file.

The pathlib module provides an object-oriented approach:
from pathlib import Path
path = Path("data") / "input.txt"
if path.exists():
print(path.read_text())

Using pathlib improves portability because it handles operating-system-specific path separators more consistently.

Binary data consists of raw bytes rather than human-readable characters. Images, audio, video, executable files, and serialized objects are examples of binary data.

A binary file is opened using a mode containing "b":
with open("image.jpg", "rb") as source:
data = source.read()

Binary data can be written as follows:
with open("copy.jpg", "wb") as target:
target.write(data)

Difference between text and binary mode:

• Text mode, such as "r", returns strings and performs character encoding and newline translation.
• Binary mode, such as "rb", returns objects of type bytes and does not perform text decoding.
• Text mode is suitable for files such as .txt and .csv.
• Binary mode is necessary for images and other non-text files.

Opening binary data in text mode can cause decoding errors or corrupt the data.

Command-line arguments allow users to provide file names when starting a program instead of hard-coding them.

Using the sys module:
import sys
input_file = sys.argv[1]
output_file = sys.argv[2]

If the program is executed as python process.py input.txt output.txt, then sys.argv[0] contains the program name, while the following elements contain the supplied arguments.

A more robust approach uses argparse:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("input_file")
parser.add_argument("output_file")
args = parser.parse_args()

The program can then read from args.input_file and write to args.output_file. Command-line arguments make programs reusable, configurable, and suitable for automation. Input validation should be performed to check file existence and permitted extensions.

A CSV file stores tabular data in rows and columns. Each row is usually placed on a separate line, and values are separated by commas. The first row may contain column headings.

Example CSV content:
Name,Age,Course
Asha,20,Python
Ravi,21,Statistics

Reading a CSV file:
import csv
with open("students.csv", newline="") as file:
reader = csv.DictReader(file)
for row in reader:
print(row["Name"], row["Age"])

Writing a CSV file:
with open("students.csv", "w", newline="") as file:
writer = csv.writer(file)
writer.writerow(["Name", "Age"])
writer.writerow(["Asha", 20])

csv.reader returns rows as lists, while csv.DictReader represents rows as dictionaries using column headings. The csv module also correctly handles quoted values and embedded commas.

Files can be downloaded from the Internet using libraries such as urllib.request or requests.

Using urllib.request:
from urllib.request import urlretrieve
urlretrieve("https://example.com/data.csv", "data.csv")

Using requests:
import requests
response = requests.get("https://example.com/data.csv", timeout=30)
response.raise_for_status()
with open("data.csv", "wb") as file:
file.write(response.content)

Important safety and reliability practices:

• Use HTTPS whenever possible.
• Set a timeout to avoid waiting indefinitely.
• Check the response status using raise_for_status().
• Save downloaded content in binary mode.
• Validate the file type and size before processing it.
• Avoid executing downloaded files without verification.
• Handle network and permission errors with suitable exception handling.

The downloaded file can then be loaded using a CSV reader or a data-analysis library.

Pandas provides convenient functions for loading structured data into a DataFrame.

For a CSV file:
import pandas as pd
df = pd.read_csv("sales.csv")

For an Excel file:
df = pd.read_excel("sales.xlsx", sheet_name="Sheet1")

Typical loading steps are:

• Import the required library.
• Provide the correct file path or URL.
• Inspect the loaded data using df.head().
• Check dimensions using df.shape.
• Examine column names using df.columns.
• Check data types and missing values using df.info() and df.isnull().sum().
• Convert columns to suitable types if required.

Pandas automatically represents tabular data as rows and named columns. Options such as sep, encoding, na_values, and usecols can be supplied to handle different file formats and data-quality conditions.

Pandas supports selection by labels and by integer positions.

Column selection:

• One column: df["marks"]
• Multiple columns: df[["name", "marks"]]

Using loc:
loc selects using row and column labels:
df.loc[0, "marks"]
df.loc[:, ["name", "marks"]]
df.loc[df["marks"] >= 50, ["name", "marks"]]

Using iloc:
iloc selects using integer positions:
df.iloc[0, 2]
df.iloc[:, 0:2]

The main distinction is that loc is label-based, whereas iloc is position-based. In a slice, loc generally includes the ending label, while iloc follows ordinary Python slicing and excludes the ending position. Selecting only required columns improves clarity and can reduce memory usage.

Filtering creates a subset of rows that satisfy a Boolean condition.

For one condition:
high_scores = df[df["marks"] >= 80]

For multiple conditions, use parentheses and bitwise operators:
result = df[(df["marks"] >= 50) & (df["department"] == "CS")]

Use | for OR and ~ for NOT:
df[(df["age"] < 20) | (df["age"] > 25)]

Filtering missing values:
df[df["email"].notna()]
df[df["marks"].isna()]

String filtering:
df[df["name"].str.startswith("A", na=False)]
df[df["course"].str.contains("Python", na=False)]

Pandas conditions produce a Boolean Series, and rows corresponding to True are retained. Parentheses are important because Python operator precedence can otherwise produce errors or unintended results.

Data visualization represents data graphically so that trends, comparisons, distributions, and relationships can be understood more easily.

Basic steps are:

• Load the data into a structure such as a Pandas DataFrame.
• Inspect and clean missing or invalid values.
• Select the variables relevant to the question.
• Choose an appropriate chart type.
• Create the chart using Matplotlib or Seaborn.
• Add a title, axis labels, legend, and suitable formatting.
• Display or save the figure.

Example:
import matplotlib.pyplot as plt
plt.plot(df["month"], df["sales"])
plt.title("Monthly Sales")
plt.xlabel("Month")
plt.ylabel("Sales")
plt.show()

A good visualization should be accurate, readable, appropriately scaled, and free from unnecessary decoration.

Matplotlib is a general-purpose plotting library that provides detailed control over figures, axes, colors, markers, annotations, and layouts. It supports many chart types and is suitable for highly customized plots.

Seaborn is built on Matplotlib and provides a higher-level interface designed especially for statistical visualization. It integrates well with Pandas DataFrames and offers attractive default styles.

Similarities:

• Both can create charts such as line plots, bar charts, histograms, and scatter plots.
• Both support titles, labels, legends, and figure customization.
• Seaborn plots can be further modified using Matplotlib commands.

Differences:

• Matplotlib offers lower-level and more detailed control.
• Seaborn provides simpler syntax for statistical relationships and grouped data.
• Seaborn includes features such as confidence intervals, color palettes, and themes.

They are often used together: Seaborn creates the plot and Matplotlib adjusts or saves it.

A line plot connects ordered data points with line segments. It is especially useful for showing trends over time or another continuous, ordered variable.

Matplotlib example:
import matplotlib.pyplot as plt
plt.plot(df["month"], df["sales"], marker="o", label="Sales")
plt.xlabel("Month")
plt.ylabel("Sales")
plt.title("Sales Trend")
plt.legend()
plt.grid(True)
plt.show()

Seaborn example:
import seaborn as sns
sns.lineplot(data=df, x="month", y="sales", marker="o")

Line plots can be customized using color, marker style, line width, line style, labels, legends, grid lines, and figure size. Multiple lines can compare categories, but each line should be clearly labeled. A line plot is not ideal for unordered categorical data because connecting unrelated categories may suggest a false sequence.

A bar chart represents categorical values using rectangular bars. The length or height of each bar is proportional to the associated value.

• Vertical bar chart: Categories are placed on the horizontal axis and values on the vertical axis. Matplotlib uses plt.bar().
• Horizontal bar chart: Categories are placed on the vertical axis. Matplotlib uses plt.barh(). It is useful when category names are long.
• Grouped bar chart: Bars for multiple series are placed side by side to compare categories.
• Stacked bar chart: Series are placed on top of one another to show both totals and composition.

Example:
plt.bar(df["product"], df["quantity"], color="steelblue")
plt.xlabel("Product")
plt.ylabel("Quantity")
plt.title("Quantity by Product")

Bars should usually begin at zero so that their lengths represent values honestly. Bar charts are appropriate for comparing discrete categories.

A histogram displays the distribution of a numerical variable by dividing its range into intervals called bins. The height of each bar represents the number of observations in that interval.

Example:
plt.hist(df["age"], bins=10, edgecolor="black")
plt.xlabel("Age")
plt.ylabel("Frequency")
plt.title("Age Distribution")

Important concepts:

• Bins: Intervals into which numerical values are grouped.
• Frequency: The number of observations in each bin.
• Density: A normalized scale where the total area of the bars is approximately .

A histogram is used for continuous or discrete numerical data and helps reveal center, spread, skewness, gaps, and possible outliers. A bar chart compares separate categorical values, so its bars are normally separated by gaps. Histogram bars represent adjacent numerical intervals and generally touch each other.

A scatter plot represents each observation as a point with coordinates , where and are numerical variables.

Example using Matplotlib:
plt.scatter(df["hours"], df["marks"], alpha=0.7)
plt.xlabel("Study Hours")
plt.ylabel("Marks")
plt.title("Study Hours and Marks")
plt.show()

Example using Seaborn:
sns.scatterplot(data=df, x="hours", y="marks", hue="class")

Scatter plots help identify:

• Positive or negative association.
• Linear or nonlinear patterns.
• Clusters or groups.
• Outliers.
• Changing variability across values.

A point cloud that rises from left to right suggests a positive association, while one that falls suggests a negative association. However, association does not prove causation. A trend line may summarize the relationship but should be interpreted along with context and data quality.

A complete workflow can be organized as follows:

1. Download the file:
   import requests
url = "https://example.com/sales.csv"
response = requests.get(url, timeout=30)
response.raise_for_status()
with open("sales.csv", "wb") as file:
file.write(response.content)

2. Load the data:
   import pandas as pd
df = pd.read_csv("sales.csv")

3. Inspect and clean:
   print(df.head())
df = df.dropna(subset=["region", "sales"])

4. Filter:
   filtered = df[(df["region"] == "East") & (df["sales"] > 1000)]

5. Visualize:
   import seaborn as sns
import matplotlib.pyplot as plt
sns.barplot(data=filtered, x="product", y="sales", estimator="sum")
plt.title("East Region Sales Above 1000")
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

This workflow combines Internet access, binary file writing, data loading, filtering, cleaning, and visualization.

A suitable program is:

import argparse
import pandas as pd

parser = argparse.ArgumentParser(description="Filter a CSV file")
parser.add_argument("input_file")
parser.add_argument("output_file")
args = parser.parse_args()

try:
data = pd.read_csv(args.input_file)
result = data[data["marks"] >= 50]
result.to_csv(args.output_file, index=False)
print(f"Saved {len(result)} records")
except FileNotFoundError:
print("The input file was not found.")
except KeyError:
print("The CSV must contain a marks column.")

The program can be executed as:
python filter_data.py students.csv passed.csv

Explanation:

• argparse receives file names from the command line.
• read_csv() loads the input into a DataFrame.
• Boolean filtering retains rows with marks greater than or equal to .
• to_csv() writes the selected records without adding the DataFrame index.
• Exception handling provides meaningful messages for common errors.