Unit 9: Handling data with pandas - Subjective Questions

pandas is an open-source Python library used for data manipulation and analysis. It provides labeled data structures that make structured data easy to organize, inspect, transform, and summarize.

Its importance includes:

• It provides the one-dimensional Series and two-dimensional DataFrame data structures.
• It can read and write data in formats such as CSV, Excel, JSON, and SQL tables.
• It supports filtering, sorting, grouping, aggregation, and handling missing values.
• It provides fast, vectorized operations that reduce the need for explicit loops.
• It integrates well with libraries such as NumPy, Matplotlib, and scikit-learn.

It is conventionally imported using:

import pandas as pd

The two principal pandas data structures are:

• Series: A one-dimensional labeled array that stores values of a common or compatible data type. Every value has an index label.
• DataFrame: A two-dimensional labeled table consisting of rows and columns. Different columns can contain different data types.

Example:

marks = pd.Series([78, 85, 91], index=["A", "B", "C"])

table = pd.DataFrame({"Name": ["Asha", "Ravi"], "Marks": [78, 85]})

A DataFrame can be viewed as a collection of Series objects sharing a common row index.

A Series is a one-dimensional labeled data structure capable of storing numbers, strings, Boolean values, or other Python objects.

A Series can be created in several ways:

• From a list: pd.Series([10, 20, 30])
• From a list with custom indexes: pd.Series([10, 20], index=["x", "y"])
• From a dictionary: pd.Series({"Math": 90, "Science": 85})
• From a scalar: pd.Series(5, index=["a", "b", "c"])
• From a NumPy array: pd.Series(np.array([2, 4, 6]))

The indexes identify the values and enable label-based access. If custom indexes are not supplied, pandas creates integer indexes beginning at 0.

Indexing selects individual values, while slicing selects a range of values from a Series.

Consider:

s = pd.Series([60, 70, 80], index=["a", "b", "c"])

• s["b"] returns the value associated with label b.
• s.loc["a":"c"] performs label-based slicing and includes both boundary labels.
• s.iloc[0] selects the value at integer position 0.
• s.iloc[0:2] performs position-based slicing and excludes position 2.

Difference:

• loc uses index labels and generally includes the ending label in a label slice.
• iloc uses zero-based integer positions and follows ordinary Python slicing rules.

Explicitly using these accessors avoids ambiguity when a Series has integer labels.

Arithmetic operators such as +, -, *, and / can be applied to a Series. Operations with scalars are vectorized and affect every value.

Example: s * 2 multiplies every element of s by .

When two Series are combined, pandas aligns their values by index label, not merely by position:

s1 = pd.Series({"a": 10, "b": 20})

s2 = pd.Series({"b": 5, "c": 8})

For s1 + s2, only label b exists in both Series, so its result is . Labels a and c produce missing values (NaN) because one operand is absent.

Methods such as s1.add(s2, fill_value=0) can replace missing operands during the calculation.

A DataFrame is a two-dimensional, size-mutable, labeled data structure with rows and columns. Each column may have a different data type.

Common creation methods include:

• Dictionary of lists: pd.DataFrame({"Name": ["Asha", "Ravi"], "Age": [18, 19]})
• List of dictionaries: pd.DataFrame([{"Name": "Asha", "Age": 18}, {"Name": "Ravi", "Age": 19}])
• List of lists: pd.DataFrame([["Asha", 18], ["Ravi", 19]], columns=["Name", "Age"])
• Dictionary of Series: pd.DataFrame({"Math": pd.Series([80, 90]), "Science": pd.Series([75, 88])})

The optional index and columns arguments define custom row and column labels.

DataFrame data can be selected in the following ways:

• df["Name"] selects one column and returns a Series.
• df[["Name", "Marks"]] selects multiple columns and returns a DataFrame.
• df.loc["r1"] selects the row whose label is r1.
• df.loc["r1", "Marks"] selects a value using its row and column labels.
• df.iloc[0] selects the first row by position.
• df.iloc[0:3, 1:3] selects rows and columns by positional slices.
• df.loc[df["Marks"] >= 50] selects rows satisfying a Boolean condition.

Thus, bracket notation is commonly used for columns, loc is label-based, and iloc is position-based.

A DataFrame can be modified in several ways:

• Add a column: df["Total"] = df["Math"] + df["Science"]
• Modify a column: df["Marks"] = df["Marks"] + 5
• Add a row: df.loc[len(df)] = ["Neha", 82] when the values match the columns.
• Rename columns: df.rename(columns={"Marks": "Score"}, inplace=True)
• Delete a column: df.drop(columns=["Score"], inplace=True) or del df["Score"]
• Delete a row: df.drop(index=[2], inplace=True)

Most pandas transformation methods return a new DataFrame by default. Assigning the result back or using inplace=True where supported applies the intended change.

Important attributes and inspection methods include:

• df.shape: Returns a tuple containing the number of rows and columns.
• df.size: Returns the total number of elements.
• df.ndim: Returns the number of dimensions.
• df.index: Displays row labels.
• df.columns: Displays column labels.
• df.dtypes: Shows the data type of each column.
• df.head(n): Displays the first rows; the default is .
• df.tail(n): Displays the last rows.
• df.info(): Summarizes columns, non-null counts, data types, and memory usage.
• df.describe(): Produces descriptive statistics for numeric columns by default.

These features help detect structural, type, range, and missing-data issues before analysis.

sort_values() arranges data according to the actual values in one or more columns or in a Series. For example:

df.sort_values(by="Marks", ascending=False)

This arranges rows from the highest to the lowest mark.

sort_index() arranges data according to row or column labels. For example:

df.sort_index(ascending=True)

This arranges rows in ascending index order. To sort column labels, use df.sort_index(axis=1).

Key distinction:

• sort_values() sorts by stored data values.
• sort_index() sorts by axis labels.

Both methods return a sorted object by default; inplace=True can modify the original object where supported.

A DataFrame can be sorted by multiple columns by passing a list to by:

df.sort_values(by=["Class", "Marks"], ascending=[True, False])

This first sorts Class in ascending order. Within each class, it sorts Marks in descending order.

Missing values can be positioned with the na_position argument:

• na_position="last" places missing values at the end and is the default.
• na_position="first" places missing values at the beginning.

Other useful arguments include:

• axis: Selects the axis to sort.
• inplace: Controls whether the original object is modified.
• ignore_index=True: Replaces the resulting row index with 0, 1, 2, ....

Sorting does not normally reset the original index unless requested.

A CSV, or comma-separated values, file stores tabular data as plain text. Each line usually represents a row, and delimiters separate fields.

A CSV file can be loaded using:

df = pd.read_csv("students.csv")

Useful arguments include:

• sep: Specifies the delimiter, such as sep=";".
• header: Identifies the row containing column names.
• names: Supplies custom column names.
• index_col: Uses a selected column as the row index.
• usecols: Loads only selected columns.
• nrows: Limits the number of rows loaded.
• skiprows: Skips specified rows.
• na_values: Identifies additional strings that represent missing data.

The result is a DataFrame that can be inspected and transformed with pandas operations.

A DataFrame is written to a CSV file using to_csv():

df.to_csv("output.csv", index=False)

Common arguments include:

• index: Controls whether row labels are written.
• columns: Selects the columns to export.
• sep: Specifies the output delimiter.
• header: Controls whether column names are written.
• na_rep: Defines how missing values are represented.
• encoding: Specifies the text encoding.

index=False is commonly used because a default DataFrame index usually has no meaning outside pandas. Omitting it prevents an unnecessary extra column from appearing when the file is read again. If the index contains meaningful identifiers, it may instead be preserved.

A suitable procedure is:

1. Read the file: df = pd.read_csv("students.csv").
2. Inspect the data: Use df.head(), df.shape, df.info(), and df.describe().
3. Check missing values: Use df.isna().sum().
4. Clean missing values: Apply dropna() when incomplete rows are unusable or fillna() when replacement is appropriate.
5. Remove duplicates: Use df.drop_duplicates().
6. Correct data types: For example, df["Marks"] = pd.to_numeric(df["Marks"], errors="coerce").
7. Validate values: Filter or flag impossible marks outside the permitted range.
8. Create derived columns: df["Result"] = df["Marks"].apply(lambda x: "Pass" if x >= 40 else "Fail").
9. Sort records: df = df.sort_values("Marks", ascending=False).
10. Save the result: df.to_csv("clean_students.csv", index=False).

This workflow turns raw CSV data into a consistent, analysis-ready table.

Boolean filtering selects rows for which a condition evaluates to True.

Example:

df[df["Marks"] >= 75]

This returns students scoring at least .

Multiple conditions are combined using:

• & for logical AND
• | for logical OR
• ~ for logical NOT

Each condition must be enclosed in parentheses:

df[(df["Marks"] >= 75) & (df["Attendance"] >= 80)]

Useful related methods include:

• df[df["City"].isin(["Delhi", "Pune"])]
• df[df["Marks"].between(60, 80)]
• df[df["Name"].str.startswith("A", na=False)]

Python keywords such as and and or should not be used to combine pandas Series conditions.

pandas provides vectorized methods for summarizing DataFrame data:

• sum() calculates totals.
• mean() calculates the arithmetic mean, .
• median() finds the middle value.
• min() and max() find extreme values.
• count() counts non-missing values.
• std() calculates standard deviation.
• nunique() counts distinct values.
• value_counts() calculates frequencies in a Series.
• describe() returns a collection of descriptive statistics.

Examples:

df["Marks"].mean() computes the average mark.

df[["Math", "Science"]].max() finds each subject's maximum.

Many methods accept an axis argument: axis=0 operates down columns, while axis=1 operates across rows.

Missing values are commonly represented by NaN, None, or NaT. They can be detected using:

• df.isna() or df.isnull()
• df.notna()
• df.isna().sum() for missing-value counts by column

dropna() removes rows or columns containing missing values. For example, df.dropna() removes rows with at least one missing value, while df.dropna(axis=1) removes affected columns.

fillna() retains the records and replaces missing values. Examples include:

• df["Marks"].fillna(0)
• df["Marks"].fillna(df["Marks"].mean())
• df.ffill() to propagate the preceding valid value

dropna() is appropriate when incomplete records are unnecessary or unreliable. fillna() is appropriate when data should be retained and a defensible replacement exists.

groupby() follows the split-apply-combine approach:

1. Split rows into groups using one or more keys.
2. Apply an aggregation or transformation to each group.
3. Combine the results into a Series or DataFrame.

Example:

summary = df.groupby("Department")["Salary"].agg(["count", "mean", "max"])

This groups employees by department and calculates the employee count, average salary, and maximum salary for each department.

Named aggregation can improve column labels:

summary = df.groupby("Department").agg(Employee_Count=("Salary", "count"), Average_Salary=("Salary", "mean"), Highest_Salary=("Salary", "max"))

Grouping may also use multiple keys, such as df.groupby(["Department", "Gender"]), to create more detailed summaries.

Vectorized operations apply an operation to entire Series or DataFrame columns at once. For example, df["Total"] = df["Math"] + df["Science"]. They are usually concise and efficient because pandas performs much of the work in optimized code.

apply() executes a function across a Series or along DataFrame rows or columns. For example, df["Grade"] = df["Marks"].apply(assign_grade). It is useful for logic that cannot be expressed directly with built-in vectorized methods.

Explicit Python loops process values one at a time. They are generally more verbose and slower for large datasets.

The preferred order is:

1. Use built-in vectorized operations where possible.
2. Use specialized pandas methods such as where(), map(), or string methods.
3. Use apply() for suitable custom logic.
4. Use explicit loops only when the alternatives do not fit the task.

The required analysis can be performed as follows:

1. Calculate total marks:

   df["Total"] = df[["Math", "Science", "English"]].sum(axis=1)

2. Calculate percentage: If every subject is out of ,

   df["Percentage"] = df["Total"] / 300 * 100

   Thus, .

3. Assign pass or fail:

   df["Result"] = df[["Math", "Science", "English"]].ge(40).all(axis=1).map({True: "Pass", False: "Fail"})

4. Filter successful students:

   passed = df[df["Result"] == "Pass"].copy()

5. Assign ranks:

   passed["Rank"] = passed["Total"].rank(method="dense", ascending=False).astype(int)

6. Sort the result:

   passed = passed.sort_values(["Rank", "Name"])

This solution uses row-wise aggregation, Boolean operations, filtering, ranking, and sorting without explicit loops.