Unit 13: NumPy and Pandas

I. Orientation — The Python Data-Science Ecosystem

Python (first released in 1991) is a high-level, interpreted programming language widely used in data science because it combines readable syntax with libraries for numerical computation and structured data analysis. NumPy supplies efficient multidimensional arrays, while Pandas builds labelled Series and DataFrame structures on top of NumPy.

• Core workflow: Data is imported, inspected, cleaned, transformed, analysed, and prepared for visualization or modelling.
• Python convention: Indentation defines code blocks, indexing generally begins at 0, and variables are created by assignment.
• NumPy convention: A NumPy array is usually homogeneous—all elements share one fixed data type.
• Pandas convention: A Series has one labelled axis; a DataFrame has labelled row and column axes.
• Vectorization: Operations are applied to entire arrays or columns instead of being written as explicit Python loops.
• Missing-value convention: Pandas commonly represents unavailable values with NaN, None, or pd.NA, depending on the data type.
• Standard imports:

import numpy as np
import pandas as pd

Here, np and pd are conventional aliases for the NumPy and Pandas packages.

II. Python Foundations — Language and Type System

A. Introduction to Python

Python provides the basic syntax, control structures, functions, and containers through which data-science libraries are used.

• Variables: Assignment binds a name to an object without requiring a declared type: count = 5 binds count to an integer object.
• Dynamic typing: A name can later refer to an object of another type, as in count = "five"; the object has a type, but the variable name is not permanently typed.
• Core containers:
  • List: Ordered and mutable, such as [10, 20, 30].
  • Tuple: Ordered and immutable, such as (10, 20, 30).
  • Dictionary: Stores key-value pairs, such as {"city": "Delhi", "temp": 31}.
  • Set: Stores unique, unordered elements, such as {2, 4, 6}.
• Control flow: if, elif, and else select branches; for and while repeat operations.
• Functions: def packages reusable logic, while return sends a result to the caller.

def mean(values):
    return sum(values) / len(values)

result = mean([2, 4, 6])  # 4.0

Here, values is the input list and result receives the function’s returned arithmetic mean.

• Library access: import makes external modules available; dot notation such as np.array selects an object from a module.

B. Understanding data types in Python

A data type determines a value’s representation, permitted operations, and storage behaviour.

• Numeric types:
  • int: Represents arbitrary-precision whole numbers, such as 42.
  • float: Usually represents double-precision floating-point values, such as 3.14.
  • complex: Represents numbers such as 2 + 3j, where j denotes the imaginary unit.
• Other scalar types: bool stores True or False; str stores Unicode text; NoneType has the single value None.
• Inspection: type(x) returns the class of x, while isinstance(x, int) tests whether x belongs to the specified class.
• Conversion: Constructors such as int("12"), float(5), and str(3.5) create converted values when conversion is valid.
• Mutability: Lists and dictionaries can change in place; integers, strings, and tuples cannot.
• NumPy distinction: NumPy uses fixed-width types such as np.int32, np.int64, and np.float64, allowing predictable memory use and fast computation.
• Type promotion: Combining integers and floating-point values in one NumPy array commonly promotes all elements to a compatible floating type.

a = np.array([1, 2, 3], dtype=np.int32)
b = np.array([1, 2.5, 3])

Here, a.dtype is int32, whereas b receives a floating-point type because 2.5 cannot be represented as an integer without information loss.

III. NumPy — Efficient Numerical Arrays

A. NumPy

NumPy is a numerical-computing library centred on the homogeneous, multidimensional ndarray.

• Array creation: np.array, np.zeros, np.ones, np.arange, and np.linspace create arrays from values or numerical patterns.
• Dimensions: ndim gives the number of axes, shape gives each axis length, and size gives the total element count.
• Storage: dtype records the element type, while itemsize reports bytes used by one element.
• Vectorized arithmetic: For equal-shaped arrays, a + b, a * b, and a ** 2 operate element by element without explicit loops.
• Broadcasting: NumPy can combine compatible shapes by conceptually expanding dimensions of length 1; arrays with shapes (3, 1) and (1, 4) produce a result of shape (3, 4).
• Aggregation: sum, mean, min, max, and std reduce values across an entire array or a specified axis.
• Reshaping: reshape changes dimensions without changing element count; an array of size 6 can be reshaped to (2, 3).
• Boolean filtering: A condition creates a Boolean mask that selects matching values.

scores = np.array([52, 81, 67, 90])
passed = scores[scores >= 60]
average = scores.mean()

Here, scores >= 60 produces [False, True, True, True], passed becomes [81, 67, 90], and average is 72.5.

• Limitation: Because an array normally has one dtype, mixed real-world records are often better represented by a Pandas DataFrame.

IV. Pandas — Labelled Tabular Analysis

A. Pandas for data analysis

Pandas supplies labelled structures and operations for cleaning, transforming, combining, and summarizing structured data.

• Series: pd.Series is a one-dimensional sequence whose values are associated with an index.
• DataFrame: pd.DataFrame is a two-dimensional table in which columns may have different data types.
• Construction: DataFrames can be created from dictionaries, lists of records, NumPy arrays, or imported files.
• Input and output: Functions such as pd.read_csv() load tabular data, while DataFrame.to_csv() writes it.
• Inspection: head() previews rows, shape reports dimensions, dtypes lists column types, and info() summarizes non-null counts and memory use.
• Descriptive analysis: describe() calculates statistics such as count, mean, standard deviation, quartiles, minimum, and maximum for suitable columns.
• Transformation: Column arithmetic is vectorized; assign, replace, astype, and rename alter values, types, or labels.
• Grouping: groupby() implements split-apply-combine analysis by partitioning rows, applying an aggregation, and combining results.

sales = pd.DataFrame({
    "region": ["East", "West", "East"],
    "amount": [120, 150, 180]
})
totals = sales.groupby("region")["amount"].sum()

Here, totals maps East to 300 and West to 150.

• Alignment: Pandas aligns objects by labels during operations, which is powerful but can introduce missing values when labels differ.

V. Indexing and Selection — Accessing Labelled Data

A. Data indexing and selection

Indexing identifies positions or labels, while selection extracts specific values, rows, columns, or subsets.

• Positional indexing: Python and NumPy use zero-based positions; a[0] accesses the first element.
• Slicing: a[start:stop:step] includes start but excludes stop; a[1:4] selects positions 1, 2, and 3.
• Multidimensional indexing: matrix[1, 2] selects the element at row position 1 and column position 2.
• Boolean selection: data[data > 0] retains values satisfying the condition.
• Fancy indexing: A list or array of integer positions, such as a[[0, 2]], selects non-contiguous elements.
• View versus copy: Basic NumPy slices often share memory with the original array, whereas fancy indexing generally produces a copy.
• Pandas accessors:
  1. .loc: Selects by labels, and label slices include both endpoints.
  2. .iloc: Selects by integer position, and slices exclude the stopping position.

B. Data selection in Series

Series selection can use index labels, integer positions, Boolean masks, or explicit accessors.

• Label selection: s.loc["b"] retrieves the value whose index label is "b".
• Position selection: s.iloc[1] retrieves the value at position 1, independent of its label.
• Multiple values: s.loc[["a", "c"]] returns a new Series containing those labels in the requested order.
• Conditional selection: s[s >= 20] retains values meeting the Boolean condition.
• Scalar access: .at[label] and .iat[position] are specialized accessors for one value.
• Ambiguity prevention: Explicit .loc and .iloc are preferable when an index contains integers because label 1 and position 1 need not identify the same element.

s = pd.Series([10, 20, 30], index=["a", "b", "c"])
value = s.loc["b"]       # 20
subset = s.iloc[1:]      # labels "b" and "c"

Here, value is scalar, while subset remains a Series.

C. Data selection in DataFrame

DataFrame selection operates across two labelled axes: rows and columns.

• Column selection: df["amount"] returns a Series, while df[["amount"]] returns a one-column DataFrame.
• Label-based selection: df.loc[row_labels, column_labels] selects named rows and columns.
• Position-based selection: df.iloc[row_positions, column_positions] selects by zero-based location.
• Row filtering: df.loc[df["amount"] > 100] retains complete rows satisfying the condition.
• Combined conditions: Pandas uses &, |, and ~ for elementwise AND, OR, and NOT; each comparison should be parenthesized.
• Querying: df.query("amount > 100") offers a concise expression-based alternative for suitable column names.
• Safe assignment: df.loc[condition, "status"] = "high" selects and updates in one operation, avoiding unreliable chained assignment.

selected = sales.loc[
    (sales["amount"] >= 150) & (sales["region"] == "East"),
    ["region", "amount"]
]

Here, the Boolean mask filters rows and the column list restricts the result to two named columns.

VI. Missing Values — Detection and Treatment

A. Missing data in Pandas

Missing data represents an unavailable, unknown, or inapplicable observation rather than an ordinary zero or empty string.

• Representations: np.nan commonly marks missing numeric data, None may occur in object columns, and pd.NA supports nullable Pandas dtypes.
• Detection: isna() and isnull() return equivalent Boolean masks; notna() identifies present values.
• Counting: df.isna().sum() counts missing entries in each column.
• Arithmetic behaviour: Many reductions skip missing values by default; Series([1, np.nan, 3]).mean() returns 2.0.
• Data-type effect: Introducing np.nan into a traditional integer column may convert it to floating point; nullable Int64 preserves integer semantics with pd.NA.
• Comparison rule: Equality tests are unsuitable for detecting NaN because NaN is not equal to itself; use pd.isna(value).
• Analytical risk: Missingness may be systematic—for example, absent income values may cluster within a particular group—so counts should be examined by relevant categories.

B. Handling missing data

Handling missing data means choosing a defensible strategy that preserves analytical validity while minimizing unnecessary information loss.

1. Deletion:
   • Row removal: dropna() removes rows containing missing values; subset=["age"] limits the rule to a critical column.
   • Column removal: dropna(axis="columns") removes affected columns and is justified only when their information is dispensable.
   • Threshold control: dropna(thresh=3) keeps rows with at least three non-missing values.
2. Imputation:
   • Constant filling: fillna(0) inserts a defined value, although zero must have a valid domain meaning.
   • Statistical filling: Numeric values may use a median, while categorical values may use a mode.
   • Propagation: ffill() uses the preceding observation and bfill() uses the following observation; these methods require meaningful ordering.
   • Interpolation: interpolate() estimates intermediate numeric values, especially in ordered or time-series data.

df["age"] = df["age"].fillna(df["age"].median())
df["category"] = df["category"].fillna("Unknown")

Here, age receives the column median, while missing categories receive an explicit label rather than being confused with an observed category.

• Validation: After treatment, df.isna().sum() should be recalculated, data types rechecked, and the chosen method documented because imputation changes the observed distribution.