Unit 4: Data Analysis with NumPy and Pandas

I. Orientation — Array-Based Data Analysis

NumPy and Pandas are Python libraries for representing, transforming, and analyzing structured data. NumPy provides fast numerical arrays, while Pandas builds labeled tabular structures on top of array-based computation. Both rely on systematic indexing, consistent data types, vectorized operations, and functions that operate on complete collections of values.

• Core principle: Store related values in structured containers so that one operation can process many values efficiently.
• NumPy convention: An array has a fixed data type, dimensions, shape, and zero-based indexes.
• Pandas convention: A table is generally a DataFrame containing labeled rows and columns; a single labeled column is a Series.
• Vectorization: Prefer array or column operations such as arr * 2 or df["price"] * 1.1 instead of explicit element-by-element loops.
• Data-analysis workflow: Load data, inspect its structure, select relevant values, transform them, clean errors, and summarize results.
• Missing-data principle: Missing, invalid, and duplicated values should be detected explicitly before analysis.

II. NumPy Arrays — Numerical Data Structures

NumPy’s central object is the ndarray, a homogeneous, multidimensional array designed for efficient numerical computation. It is created through the numpy library, conventionally imported as np.

A. NumPy ndarray object

The ndarray object stores values in a regular arrangement with a common data type. Its main attributes include ndim, shape, size, and dtype.

• Creation: np.array([10, 20, 30]) creates a one-dimensional integer array.
• Attributes: For a = np.array([[1, 2], [3, 4]]), a.ndim is 2, a.shape is (2, 2), a.size is 4, and a.dtype identifies the element type.
• Homogeneity: Unlike a normal Python list, an ndarray normally stores values of one compatible type, such as int64 or float64.
• Initialization: np.zeros((2, 3)) creates six zeros, while np.arange(0, 10, 2) creates [0, 2, 4, 6, 8].
• Efficiency: Contiguous numeric storage and compiled operations make a + 5 faster and more concise than a Python loop for many values.

B. Dimensions in arrays

Dimensions describe the number of indexing directions, while shape gives the length along each direction.

• Zero-dimensional: A scalar such as np.array(7) has ndim == 0 and shape ().
• One-dimensional: np.array([4, 8, 12]) has one axis, shape (3,), and is often interpreted as a vector.
• Two-dimensional: A matrix such as [[1, 2], [3, 4]] has shape (2, 2); the first axis represents rows and the second represents columns.
• Higher-dimensional: An array with shape (2, 3, 4) contains two blocks, each with three rows and four columns, for 2 × 3 × 4 = 24 elements.
• Shape condition: Reshaping must preserve element count; an array of size 12 can become (3, 4) but not (5, 2).

C. Accessing array elements

Array elements are accessed with zero-based indexing and slicing, using one index per axis.

• Single element: In a = np.array([[10, 20], [30, 40]]), a[1, 0] returns 30: row index 1, column index 0.
• Negative indexing: a[-1, -1] returns 40, the element in the last row and last column.
• Slicing: a[:, 1] selects every row of column 1; a[0, :] selects every column of row 0.
• Step values: x[::2] selects positions 0, 2, 4, ...; the slice format is start:stop:step.
• Boolean selection: x[x > 5] returns values satisfying the condition, such as all values greater than 5.
• Fancy indexing: x[[0, 2]] selects the elements at indexes 0 and 2, allowing nonconsecutive selection.

D. NumPy array manipulation

Array manipulation changes an array’s shape, orientation, or arrangement without necessarily changing its values.

• Reshaping: a.reshape(2, 3) changes a size-six array into two rows and three columns; the product 2 × 3 must equal a.size.
• Flattening: a.ravel() returns a one-dimensional view where possible, while a.flatten() returns a separate one-dimensional copy.
• Transposition: a.T exchanges rows and columns; a (2, 3) array becomes (3, 2).
• Joining: np.concatenate([a, b], axis=0) joins arrays along rows when their other dimensions match.
• Splitting: np.split(x, 2) divides an array into two equal sections when its length permits.
• Adding or removing dimensions: np.expand_dims(x, axis=0) adds an axis; np.squeeze(x) removes axes of length one.

E. N-dimensional arrays (ndarray)

An N-dimensional ndarray generalizes vectors and matrices to data with multiple axes, such as images, time series, or simulation results.

• Axis interpretation: For shape (time, height, width), axis=0 may represent time, while axes 1 and 2 represent spatial dimensions.
• Broadcasting: NumPy can combine compatible shapes; adding a shape (3,) array to a (2, 3) array applies the three values to each row.
• Reduction: a.sum(axis=0) reduces the row axis and produces column totals; the axis symbol identifies the dimension being collapsed.
• Elementwise operations: a * b multiplies corresponding values, whereas a @ b performs matrix multiplication when dimensions are compatible.
• Limitation: Irregular rows, such as [[1, 2], [3, 4, 5]], do not form a regular numeric array without object-type behavior, reducing numerical efficiency.

III. Pandas — Labeled Data Analysis

Pandas provides labeled, table-oriented structures and tools for importing, selecting, transforming, aggregating, and cleaning data. Its two fundamental structures are Series and DataFrame.

A. Pandas fundamentals

A Series is a one-dimensional labeled sequence, and a DataFrame is a two-dimensional table whose rows and columns have labels.

• Series creation: pd.Series([80, 90], index=["A", "B"]) associates scores with student labels.
• DataFrame creation: pd.DataFrame({"name": ["Ana", "Ben"], "age": [20, 21]}) creates named columns.
• Inspection: df.head() shows initial rows, df.shape gives (rows, columns), and df.info() reports types and non-null counts.
• Selection: df["age"] returns a Series; df[["name", "age"]] returns a two-column DataFrame.
• Label versus position: df.loc[0, "age"] uses labels, while df.iloc[0, 1] uses integer positions.
• Data types: df.dtypes identifies column types such as int64, float64, object, string, or datetime64.

B. Pandas with CSV and HTML data

Pandas can import common external formats into a DataFrame and export processed tables for later use.

• CSV input: pd.read_csv("sales.csv") interprets comma-separated rows; options such as sep=";", header=None, and usecols=["date", "amount"] adapt to file structure.
• CSV output: df.to_csv("clean_sales.csv", index=False) writes the table without adding the DataFrame index as an extra column.
• HTML input: pd.read_html("tables.html") returns a list of DataFrames extracted from HTML <table> elements.
• Web tables: pd.read_html(url) can retrieve published tables when the page contains readable HTML table markup.
• Validation: After loading, inspect df.head(), df.columns, df.shape, and df.info() because headers, missing values, and inferred types may differ from expectations.

C. Pandas operations

Pandas operations select, filter, sort, combine, group, and summarize labeled data.

• Filtering: df[df["age"] >= 21] keeps rows whose age value is at least 21; the comparison produces a Boolean mask.
• Multiple conditions: (df["age"] >= 18) & (df["city"] == "Delhi") combines conditions; parentheses are required around each comparison.
• Sorting: df.sort_values("salary", ascending=False) places the largest salaries first.
• Aggregation: df["sales"].mean() calculates the arithmetic mean; sum(), min(), max(), and count() provide other summaries.
• Grouping: df.groupby("department")["salary"].mean() computes the mean salary separately for each department.
• Combining tables: pd.merge(left, right, on="id", how="inner") joins tables using matching id values; concat() stacks compatible tables vertically or horizontally.

D. Pandas with functions

Functions allow reusable transformations to be applied to columns, rows, or complete tables.

• Column transformation: df["price"].apply(lambda p: p * 1.18) applies a function that increases each price by 18 percent.
• Named function: A function such as def classify(x): return "High" if x >= 50 else "Low" can be used with df["score"].apply(classify).
• Vectorized alternative: np.where(df["score"] >= 50, "Pass", "Fail") is usually clearer and faster than row-wise Python functions.
• Row-wise application: df.apply(lambda row: row["quantity"] * row["price"], axis=1) calculates a value from multiple columns; axis=1 means each row is passed to the function.
• Mapping values: df["code"].map({"A": "Active", "I": "Inactive"}) replaces category codes according to a dictionary.
• Caution: Functions that process rows individually can be slower on large datasets; built-in Pandas and NumPy operations are generally preferred.

E. Data cleaning process

Data cleaning identifies and corrects missing, duplicated, inconsistent, invalid, or incorrectly typed values before analysis.

• Detect missing values: df.isna().sum() counts missing entries in every column; isna() returns Boolean indicators.
• Remove missing data: df.dropna(subset=["age"]) removes rows missing age; deletion is suitable when missing rows are few and nonessential.
• Fill missing data: df["age"].fillna(df["age"].median()) replaces missing ages with the median, whose symbol is not needed because it is computed directly from the column.
• Remove duplicates: df.drop_duplicates() retains one copy of identical rows; subset=["email"] checks duplication using only selected columns.
• Correct types: pd.to_numeric(df["amount"], errors="coerce") converts numeric text and turns invalid entries into NaN; pd.to_datetime(df["date"], errors="coerce") parses dates similarly.
• Standardize text: df["city"].str.strip().str.lower() removes surrounding spaces and converts names to lowercase, making " Delhi " and "delhi" comparable.
• Validate ranges: A condition such as df["age"].between(0, 120) identifies plausible ages; values outside the range require correction or removal.
• Replace inconsistent values: df["gender"].replace({"M": "Male", "F": "Female"}) converts short codes into consistent category labels.
• Order of work: Inspect first, clean types and labels, handle missing and duplicate records, validate ranges, then save the cleaned dataset for analysis.