Unit 5: Handling Data with Pandas; Data Visualisation with Matplotlib

I. Orientation: The Data-Analysis Stack

Pandas (released 2008 by Wes McKinney) is a Python library built on NumPy for labelled, tabular data; Matplotlib (2003, John Hunter) is the foundational plotting library it pairs with. Together they cover the load–clean–analyse–visualise cycle.

• Import convention: the near-universal aliases are import pandas as pd and import matplotlib.pyplot as plt. All examples assume these.
• Two core objects: pandas exposes the 1-D Series and the 2-D DataFrame; every operation below returns or transforms one of these.
• Label-based indexing: unlike lists, pandas objects carry an explicit index (row labels) and, for DataFrames, columns, so data is aligned by label, not just position.
• Vectorisation: operations apply to whole columns at once (no explicit loops), inheriting NumPy speed.
• Missing data: absent values are represented by NaN (Not a Number) and handled uniformly across the library.

II. Pandas Core Structures

Series and DataFrame

A. Introduction to Pandas

Pandas turns raw rows and columns into indexed objects that support alignment, aggregation, and I/O.

• Purpose: provides fast, expressive structures for structured data — spreadsheets, SQL tables, time series.
• Foundation: built on NumPy arrays, so numeric columns are stored as ndarrays with a dtype (e.g. int64, float64, object for text).
• Design goal: make real-world "messy" data (mixed types, missing entries) easy to load, reshape, and summarise.

B. Series

A Series is a one-dimensional labelled array holding a single data type.

• Creation: from a list, dict, or scalar.
  
  PYTHON

  s = pd.Series([10, 20, 30], index=['a', 'b', 'c'])

• Index: s.index holds the labels ['a','b','c']; if omitted, a default RangeIndex (0,1,2…) is used.
• Access: by label s['b'] → 20 or by position s.iloc[1] → 20.
• Attributes: s.values (the underlying array), s.dtype, s.size.
• Vectorised ops: s * 2 yields 20,40,60; arithmetic between two Series aligns on the index, inserting NaN where labels do not match.

C. DataFrame

A DataFrame is a two-dimensional table: an ordered collection of Series sharing one index, one per column.

• Creation from a dict of lists:
  
  PYTHON

  df = pd.DataFrame({
      'Name': ['Asha', 'Ben', 'Cara'],
      'Age':  [21, 25, 23],
      'Marks':[88, 76, 92]
  })

• Anatomy: df.index (row labels), df.columns (Name, Age, Marks), df.values (2-D array), df.dtypes (per-column types), df.shape → (3, 3).
• Column selection: df['Age'] returns a Series; df[['Name','Age']] returns a DataFrame.
• Row selection: df.loc[0] (by label) and df.iloc[0] (by position) return the first row as a Series.
• Inspection methods: df.head(n) / df.tail(n) show edge rows; df.info() lists column types and non-null counts; df.describe() gives count, mean, std, min, quartiles, max for numeric columns.

III. Getting Data In and Working With It

CSV I/O and DataFrame operations

A. Working with CSV Files

CSV (comma-separated values) is the default interchange format, and pandas reads and writes it in one call.

• Reading: df = pd.read_csv('students.csv') parses the file, using the first line as column headers.
  • Key parameters: sep=';' for other delimiters; header=None when there is no header row; names=[...] to supply column names; index_col='Name' to promote a column to the index; usecols=['Age','Marks'] to load a subset.
  • Missing values: na_values=['NA','?'] maps custom tokens to NaN.
• Writing: df.to_csv('output.csv', index=False) — index=False prevents the row labels being written as an extra column.
• Result: the returned object is an ordinary DataFrame, so every operation below applies directly after loading.

B. Operations Using DataFrames

DataFrames support filtering, computation, grouping, and cleaning as chained, vectorised operations.

• Filtering (boolean masking): df[df['Marks'] > 80] keeps rows where the condition is True; combine with & (and) / | (or), parenthesising each test: df[(df.Age > 22) & (df.Marks > 80)].
• Adding / modifying columns: df['Result'] = df['Marks'] * 0.4 creates a computed column element-wise.
• Descriptive stats: column methods df['Marks'].mean(), .sum(), .max(), .min(), .count() return scalars; .value_counts() tallies category frequencies.
• Sorting: df.sort_values('Marks', ascending=False) orders rows; df.sort_index() orders by label.
• Grouping (split-apply-combine):
  
  PYTHON

  df.groupby('Grade')['Marks'].mean()

  
  splits rows by the Grade value, then averages Marks within each group.
• Handling missing data: df.dropna() removes rows containing NaN; df.fillna(0) or df.fillna(df['Marks'].mean()) substitutes a replacement; df.isnull().sum() counts gaps per column.
• Applying functions: df['Age'].apply(lambda x: x + 1) runs a function over each element.
• Worked example — class summary:
  
  PYTHON

  df = pd.read_csv('students.csv')
  top = df[df['Marks'] >= 90]          # filter high scorers
  avg = df.groupby('Grade')['Marks'].mean()   # group average
  print(avg)

  
  This loads the file, isolates top performers, and reports the mean mark per grade in three lines.

IV. Data Visualisation with Matplotlib

Turning DataFrames into figures

Matplotlib draws onto a Figure (the whole canvas) containing one or more Axes (individual plots). plt.show() renders the result; plt.savefig('name.png') saves it. Pandas objects can be plotted directly via .plot(), which delegates to Matplotlib.

A. Line Plots

Line plots connect data points with straight segments, best for trends over an ordered variable such as time.

• Basic call: plt.plot(x, y) then plt.show().
• Labelling: plt.title('Sales Trend'), plt.xlabel('Month'), plt.ylabel('Sales').
• Styling: plt.plot(x, y, color='red', linestyle='--', marker='o', label='2024'); plt.legend() displays the label; plt.grid(True) adds gridlines.
• Multiple lines: call plt.plot repeatedly before show() to overlay series on one Axes.
• From pandas: df.plot(x='Month', y='Sales', kind='line').
  
  PYTHON

  plt.plot([1,2,3,4], [10,25,18,40], marker='o')
  plt.xlabel('Quarter'); plt.ylabel('Profit'); plt.show()

B. Multiple Subplots in One Figure

Subplots place several independent Axes in a grid within a single Figure, allowing side-by-side comparison.

• Creation: fig, ax = plt.subplots(nrows=2, ncols=2) returns the Figure and a 2×2 array of Axes.
• Addressing an Axes: ax[0,0].plot(...), ax[1,1].bar(...) — each cell is drawn on independently.
• Per-Axes labels: methods take the set_ prefix: ax[0,0].set_title(...), ax[0,0].set_xlabel(...).
• Spacing: plt.tight_layout() prevents titles and labels from overlapping.
• Single-index shortcut: with one row or column, Axes form a 1-D array accessed as ax[0], ax[1].
  
  PYTHON

  fig, ax = plt.subplots(1, 2)
  ax[0].plot([1,2,3],[1,4,9])
  ax[1].bar(['A','B'],[5,8])
  plt.tight_layout(); plt.show()

C. Histograms

A histogram shows the frequency distribution of a single continuous variable by dividing its range into bins.

• Call: plt.hist(data, bins=10) — bins sets the number of intervals; each bar's height is the count of values falling in that interval.
• Distinction: bars touch (they represent a continuous range), unlike a bar chart's separated categories.
• Options: edgecolor='black' outlines bars; range=(0,100) fixes the span; density=True normalises heights to a probability.
• Use: reveals shape — skew, spread, and whether data is unimodal or clustered.
  
  PYTHON

  plt.hist(df['Marks'], bins=5, edgecolor='black')
  plt.xlabel('Marks'); plt.ylabel('Frequency'); plt.show()

D. Bar Charts

A bar chart compares a numeric value across distinct, discrete categories using rectangular bars.

• Vertical: plt.bar(categories, values); horizontal: plt.barh(categories, values) for long labels.
• Styling: color=['red','blue'], width=0.5.
• Grouped/stacked: offset x-positions for grouped bars, or pass bottom= to stack one series above another.
• Contrast with histogram:
  1. Bar chart: discrete categories, gaps between bars, order arbitrary.
  2. Histogram: continuous bins, bars adjacent, order fixed by value.
• From pandas: df['Grade'].value_counts().plot(kind='bar').

E. Pie Charts

A pie chart shows each category's share of a whole as a proportional slice of a circle.

• Call: plt.pie(values, labels=names); slice angle is proportional to each value's fraction of the total.
• Percentages: autopct='%1.1f%%' prints each share to one decimal.
• Emphasis: explode=(0, 0.1, 0) pulls out one slice; startangle=90 rotates the layout; shadow=True adds depth.
• Constraint: suited to a small number of parts summing to a meaningful whole; poor for many near-equal or comparative values, where a bar chart reads more accurately.
  
  PYTHON

  plt.pie([40, 35, 25], labels=['A','B','C'], autopct='%1.1f%%')
  plt.title('Market Share'); plt.show()