Unit 14: Machine Learning Packages in Python - Subjective Questions

Data import is the process of loading data from an external source into a Python environment for inspection, cleaning, analysis, or modeling.

A CSV file can be imported with Pandas as follows:

import pandas as pd
data = pd.read_csv("data.csv")

• pd.read_csv() reads comma-separated data.
• The result is stored as a Pandas DataFrame.
• data.head() displays the first five rows.
• data.info() shows column names, data types, and non-null counts.
• Options such as sep, header, names, index_col, and na_values can control how the file is interpreted.

Pandas provides separate functions for importing common file formats:

• CSV: pd.read_csv("file.csv")
• Excel: pd.read_excel("file.xlsx", sheet_name="Sheet1")
• JSON: pd.read_json("file.json")

All three functions normally return a DataFrame, but they interpret different storage structures:

• CSV stores tabular data as delimited text.
• Excel stores data in workbooks containing one or more sheets.
• JSON represents data through objects, arrays, and key-value pairs.

After import, useful validation commands include head(), shape, info(), and isnull().sum(). Correct parameters must be supplied when a source has unusual delimiters, headers, encodings, or missing-value symbols.

Inspection confirms that the data has been loaded correctly and is suitable for analysis.

Important checks include:

• Preview records: df.head() and df.tail() reveal whether headers and values were parsed correctly.
• Check dimensions: df.shape returns the numbers of rows and columns.
• Review structure: df.info() displays column names, data types, and non-null counts.
• Generate summaries: df.describe() reports descriptive statistics for numerical columns.
• Find missing values: df.isnull().sum() counts null values in each column.
• Detect duplication: df.duplicated().sum() identifies repeated rows.

These checks prevent misleading visualizations and machine-learning results caused by malformed columns, missing values, duplicates, or incorrect data types.

Matplotlib is a Python library used to create static, animated, and interactive visualizations. It supports line plots, scatter plots, bar charts, histograms, pie charts, and many other graph types.

matplotlib.pyplot is a convenient plotting interface that is commonly imported as:

import matplotlib.pyplot as plt

Its functions manage figures, axes, plotting commands, labels, legends, and display operations. For example, plt.plot() creates a line plot, while plt.show() displays the completed figure.

Matplotlib is valuable because it:

• Reveals trends, patterns, distributions, and outliers.
• Offers detailed control over plot appearance.
• Integrates with NumPy, Pandas, Seaborn, and Jupyter notebooks.

A Matplotlib visualization is organized hierarchically:

• A figure is the complete drawing canvas or output window.
• An axes object is an individual plotting area inside the figure.
• An axis controls a coordinate direction, such as the x-axis or y-axis.
• Artists are visible elements such as lines, text, legends, and markers.

A figure can contain multiple axes objects. They can be created with:

fig, ax = plt.subplots()
ax.plot(x, y)

Here, fig refers to the entire canvas and ax refers to the plotting region. The object-oriented approach is particularly useful when creating multiple subplots or precisely customizing a visualization.

A line plot displays ordered observations connected by straight lines and is commonly used to show change over time.

import matplotlib.pyplot as plt

days = [1, 2, 3, 4, 5]
sales = [20, 28, 25, 35, 42]

plt.plot(days, sales, color="blue", marker="o", linestyle="--", label="Sales")
plt.title("Daily Sales")
plt.xlabel("Day")
plt.ylabel("Units Sold")
plt.grid(True)
plt.legend()
plt.show()

Important customization options are:

• color changes the line color.
• marker marks individual observations.
• linestyle changes the line pattern.
• title(), xlabel(), and ylabel() provide context.
• legend() identifies plotted series.
• grid() makes values easier to compare.

A scatter plot represents each observation as a point with coordinates . It is created in Matplotlib using plt.scatter(x, y).

A scatter plot can reveal:

• Positive association: generally increases as increases.
• Negative association: generally decreases as increases.
• No clear association: points show no systematic pattern.
• Nonlinear relationships: points follow a curve rather than a straight line.
• Clusters: observations form distinct groups.
• Outliers: individual points lie far from the main pattern.

A visible association does not by itself establish causation. Statistical analysis and domain knowledge are needed before making causal conclusions.

Line plot:

• Connects observations in a specified order.
• Emphasizes continuity, progression, or trends.
• Is appropriate for time-series data, such as monthly revenue.
• Is created with plt.plot(x, y).

Scatter plot:

• Displays observations as separate points.
• Emphasizes the relationship between two numerical variables.
• Is appropriate for examining variables such as height and weight.
• Is created with plt.scatter(x, y).

Connecting unordered observations with lines can falsely imply continuity. Conversely, a scatter plot may not emphasize sequential change as clearly as a line plot.

Multiple series can be plotted on the same axes by calling a plotting function once for each series:

plt.plot(x, y1, color="blue", marker="o", label="Product A")
plt.plot(x, y2, color="red", marker="s", label="Product B")
plt.xlabel("Month")
plt.ylabel("Sales")
plt.title("Product Sales Comparison")
plt.legend()
plt.grid(True)
plt.show()

The series should be distinguished through:

• Contrasting colors.
• Different markers or line styles.
• Clear labels and a legend.
• Appropriate axis labels and units.

Design should not rely only on color because some readers may have color-vision deficiencies. Markers and line patterns improve accessibility.

Subplots place multiple axes within a single figure. They are useful for comparing different variables or plot types while keeping the visualizations organized.

A arrangement can be created as follows:

import matplotlib.pyplot as plt

fig, axes = plt.subplots(2, 2, figsize=(10, 8))
axes[0, 0].plot(x, y1)
axes[0, 1].scatter(x, y2)
axes[1, 0].hist(y1)
axes[1, 1].bar(x, y2)
fig.tight_layout()
plt.show()

• plt.subplots(2, 2) creates four axes.
• axes[row, column] selects a specific subplot.
• figsize controls the overall dimensions.
• tight_layout() reduces overlap among titles and labels.

Seaborn is a high-level statistical visualization library built on Matplotlib. It is commonly imported with:

import seaborn as sns

Seaborn extends Matplotlib by providing:

• Attractive default themes and color palettes.
• Simple syntax for complex statistical graphics.
• Direct integration with Pandas DataFrames.
• Automatic grouping through parameters such as hue, style, and size.
• Functions for distributions, categorical comparisons, regression relationships, and heatmaps.

For example, sns.scatterplot(data=df, x="age", y="income", hue="group") maps DataFrame columns directly to visual properties. Matplotlib functions can still be used to customize Seaborn plots.

Matplotlib is a general-purpose, lower-level visualization library. It provides detailed control over almost every graphical element and is suitable for highly customized plots.

Seaborn is a higher-level library built on Matplotlib. It provides concise functions for statistical graphics and attractive defaults.

Key differences are:

• Abstraction: Seaborn requires less code for many statistical plots.
• Customization: Matplotlib offers more direct, fine-grained control.
• Appearance: Seaborn provides polished themes and coordinated palettes by default.
• Data handling: Seaborn works naturally with named DataFrame columns.
• Typical use: Matplotlib is ideal for custom general plotting, while Seaborn is ideal for exploratory statistical visualization.

They are complementary rather than competing tools because Seaborn plots can be refined using Matplotlib.

Seaborn can map additional variables to visual properties:

• hue maps a variable to color.
• style maps a variable to marker shape.
• size maps a variable to marker size.

Example:

sns.scatterplot(
    data=df,
    x="experience",
    y="salary",
    hue="department",
    style="employment_type",
    size="performance_score"
)

This plot shows more than two dimensions of information at once. However, too many categories or encodings can make a graph difficult to read. Clear legends, suitable palettes, and a limited number of categories should therefore be used.

A heatmap represents values in a two-dimensional matrix by mapping numerical magnitude to color.

Heatmaps can represent:

• Correlation matrices.
• Confusion matrices.
• Missing-value patterns.
• Frequency tables.
• Measurements observed across categories and time periods.

A basic Seaborn heatmap is created using:

sns.heatmap(matrix, cmap="viridis", annot=True)

Here, cmap selects the color map and annot=True writes values inside the cells. A color bar explains the relationship between colors and numerical values. Meaningful row labels, column labels, and a suitable color scale are essential for correct interpretation.

A correlation heatmap summarizes pairwise linear relationships among numerical variables.

import seaborn as sns
import matplotlib.pyplot as plt

corr = df.corr(numeric_only=True)
sns.heatmap(corr, annot=True, cmap="coolwarm", vmin=-1, vmax=1)
plt.title("Correlation Matrix")
plt.show()

For variables and , Pearson correlation is:

Interpretation:

• close to indicates strong positive linear association.
• close to indicates strong negative linear association.
• close to indicates weak linear association.
• The diagonal equals because each variable is perfectly correlated with itself.

Correlation does not imply causation and may fail to capture nonlinear relationships.

Scikit-learn, imported as sklearn, is an open-source Python package for machine learning. It offers a consistent estimator interface and integrates with NumPy, SciPy, Pandas, and Matplotlib.

It supports:

• Classification: predicting categories.
• Regression: predicting continuous values.
• Clustering: finding groups in unlabeled data.
• Dimensionality reduction: reducing the number of features.
• Preprocessing: scaling, encoding, and transforming data.
• Model selection: train-test splitting, cross-validation, and hyperparameter tuning.
• Evaluation: metrics such as accuracy, precision, recall, mean squared error, and .
• Pipelines: combining preprocessing and estimation into a repeatable workflow.

A standard supervised-learning workflow contains the following stages:

1. Import and inspect data: Load the dataset and identify quality issues.
2. Separate features and target: Store predictors in and the output in .
3. Split the data: Use train_test_split() to create training and testing subsets.
4. Preprocess features: Impute missing values, encode categories, or scale numerical columns.
5. Select an estimator: Choose a classifier or regressor appropriate for the problem.
6. Train: Call model.fit(X_train, y_train).
7. Predict: Call model.predict(X_test).
8. Evaluate: Compare predictions with y_test using suitable metrics.
9. Improve: Tune hyperparameters or compare alternative models.

The test data must remain separate during training and tuning to provide an unbiased estimate of final generalization performance.

The training set is used to estimate a model's parameters. The testing set contains unseen observations used to evaluate how well the trained model generalizes.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

• test_size=0.2 assigns of observations to testing.
• random_state=42 makes the split reproducible.
• For classification, stratify=y can preserve class proportions.

Evaluating a model on the same data used for training can produce an overly optimistic score. A separate test set gives a more realistic estimate of performance on new data.

Scikit-learn estimators follow a consistent application programming interface:

• fit(X_train, y_train) learns model parameters from training data.
• predict(X_test) generates predicted outputs for new feature values.
• score(X_test, y_test) returns an estimator-specific default performance measure.

Example:

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
result = model.score(X_test, y_test)

For LinearRegression, score() normally returns the coefficient of determination:

Because the meaning of score() varies by estimator, explicit metric functions should be used when a particular evaluation criterion is required.

An end-to-end workflow can be organized as follows:

1. Import libraries and data:

   import pandas as pd
   import seaborn as sns
   import matplotlib.pyplot as plt
   from sklearn.model_selection import train_test_split
   from sklearn.linear_model import LinearRegression
   from sklearn.metrics import mean_squared_error, r2_score

   df = pd.read_csv("data.csv")

2. Inspect the dataset: Use df.head(), df.info(), df.describe(), and df.isnull().sum().

3. Visualize relationships:

   sns.scatterplot(data=df, x="feature", y="target")
   sns.heatmap(df.corr(numeric_only=True), annot=True)
   plt.show()

4. Prepare and split data:

   X = df[["feature"]]
   y = df["target"]
   X_train, X_test, y_train, y_test = train_test_split(
   X, y, test_size=0.2, random_state=42
   )

5. Train and predict:

   model = LinearRegression()
   model.fit(X_train, y_train)
   y_pred = model.predict(X_test)

6. Evaluate:

   mse = mean_squared_error(y_test, y_pred)
   r2 = r2_score(y_test, y_pred)

Mean squared error is calculated as:

A lower MSE indicates smaller prediction errors, while a higher generally indicates that the model explains more target variance. The workflow should also address missing values, categorical variables, data leakage, and model assumptions where applicable.