Unit 6: Spark ML Programming and PySpark

I. Machine Learning Foundations

Machine learning enables computers to discover patterns from data and make predictions without every decision rule being explicitly programmed. PySpark provides Python access to Apache Spark’s distributed processing engine, allowing these methods to scale across a cluster.

Defining characteristics:

• Data-driven learning: Models estimate relationships from examples containing features and, in supervised learning, labels.
• Distributed execution: Spark partitions data across worker nodes and performs transformations in parallel.
• General workflow: Data collection → preprocessing → feature engineering → training → evaluation → deployment.
• Core assumption: Training data should represent the population on which the model will operate.
• Primary objective: A model must generalize to unseen data rather than merely memorize its training records.

A. Introduction to machine learning and PySpark

Machine learning uses statistical and computational methods to learn a mapping or structure from data, while PySpark executes the required operations through Python APIs.

• Basic terminology:
  • Features: Input variables, such as age, income, or temperature.
  • Label: Target value to be predicted, such as price or class.
  • Model: Learned mathematical relationship between features and output.
  • Training set: Data used to estimate model parameters.
  • Test set: Unseen data used for final performance measurement.
• PySpark architecture: A Python driver creates a SparkSession; tasks are scheduled on executors that process partitions.
• Data representation: Spark ML primarily uses distributed DataFrames, where rows are observations and columns contain labels, features, or predictions.
• Train-test division: The split should occur before model fitting to prevent information leakage.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MLApp").getOrCreate()
train, test = data.randomSplit([0.8, 0.2], seed=42)

B. Techniques of machine learning

Machine-learning techniques are distinguished by the type of feedback available and the objective being optimized.

• Supervised learning: Learns from labelled examples (x, y).
  • Regression: Predicts continuous values, such as sales.
  • Classification: Predicts categories, such as spam or non-spam.
• Unsupervised learning: Finds patterns without labels.
  • Clustering: Groups similar records; K-means is a standard example.
  • Dimensionality reduction: Compresses features while retaining useful information.
• Semi-supervised learning: Combines a small labelled dataset with a larger unlabelled dataset.
• Reinforcement learning: An agent learns actions from rewards and penalties; it is not a central Spark ML API focus.
• Generalization control: Cross-validation and regularization reduce overfitting, while suitable model complexity prevents underfitting.

II. Spark Machine-Learning Framework

Spark supplies scalable abstractions for transforming features, fitting algorithms, constructing workflows, and evaluating predictions.

A. Introduction to Spark MLlib

MLlib is Apache Spark’s machine-learning library for distributed model training and data processing.

• Two API families:
  1. spark.mllib: Older RDD-based API, currently in maintenance mode.
  2. spark.ml: Preferred DataFrame-based API with pipelines and parameter management.
• Capabilities: Classification, regression, clustering, recommendation, feature extraction, model selection, and evaluation.
• Scalability: Computation is distributed across partitions, although not every algorithm scales equally for every dataset.
• Typical imports:

from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.classification import LogisticRegression

B. Key concepts of Spark ML

Spark ML organizes machine-learning workflows around DataFrames, transformers, estimators, parameters, and pipelines.

• Transformer: Converts one DataFrame into another using transform().
  • A fitted model is a transformer because it adds a prediction column.
• Estimator: Learns from a DataFrame using fit() and produces a transformer.
• Feature vector: Most algorithms expect a single vector column conventionally named features.
• VectorAssembler: Combines numeric input columns into one feature vector.
• Pipeline: Chains processing stages so that identical transformations are applied during training and prediction.
• Parameters: Options such as maxIter, regParam, labelCol, and featuresCol configure an estimator.
• Model selection: CrossValidator or TrainValidationSplit compares parameter combinations using an evaluator.

C. Spark ML algorithms using PySpark

PySpark algorithms generally follow the sequence of assembling features, fitting an estimator, transforming test data, and evaluating predictions.

• Standard pattern:

assembler = VectorAssembler(
    inputCols=["x1", "x2"], outputCol="features"
)
prepared = assembler.transform(data)

model = estimator.fit(prepared)
predictions = model.transform(prepared)

• Input requirement: Categorical features usually require StringIndexer and, where appropriate, OneHotEncoder.
• Pipeline advantage: Fitting preprocessing inside a pipeline limits inconsistent transformations and data leakage.
• Persistence: Models can be stored with model.write().overwrite().save(path) and later loaded for prediction.

III. Regression Models

Regression models estimate a numerical target from one or more explanatory variables.

A. Linear regression

Linear regression models the target as a weighted sum of features and an error term.

ŷ = β₀ + β₁x₁ + β₂x₂ + ... + βₚxₚ

Here, ŷ is the prediction, β₀ is the intercept, βⱼ is a learned coefficient, and xⱼ is feature j.

• Objective: Ordinary least squares minimizes the sum of squared residuals, where a residual is yᵢ − ŷᵢ.
• Regularization: L1 encourages zero coefficients; L2 shrinks coefficients; elastic net combines both.
• Assumptions: Approximate linearity, independent errors, constant error variance, and limited harmful multicollinearity.
• PySpark implementation:

from pyspark.ml.regression import LinearRegression

lr = LinearRegression(labelCol="label", featuresCol="features")
model = lr.fit(train)
predictions = model.transform(test)

• Limitation: Strong nonlinear relationships require transformed features or a nonlinear algorithm.

IV. Classification Models

Classification assigns observations to discrete classes and may also estimate class probabilities.

A. Logistic regression

Logistic regression models the probability of a class through the logistic function.

P(y = 1 | x) = 1 / (1 + e^−z), where z = β₀ + βᵀx

• Decision rule: Binary prediction is commonly class 1 when probability exceeds a threshold such as 0.5.
• Training objective: Parameters minimize log loss, optionally with L1 or L2 regularization.
• PySpark class: pyspark.ml.classification.LogisticRegression.
• Outputs: rawPrediction, probability, and prediction.
• Strength: Coefficient signs show how features affect log-odds; however, the decision boundary is linear.

B. Decision tree

A decision tree recursively divides records using feature-based rules that increase the purity of resulting groups.

• Classification criteria: Gini impurity or entropy measures class mixture.
• Regression criterion: Variance reduction selects splits for continuous targets.
• Controls: maxDepth, maxBins, and minInstancesPerNode restrict tree complexity.
• PySpark classes: DecisionTreeClassifier and DecisionTreeRegressor.
• Advantages: Handles nonlinear interactions, needs little feature scaling, and yields interpretable rules.
• Limitation: Deep trees can overfit and may change greatly after small data variations.

C. SVM

A support vector machine finds a separating hyperplane with the largest possible margin between classes.

Decision function: f(x) = wᵀx + b

Here, w is the weight vector, x is the feature vector, and b is the intercept.

• Support vectors: Training points nearest the decision boundary determine the margin.
• Soft margin: Parameter regParam balances a wider margin against classification errors.
• Scaling: Standardization is important because feature magnitude affects distances and coefficients.
• PySpark implementation: LinearSVC provides linear binary classification.
• Limitation: Spark’s standard LinearSVC does not directly provide nonlinear kernel SVM or multiclass classification.

D. Naive Bayes

Naive Bayes applies Bayes’ theorem while conditionally treating features as independent within each class.

P(C | x) ∝ P(C) × ∏ P(xⱼ | C)

Here, C is a class, xⱼ is feature j, P(C) is the prior, and P(xⱼ|C) is a likelihood.

• Model types: Spark supports multinomial, Bernoulli, complement, and Gaussian variants subject to API version.
• Common use: Multinomial Naive Bayes is effective for non-negative word-count or term-frequency vectors.
• Smoothing: A positive smoothing value prevents zero-frequency probabilities.
• Strength: Training is fast for high-dimensional sparse data.
• Limitation: Correlated features violate the independence assumption and can weaken probability estimates.

V. Clustering

Clustering discovers groups of similar observations without requiring labelled outputs.

A. K-means

K-means partitions observations into K clusters by minimizing squared distance from each point to its assigned centroid.

Objective = Σᵢ ||xᵢ − μc(i)||²

Here, xᵢ is observation i, μc(i) is its assigned centroid, and ||·||² is squared Euclidean distance.

• Iteration: Assign points to nearest centroids, recompute centroid means, and repeat until convergence.
• PySpark class: pyspark.ml.clustering.KMeans.
• Parameters: k sets cluster count; seed supports reproducibility; maxIter limits iterations.
• Evaluation: Smaller silhouette-distance separation is undesirable; a higher silhouette score generally indicates compact, separated clusters.
• Limitations: Results depend on feature scale, outliers, initialization, and the chosen value of K.

VI. Model Evaluation and Diagnostic Measures

Evaluation quantifies predictive quality on validation or test data and helps compare models fairly.

A. Evaluation and performance matrix

Evaluation uses task-appropriate metrics rather than relying on training accuracy alone.

• Regression evaluation: RegressionEvaluator supports RMSE, MSE, MAE, explained variance, and R².
• Classification evaluation: BinaryClassificationEvaluator supports area under ROC or precision-recall curves.
• Multiclass evaluation: MulticlassClassificationEvaluator provides accuracy, weighted precision, weighted recall, and F1.
• Clustering evaluation: ClusteringEvaluator commonly computes silhouette score.
• Reliable procedure: Fit on training data, tune on validation folds, and report final results once on untouched test data.

B. Confusion matrix

A confusion matrix counts correct and incorrect classification outcomes by actual and predicted class.

• Accuracy: (TP + TN) / (TP + TN + FP + FN).
• Precision: TP / (TP + FP); measures reliability of positive predictions.
• Recall: TP / (TP + FN); measures detection of actual positives.
• F1-score: 2 × Precision × Recall / (Precision + Recall).
• Interpretation: For rare fraud cases, recall and precision are usually more informative than accuracy.

C. R2

R² measures the proportion of target variation explained by a regression model relative to predicting the mean.

R² = 1 − [Σ(yᵢ − ŷᵢ)² / Σ(yᵢ − ȳ)²]

• Symbols: yᵢ is actual value, ŷᵢ is prediction, and ȳ is the observed mean.
• Interpretation: 1 is perfect fit; 0 matches mean prediction; negative test R² indicates performance worse than the mean baseline.
• Caution: A high R² does not establish causality or guarantee small prediction errors.

D. RMSE

RMSE is the square root of average squared prediction error.

RMSE = √[(1/n) Σ(yᵢ − ŷᵢ)²]

• Meaning: n is the number of observations; RMSE has the same unit as the target.
• Sensitivity: Squaring gives large errors greater influence.
• Use: It is suitable when substantial errors should receive a strong penalty.
• Comparison rule: Lower RMSE indicates better performance on the same dataset and target scale.

E. MAE

MAE is the average absolute difference between actual and predicted values.

MAE = (1/n) Σ|yᵢ − ŷᵢ|

• Interpretation: An MAE of 3 means predictions differ from actual values by three target units on average.
• Robustness: MAE is less sensitive to extreme errors than RMSE.
• Limitation: It does not strongly distinguish occasional very large errors from moderate errors.
• Comparison: If RMSE greatly exceeds MAE, the model probably makes some unusually large errors.

F. Correlation heat map

A correlation heat map visually displays pairwise relationships among numerical features.

• Coefficient: Pearson correlation ranges from −1 to +1; sign indicates direction and magnitude indicates linear association strength.
• Spark calculation: pyspark.ml.stat.Correlation.corr() computes a matrix from a vector column.
• Visualization: The small matrix may be converted to local form and plotted with Matplotlib or Seaborn.
• Application: Highly correlated predictors can reveal redundancy or multicollinearity before linear modelling.
• Cautions: Correlation does not imply causation, misses nonlinear relationships, and a large feature matrix should not be collected to the driver without memory checks.