Unit 6: Spark ML Programming and PySpark - Subjective Questions

Machine learning is a computational approach in which algorithms learn patterns from data and use those patterns to make predictions or decisions without being explicitly programmed for every case.

PySpark supports large-scale machine learning through the following features:

• Distributed processing: Data is divided across multiple cluster nodes and processed in parallel.
• Scalability: PySpark can handle datasets that are too large for a single machine.
• DataFrame-based processing: Structured data can be prepared using Spark DataFrames.
• Integration with Spark ML: PySpark provides APIs for building machine learning pipelines.
• Fault tolerance: Spark can recover lost data partitions through lineage information.
• Parallel algorithms: Algorithms such as linear regression, logistic regression, decision trees, K-means, SVM, and Naive Bayes are available.

Thus, PySpark combines distributed data processing with machine learning capabilities for efficient analysis of big data.

The major techniques of machine learning are:

• Supervised learning: The model learns from labeled data containing input features and known target values. Examples include linear regression for predicting house prices and logistic regression for classifying emails as spam or not spam.
• Unsupervised learning: The model identifies hidden patterns in data without labeled output values. K-means clustering is used to group customers with similar behavior.
• Semi-supervised learning: The model uses a small amount of labeled data along with a large amount of unlabeled data. This is useful when labeling data is expensive.
• Reinforcement learning: An agent learns by interacting with an environment and receiving rewards or penalties. Examples include robotics and game-playing systems.

The choice of technique depends on the availability of labels, the objective of the problem, and the type of data being analyzed.

Spark MLlib is Apache Spark's distributed machine learning library. It provides algorithms, utilities, and tools for scalable machine learning.

The two commonly associated APIs are:

• MLlib RDD-based API: Uses Resilient Distributed Datasets and represents the older machine learning API.
• Spark ML DataFrame-based API: Uses DataFrames and is the recommended API for new applications.

Important differences include:

Spark ML is generally preferred because it provides a consistent interface for preprocessing, model training, evaluation, and deployment.

The important concepts in Spark ML are:

• Feature vector: A numerical vector containing the input variables used by a machine learning algorithm. Spark commonly stores it in a column named features.
• Transformer: Converts one DataFrame into another using the transform() method. Examples include StandardScalerModel, StringIndexerModel, and trained prediction models.
• Estimator: Learns parameters from data using the fit() method and produces a Transformer. Examples include LinearRegression and LogisticRegression.
• Model: A trained form of an estimator that can make predictions on new data.
• Pipeline: A sequence of transformers and estimators that are executed in a defined order.
• Param: A configurable parameter used to control the behavior of an estimator or transformer.

A typical pipeline first indexes categorical data, assembles features, scales the features, trains a model, and produces predictions. This makes the workflow reproducible and reduces errors caused by manually applying preprocessing steps.

A general PySpark machine learning workflow consists of the following steps:

1. Create a Spark session: Initialize SparkSession for accessing Spark functionality.
2. Load the data: Read CSV, JSON, Parquet, or database data into a DataFrame.
3. Clean the data: Handle missing values, duplicate records, and invalid values.
4. Prepare features: Convert categorical variables into numerical values and combine input columns into a features vector using VectorAssembler.
5. Split the data: Divide the dataset into training and testing subsets.
6. Create the algorithm object: Select an estimator such as LinearRegression or DecisionTreeClassifier.
7. Train the model: Call fit() on the training DataFrame.
8. Generate predictions: Apply transform() to the test DataFrame.
9. Evaluate performance: Use suitable evaluators and metrics.
10. Tune and deploy: Adjust parameters, compare models, and save the best model.

This process supports repeatable and distributed machine learning applications.

In linear regression, the predicted value is modeled as:

where is the intercept and are model coefficients.

The difference between the actual and predicted values is called the residual:

The ordinary least squares method estimates the coefficients by minimizing the sum of squared errors:

The objective may also be expressed as mean squared error:

In PySpark, linear regression can be implemented by assembling features into a features column, creating a LinearRegression estimator, fitting it to the training DataFrame, and applying the trained model to test data. Its performance can be evaluated using RMSE and .

Logistic regression is a supervised classification algorithm used to predict the probability that an observation belongs to a class.

For binary classification, the linear combination of features is:

The sigmoid function converts this value into a probability between 0 and 1:

A threshold, commonly , is used to convert the probability into a class label.

In PySpark, LogisticRegression is trained using a DataFrame containing a label column and a vector column named features. The model returns columns such as rawPrediction, probability, and prediction.

Logistic regression is useful for problems such as:

• Predicting whether a transaction is fraudulent.
• Classifying an email as spam or legitimate.
• Predicting whether a customer will leave a service.

It can also support multinomial classification when there are more than two classes.

A decision tree is a supervised learning model that recursively divides data according to feature-based conditions. Each internal node represents a test, each branch represents an outcome, and each leaf represents a predicted class or value.

The algorithm selects a split that improves the purity of the resulting groups. Common impurity measures are:

• Gini impurity:
  
• Entropy:
  

Important Spark ML parameters include:

• maxDepth: Maximum depth of the tree.
• maxBins: Number of bins used for discretizing continuous features.
• minInstancesPerNode: Minimum number of instances required in a node.
• minInfoGain: Minimum information gain required for a split.
• impurity: Criterion such as Gini or entropy.

Decision trees are easy to interpret, but a very deep tree may overfit the training data. Regularization parameters and validation should therefore be used.

K-means is an unsupervised clustering algorithm that divides observations into groups called clusters.

The algorithm works as follows:

1. Choose the number of clusters, .
2. Initialize centroids.
3. Assign every data point to the nearest centroid.
4. Recalculate each centroid as the mean of the points assigned to it.
5. Repeat the assignment and update steps until convergence.

The objective is to minimize the within-cluster sum of squared distances:

In PySpark, numerical columns are combined into a features vector, and the KMeans estimator is fitted to the DataFrame. The resulting model assigns a cluster ID to each record.

Important parameters include k, maxIter, tol, and seed. The appropriate value of can be selected using domain knowledge, the elbow method, or a silhouette score.

A Support Vector Machine, or SVM, is a supervised classification algorithm that finds a decision boundary separating classes with the largest possible margin.

The decision boundary can be written as:

For a linearly separable dataset, the SVM attempts to satisfy:

while minimizing:

The observations closest to the boundary are called support vectors. They determine the location of the optimal hyperplane.

For non-separable data, slack variables and a regularization parameter allow some classification errors. A larger penalizes errors more heavily and may produce a narrower margin.

A kernel function can map data into a higher-dimensional space so that nonlinear relationships become separable. Spark ML provides LinearSVC for linear support vector classification. Feature scaling is usually important because SVMs are sensitive to the relative magnitudes of features.

Naive Bayes is a probabilistic classification algorithm based on Bayes' theorem:

Here, is a class and is the observed feature vector. The classifier predicts the class with the highest posterior probability.

The naive assumption is that features are conditionally independent given the class. Therefore:

The predicted class is:

In Spark ML, the NaiveBayes estimator can be trained using a label column and a numerical feature vector. It is commonly used for text classification, document categorization, and spam detection.

Advantages include fast training, simple implementation, and good performance with high-dimensional data. Its main limitation is that the conditional independence assumption may not hold in real-world datasets.

Linear regression models a continuous target, whereas logistic regression predicts class probabilities. Decision trees use hierarchical feature splits and are easy to interpret. SVM attempts to maximize the class-separating margin. Naive Bayes is probabilistic and assumes conditional independence among features. K-means differs from all the others because it does not require labeled training data.

Model evaluation measures how well a trained model performs on unseen data. Evaluation should generally be performed on a test set or through cross-validation rather than only on the training set.

Regression metrics include:

• RMSE: Measures the square root of the average squared prediction error.
• MAE: Measures the average absolute prediction error.
• : Measures the proportion of target variance explained by the model.

Classification metrics include:

• Accuracy
• Precision
• Recall
• F1-score
• Area under the ROC curve
• Area under the precision-recall curve

Spark ML provides evaluators such as RegressionEvaluator, MulticlassClassificationEvaluator, and BinaryClassificationEvaluator.

The metric must match the objective. For example, RMSE is useful when large regression errors should receive greater penalty, whereas recall is important when missing positive cases is costly, such as in disease detection.

A confusion matrix summarizes the results of a classification model by comparing actual and predicted class labels.

Its four components are:

• True Positive (TP): A positive instance correctly classified as positive.
• True Negative (TN): A negative instance correctly classified as negative.
• False Positive (FP): A negative instance incorrectly classified as positive.
• False Negative (FN): A positive instance incorrectly classified as negative.

The main metrics are:

• Accuracy:
  
• Precision:
  
• Recall:
  
• F1-score:
  

Accuracy measures overall correctness. Precision measures the reliability of positive predictions, while recall measures how many actual positive cases were detected. F1-score balances precision and recall.

Let be the actual value, be the predicted value, and be the mean of actual values.

• Mean Absolute Error:
  
  MAE represents the average absolute prediction error in the same units as the target. It is less sensitive to outliers than RMSE.

• Root Mean Squared Error:
  
  RMSE is also expressed in target units, but it penalizes large errors more strongly because errors are squared.

• Coefficient of determination:
  
  indicates how much variance is explained by the model. A value close to 1 generally indicates a good fit, although it does not directly show the prediction error in target units.

MAE is easy to interpret, RMSE highlights large mistakes, and is useful for assessing explanatory power. None of these metrics alone is sufficient for every application.

A correlation heat map is a visual representation of the pairwise correlation coefficients between variables. Correlation is commonly measured using Pearson's coefficient:

The value of lies between and :

• indicates perfect positive linear correlation.
• indicates perfect negative linear correlation.
• indicates no linear correlation.

In a heat map, colors represent the strength and direction of correlations. It can be used to:

• Identify features strongly related to the target variable.
• Detect redundant features that are highly correlated with one another.
• Discover possible multicollinearity in regression models.
• Understand relationships before selecting an algorithm.
• Support dimensionality reduction and feature engineering.

Correlation does not prove causation, and Pearson correlation may fail to detect nonlinear relationships. Therefore, the heat map should be combined with domain knowledge and other exploratory techniques.

Feature preprocessing converts raw data into a form that machine learning algorithms can use effectively.

• Feature vector assembly: VectorAssembler combines several numerical columns into a single vector column named features.
• Categorical encoding: StringIndexer converts string categories into numerical indices. OneHotEncoder can then represent categories without implying an ordinal relationship.
• Scaling: StandardScaler transforms features so they have comparable scales. Standardization is commonly expressed as:
  
• Missing-value handling: Missing values can be replaced using imputation or removed when appropriate.
• Outlier handling: Extreme values may be investigated, transformed, or removed depending on the problem.

Scaling is especially important for SVM, logistic regression, and K-means because these methods depend on distances or coefficient magnitudes. Decision trees are generally less sensitive to feature scaling. Spark ML transformers allow these preprocessing operations to be included in a pipeline.

A classification pipeline combines data preparation and model training into a single sequence of stages.

A typical pipeline contains:

1. String indexing: Convert the target column and categorical feature columns into numerical indices.
2. One-hot encoding: Represent nominal categorical variables as binary vectors when required.
3. Vector assembly: Combine all prepared feature columns into a features vector.
4. Optional scaling: Standardize the feature vector for algorithms that are scale-sensitive.
5. Classification estimator: Add an estimator such as LogisticRegression, DecisionTreeClassifier, LinearSVC, or NaiveBayes.
6. Pipeline creation: Place all stages in the correct order.
7. Model fitting: Call fit() on the training DataFrame.
8. Prediction: Call transform() on validation or test data.
9. Evaluation: Use classification metrics such as accuracy, precision, recall, F1-score, or area under the ROC curve.

Pipelines ensure that the same transformations are applied consistently during both training and prediction.

Overfitting occurs when a model learns noise and details specific to the training data. It usually has low training error but high test error. Underfitting occurs when a model is too simple to capture the underlying pattern, causing high error on both training and test data.

Methods to improve generalization include:

• Split data into training, validation, and test sets.
• Use cross-validation to estimate performance more reliably.
• Apply regularization in linear and logistic regression.
• Limit maxDepth and increase minInstancesPerNode for decision trees.
• Select an appropriate value of in K-means.
• Scale features when required by the algorithm.
• Remove irrelevant or redundant features.
• Use hyperparameter tuning with ParamGridBuilder and CrossValidator.
• Use class balancing or suitable evaluation metrics for imbalanced classification.

The final model should be selected based on validation performance, while the test set should be reserved for final unbiased evaluation.

For linear regression, the mean squared error objective can be written as:

where:

The gradient of the objective with respect to the coefficient vector is:

The gradient descent update rule is:

where is the learning rate.

In distributed computing, the dataset is partitioned across worker nodes. Each worker can calculate a partial gradient from its partition. These partial gradients are aggregated to obtain the full gradient, after which the coefficient vector is updated. This allows large datasets to be processed in parallel. Spark ML hides much of this distributed implementation while providing an estimator-based API for training the model.