Unit 5 - Practice Quiz

The 'Population' is the complete and entire set of items, individuals, or events that share a common characteristic and are the subject of a statistical inquiry. A sample is a subset of the population.

Sampling allows us to study a large population efficiently and cost-effectively by analyzing a smaller, representative sample. It's often impractical or impossible to study the entire population.

The null hypothesis () represents the default or status-quo assumption. The goal of a hypothesis test is to determine if there is enough evidence to reject this default assumption in favor of an alternative hypothesis.

A 95% confidence interval provides a plausible range for an unknown population parameter (like the mean). It means that if we were to repeat the sampling process many times, 95% of the calculated confidence intervals would contain the true population parameter.

A correlation coefficient of -1.0 means that as one variable increases, the other variable decreases in a perfectly predictable, linear fashion.

MLE is a method for estimating the parameters of a statistical model. It works by finding the parameter values that make the observed data most probable under the assumed model.

A loss function (or cost function) quantifies the model's error for a given set of parameters. The goal of the training process is to find parameters that minimize this loss.

The gradient of a function points in the direction of the steepest ascent. To find a minimum, Gradient Descent moves in the opposite direction (the negative gradient), effectively walking 'downhill' on the loss surface.

A high learning rate causes the algorithm to take very large steps. These steps can be so large that they jump over the minimum of the loss function, leading to oscillation or divergence where the loss actually increases.

The global minimum represents the best possible solution, corresponding to the set of model parameters that results in the absolute lowest loss. It is the primary target of the optimization process.

A local minimum is a 'valley' in the loss surface. An optimization algorithm might get 'stuck' in a local minimum, thinking it has found the best solution when a better one (the global minimum) exists elsewhere.

Standard (Batch) Gradient Descent computes the gradient using the entire training dataset for one parameter update. SGD uses just one example (or a 'mini-batch'), making each update much faster but also noisier. This trade-off is highly effective for large datasets.

Convex functions have a unique 'bowl' shape. This property guarantees that any minimum you find is the absolute global minimum, which greatly simplifies the optimization problem as you don't have to worry about getting stuck in suboptimal local minima.

A saddle point is a critical point that is a maximum along one direction and a minimum along another, resembling the shape of a horse's saddle. Optimization algorithms can sometimes slow down or get stuck at saddle points.

Momentum adds a fraction of the previous update step to the current one. This helps the optimizer 'build up speed' in a consistent direction, allowing it to navigate ravines and overcome small local minima more effectively.

Adam (Adaptive Moment Estimation) maintains a per-parameter learning rate that is adapted based on the historical first and second moments of the gradients, making it very effective and popular in practice.

Optimization is the core process of training. It involves systematically adjusting the model's internal parameters (weights and biases) to reduce the error (loss) on the training data, thereby 'learning' the patterns in the data.

Mean Squared Error () is a standard loss function for regression tasks. It measures the average of the squares of the errors between the predicted and actual values. The other options are typically used for classification tasks.

A correlation of 0 specifically indicates the absence of a linear association. It is important to remember that there could still be a strong non-linear relationship between the variables.

A very small learning rate causes the optimizer to take tiny steps towards the minimum. This leads to very slow convergence and increases the risk of getting trapped in a shallow local minimum or stopping prematurely.

The p-value (0.02) is less than the significance level (). This indicates that the observed result is statistically significant. Therefore, we reject the null hypothesis () in favor of the alternative hypothesis (), concluding that the new algorithm's performance is significantly better.

Batch Gradient Descent computes the gradient using the entire dataset, giving a precise direction towards the minimum of the loss function. SGD uses only a small random sample (a mini-batch or a single example), so the calculated gradient is just an estimate. This estimate has high variance, causing the optimization path and the loss to fluctuate.

Momentum introduces a 'velocity' term which is an exponentially decaying moving average of past gradients. In a loss landscape with long, narrow ravines, the gradient oscillates across the narrow direction. Momentum dampens these oscillations and accelerates movement along the consistent, downhill direction, leading to faster convergence.

For a Gaussian (Normal) distribution, the Maximum Likelihood Estimate (MLE) for the mean parameter is simply the sample mean of the data. The sample mean is calculated as .

A learning rate that is too large will cause the parameter updates to be too aggressive. The optimizer will repeatedly jump over the minimum of the loss function, moving to a point where the loss is even higher. This leads to a diverging (exploding) loss.

A confidence interval provides a range of plausible values for a population parameter (in this case, the true mean inference time). The 95% confidence level means that if we were to repeat the sampling process many times and construct an interval each time, 95% of those intervals would contain the true population mean. It does not describe the probability of a single data point or a sample mean.

A saddle point is a critical point (gradient is zero) that is a minimum along some dimensions and a maximum along others. This corresponds to the Hessian matrix (matrix of second derivatives) having both positive and negative eigenvalues. In high-dimensional spaces, saddle points are far more common than local minima and are a significant challenge for optimization.

Adam (Adaptive Moment Estimation) combines the key ideas of two other popular optimizers. It uses the moving average of the gradient (first moment), similar to Momentum, to accelerate convergence. It also uses the moving average of the squared gradient (second moment), similar to RMSProp, to provide an adaptive, per-parameter learning rate.

The loss function for logistic regression is convex. For a convex function, any local minimum is also a global minimum, and there are no other local minima or saddle points to get stuck in. Therefore, a simple optimization algorithm like Gradient Descent is guaranteed to converge to the unique global minimum.

Mean Squared Error (MSE), , penalizes large errors quadratically. This means outliers have a disproportionately large influence on the gradient and the model's training. Mean Absolute Error (MAE), , has a penalty that grows linearly with the error, making it less sensitive to outliers. Huber Loss is a combination of both, acting like MSE for small errors and MAE for large errors, providing a good balance of sensitivity and robustness.

The Pearson correlation coefficient measures the strength and direction of a linear relationship. A value close to -1 indicates a strong negative linear relationship, meaning that as one variable increases, the other tends to decrease in a linear fashion. It is important to remember that correlation does not imply causation.

The defining characteristic of batch Gradient Descent is that it uses the entire dataset to compute the gradient of the loss function at each step. This provides a very accurate estimate of the gradient, leading to a stable convergence path, but it is computationally very expensive for large datasets.

The population is the entire group that you want to draw conclusions about (all 1,000,000 users). The sample is the specific group that you will collect data from (the 1,000 surveyed users). The sample is a subset of the population.

Both Adagrad and RMSProp are adaptive learning rate methods. They maintain a per-parameter learning rate that is adjusted based on past gradients. Adagrad accumulates all past squared gradients, while RMSProp uses an exponentially decaying average of squared gradients. This mechanism effectively gives frequently updated parameters a smaller learning rate and infrequently updated parameters a larger one.

A Type I error occurs when you incorrectly reject a true null hypothesis. In this case, the null hypothesis is that there is no effect. Rejecting it means you conclude there is an effect. So, a Type I error is a 'false positive': concluding the new design is better (or different) when, in reality, it isn't.

In the initial stages of training, a larger learning rate helps the model make significant progress quickly. However, as the parameters get closer to an optimal value, a large learning rate can cause the optimizer to bounce around the minimum without settling. Decreasing the learning rate allows for smaller, more precise adjustments, facilitating convergence to a good solution.

The likelihood of an i.i.d. dataset is the product of individual probabilities. For large datasets, this product can become extremely small, leading to numerical underflow. Since the logarithm is a monotonically increasing function, maximizing is equivalent to maximizing . The logarithm turns the product into a sum (), which is much easier to differentiate and work with numerically.

Batch Gradient Descent and methods like Newton's Method require loading the entire dataset into memory to compute the gradient or Hessian matrix for each step. This is infeasible for very large datasets. SGD and Mini-batch GD process the data in small chunks, allowing the model to be trained without ever needing to hold the full dataset in memory at once.

Non-convex functions can have multiple local minima, which are points that are optimal within a neighborhood but not globally. Gradient-based methods follow the steepest descent and will stop when they reach a point where the gradient is zero. In a non-convex landscape, this point could be a local minimum or a saddle point, with no guarantee that it is the best possible solution (the global minimum).

Second-order methods use the Hessian matrix (matrix of second partial derivatives) to get information about the curvature of the loss surface. For a neural network with parameters, the Hessian is an matrix. Calculating, storing, and inverting this matrix is computationally prohibitive for modern deep learning models where can be in the millions or billions. The cost is approximately , making it impractical.

This question addresses the multiple comparisons problem. The probability of not making a Type I error (a false positive) in one test is . If the tests are independent, the probability of not making a Type I error in any of the 100 tests is . Therefore, the probability of making at least one Type I error is , or about 99.4%. This highlights the danger of running many tests without correcting the significance level (e.g., using Bonferroni correction).

The likelihood function is . For a uniform distribution , the probability density function is for , and 0 otherwise. This means but only if all . If any , the likelihood is 0. To maximize the likelihood, we need to make as small as possible to maximize , but it must also satisfy the constraint for all . Therefore, the smallest possible value for that satisfies this is .

The first and second moment estimates ( and ) are calculated as exponential moving averages. Since they are initialized to zero, they are heavily biased towards zero at the beginning of training, before they have had enough steps to 'warm up'. The bias correction terms, dividing by and respectively, counteract this initialization bias. This allows for larger, more reasonable steps at the beginning of training, preventing the updates from being artificially small.

The Hessian has one positive eigenvalue (2) and one negative eigenvalue (-3), confirming a saddle point. The direction corresponds to a local minimum (positive curvature) and corresponds to a local maximum (negative curvature). Vanilla GD will slow down dramatically as it approaches the critical point because the gradient approaches zero. It will move away from the saddle point in the direction (down the 'hill') but converge towards it in the direction (into the 'valley'), effectively getting stuck. Momentum, however, accumulates velocity. While the gradient in the direction might become small near the saddle, the accumulated momentum will carry it past the point, causing it to 'overshoot' and escape.

While MAE is more robust than MSE, Huber Loss is designed to combine the best of both. For small errors, it behaves like MSE (quadratic), which has nice properties like a unique minimum and a continuously differentiable gradient. For large errors (outliers), it behaves like MAE (linear). This transition to linear for large errors prevents the gradient from becoming excessively large due to an outlier, as would happen with MSE's quadratic penalty. This makes the training process less sensitive to and more robust against outliers than both pure MSE and MAE (which has a discontinuous gradient at zero).

The variance in the gradient estimates from mini-batches acts as a form of noise. This noise can prevent the optimizer from getting stuck in sharp local minima, which often do not generalize well to unseen data. The optimizer might 'bounce around' inside a minimum, and if the minimum is very sharp and narrow, the noise can be large enough to 'kick' the parameters out of it and into a wider, flatter basin. Flatter minima are associated with better generalization performance, as small perturbations in the input data are less likely to cause large changes in the output.

This is a subtle but crucial point in frequentist statistics. The true mean accuracy is considered a fixed, unknown constant. It either is or is not in the interval [0.88, 0.92]. The probability is 1 or 0, not 95%. The 95% confidence level refers to the procedure of creating the interval. It means that if we were to draw many random samples and construct a confidence interval for each, 95% of those intervals would capture the true mean. Our confidence is in the method, not in the specific interval calculated from a single sample. Therefore, stating there is a 95% probability that the true mean is in this specific interval is an incorrect, Bayesian interpretation of a frequentist concept.

To determine convexity, we analyze the Hessian matrix. The gradient is . The Hessian is . For the function to be convex, the Hessian must be positive semidefinite everywhere. This requires its principal minors to be non-negative. The first minor is , which is always true. The second minor is the determinant: . We need . However, if we test the point , the determinant is -16, which is negative. Since the Hessian is not positive semidefinite everywhere (e.g., at the origin), the function is non-convex.

In an ill-conditioned problem (like a long, narrow ravine), the gradient is very large in the direction perpendicular to the valley floor (the steep walls) and very small along the valley floor. A learning rate that is too high will cause the updates to overshoot the minimum across the narrow dimension, leading to oscillations from one wall to the other. At the same time, the gradient component along the valley is small, so even with this 'high' learning rate, progress towards the actual minimum is excruciatingly slow. This zigzagging behavior is a classic sign of a suboptimal learning rate in an ill-conditioned landscape.

The Pearson correlation coefficient specifically measures the strength and direction of a linear relationship between two variables. A value of 0 indicates the absence of a linear association. However, it does not rule out the possibility of a strong non-linear relationship. For example, if Y = X² for X in [-1, 1], the variables are perfectly dependent, but their Pearson correlation is 0. Statistical independence is a much stronger condition that implies a correlation of 0, but the reverse is not true.

For a function that is L-smooth (meaning its gradient is Lipschitz continuous with constant L), the convergence of Gradient Descent is guaranteed if the learning rate is small enough to prevent overshooting. The standard theoretical guarantee for convergence requires the learning rate to be in the range . Choosing can lead to divergence. The condition of -strong convexity ensures that there is a unique global minimum and that the convergence rate will be linear (geometric), but the condition on the learning rate for guaranteed convergence is primarily determined by the smoothness constant L.

Momentum's core idea is to accelerate SGD in the relevant direction and dampen oscillations. It does this by adding a fraction of the previous update vector to the current one (a moving average of gradients). It uses a single global learning rate. RMSProp, on the other hand, maintains a moving average of the squared gradients for each parameter. It then divides the learning rate by the square root of this average. This means parameters with large gradients will have their effective learning rate reduced, and parameters with small gradients will have their effective learning rate increased. This per-parameter adaptation makes RMSProp an adaptive learning rate method, which is fundamentally different from Momentum's velocity-based approach.

The likelihood for a single data point in logistic regression, where and , is given by the Bernoulli probability mass function: . The log-likelihood is . To use gradient descent, we minimize the negative log-likelihood, which is . This is precisely the definition of the Binary Cross-Entropy loss. Therefore, performing MLE for the parameters of a logistic regression model is equivalent to minimizing the cross-entropy loss.

With extreme class imbalance, a simple random sample might contain very few or even zero instances of the minority class (fraud). A model trained on such data will learn that the best strategy is to always predict the majority class, achieving high accuracy but being useless for fraud detection. This is a form of model bias. Stratified sampling is the most appropriate technique here because it ensures that the proportions of the classes (strata) in the sample are the same as in the population. You can even use it to oversample the minority class and/or undersample the majority class to create a more balanced training set, forcing the model to learn the patterns of the rare fraud cases.

A p-value of 0.04 is less than the significance level , so we reject the null hypothesis and conclude the result is statistically significant. The 95% CI not containing zero ([0.001, 0.009]) corroborates this. However, statistical significance does not imply practical significance. The confidence interval shows that the true improvement of B over A could be as small as 0.1 percentage points (0.001). Whether this tiny increase in CTR is worth the cost of implementing the new design is a business decision, not a statistical one. The result is statistically robust but may be practically unimportant.

For SGD to be guaranteed to converge to a local minimum, the learning rates must satisfy the Robbins-Monro conditions: and . The first condition ensures the learning rate is large enough to traverse any distance and not get stuck prematurely. The second condition ensures that the learning rate eventually becomes small enough for the algorithm to converge, damping the inherent noise of the stochastic gradients. A learning rate schedule like satisfies these conditions. Using a fixed learning rate with SGD will cause the optimizer to bounce around the minimum indefinitely due to the gradient noise.

The product of two convex functions is not generally convex. For a simple counterexample, consider and . Both are convex. Their product is . The second derivative is . This is negative for some values of x (e.g., at x=1, ), so is not convex. The sum, pointwise maximum, and composition with a non-decreasing outer function are all operations that preserve convexity.

Research has shown that for high-dimensional functions, most local minima are of high quality (have a loss close to the global minimum). The real problem is the proliferation of saddle points. In high dimensions, for a point to be a local minimum, the Hessian must be positive semi-definite (all non-negative eigenvalues). For it to be a saddle point, it just needs at least one positive and one negative eigenvalue. The probability of the latter is much higher. The most difficult saddle points are those where there are many directions of positive curvature (valleys) and only one or a few directions of negative curvature (hills). First-order methods like SGD see very small gradients along the 'valley' directions and struggle to find the 'hill' direction to escape, leading to long plateaus in the training curve.

In this scenario: 1) With Momentum, the consistently large gradient will cause the velocity term () to grow, leading to large, accelerated steps. 2) With RMSProp, the consistently large gradient will cause the squared gradient average () to grow, leading to a large denominator and thus smaller, dampened steps. 3) Adam combines both. The numerator (the momentum term ) will be large. However, the denominator (the RMSProp term ) will also become very large due to the consistently large gradient. The per-parameter scaling effect of the denominator will dominate, significantly reducing the effective learning rate for this specific weight. This dampening effect will be more pronounced than in RMSProp alone because the large gradients are squared in the denominator's calculation, making it grow faster than the linear growth in the numerator's momentum term. Therefore, Adam will take smaller steps for this stable, high-gradient parameter compared to either of its parent algorithms.

Newton's method updates parameters using the formula , where H is the Hessian matrix. For a model with parameters, the Hessian is an matrix. Storing this matrix requires memory, and computing its inverse requires operations. For a typical deep learning model where can be in the millions or billions, both of these requirements are computationally infeasible. While quasi-Newton methods like L-BFGS exist to approximate the inverse Hessian, even they are often too costly compared to the simple, scalable, and surprisingly effective first-order methods like SGD and its variants.