Unit 4 - Practice Quiz

A scalar is a single numerical quantity, like 5, -3.14, or 1/2. It is used to scale vectors and matrices but does not have a direction or multiple components itself.

A matrix is a rectangular grid of numbers. For example, a 3x2 matrix has 3 rows and 2 columns. It is a fundamental object in linear algebra.

A vector is a 1D array or an ordered list of numbers, often representing a point in space or a quantity with both magnitude and direction.

A tensor is a multi-dimensional array. A 0D tensor is a scalar, a 1D tensor is a vector, a 2D tensor is a matrix, and tensors can have 3 or more dimensions.

In the characteristic equation , is the eigenvector and the scalar is the corresponding eigenvalue. The equation means that multiplying the matrix by its eigenvector scales the eigenvector by the scalar eigenvalue .

By definition, the probability of any event must be a non-negative value, with 0 representing an impossible event and 1 representing a certain event. All probabilities fall within the inclusive range [0, 1].

A random variable assigns a numerical value to each possible outcome of a random process. For instance, the number of heads in three coin flips is a random variable that can take values {0, 1, 2, 3}.

A probability distribution provides the probabilities for all possible values that a random variable can take. For example, for a fair die, the distribution would assign a probability of 1/6 to each outcome from 1 to 6.

The mean is a measure of the central location of a distribution. It is calculated by summing all values and dividing by the number of values.

Variance quantifies the variability or dispersion of a set of data points. A low variance indicates that the data points tend to be close to the mean, while a high variance indicates that they are spread out.

Covariance measures the joint variability of two random variables. A positive value indicates a direct relationship, while a negative value indicates an inverse relationship.

This is the definition of conditional probability, denoted as . It is the probability of one event under the condition of another.

Bayes' Theorem provides a way to revise existing predictions or theories (prior probabilities) given new or additional evidence (likelihood), resulting in a new, updated probability (posterior probability).

Probability quantifies the chance of data occurring given a model (e.g., probability of heads is 0.5 for a fair coin). Likelihood quantifies how good a model is at explaining the data you have observed (e.g., given 8 heads in 10 flips, what is the likelihood the coin is fair?).

The core definition of a function is that for any given input from its domain, it produces a single, uniquely determined output in its codomain.

The gradient vector points in the direction where the function increases most rapidly. In machine learning, the negative of the gradient is used to move in the direction of steepest descent to find a minimum.

The partial derivative measures the rate of change of the function as only one variable changes, while all other variables are held constant.

The Chain Rule is used to differentiate functions that are nested within each other, such as . This rule is essential for the backpropagation algorithm used to train neural networks.

Joint probability is the probability that multiple events happen at the same time. It is denoted as or .

The gradient points in the direction of the steepest increase of the loss function. Optimization algorithms like Gradient Descent use the negative of the gradient to iteratively adjust the model's parameters (weights and biases) in the direction that decreases the loss the most.

By definition, an eigenvector of a matrix is a non-zero vector that, when the matrix is applied to it, does not change direction but is only scaled by a scalar factor. This scalar factor is the eigenvalue . Therefore, .

First, find the partial derivatives: and . The gradient is . Evaluating at the point : .

Let S be the event 'email is spam' and F be 'email is flagged as spam'. We want . We have , . The False Positive Rate is . By Bayes' Theorem, . First find . Then, . The closest answer is ~35.7%, which may arise from slight variations in rounding or problem interpretation. Re-calculating: . Let's recheck the options. Maybe there's a misunderstanding. Wait, . Then . The options seem slightly off. Let's create a better question or options. Let's adjust the rates. Let's use a standard setup. Let's use the provided explanation logic for ~35.7%: If , then . Where does come from? It does not seem derivable from the question. Let's fix the question/options. Okay, let's reframe. Question: ... , , . Find . . . Let's try to make a problem that resolves to a clean answer. Re-drafting the question from scratch. A disease has a prevalence of 1/1000. A test has a 99% true positive rate and a 2% false positive rate (). What is ? . . , or 4.7%. This is a good medium-level question. Finalizing this one.

Let D be the event of having the disease, and T be the event of a positive test. We want to find . We are given: , so . The true positive rate is . The true negative rate is , so the false positive rate is . Using Bayes' Theorem: . First, we find the total probability of testing positive, . Then, , or about 4.7%.

Covariance measures the strength and direction of a linear relationship between two variables. A covariance of zero means that there is no linear association. However, it does not imply statistical independence, as there could still be a non-linear relationship between the variables (e.g., ).

To find the partial derivative with respect to , we treat as a constant and apply the chain rule. The derivative of is . Here, . The derivative of with respect to is . Therefore, .

The formula for conditional probability is . From the table, . To find the marginal probability , we sum the probabilities in the 'Y=1' column: . Therefore, .

The product of two matrices and is defined if and only if the number of columns in the first matrix () equals the number of rows in the second matrix (). Here, and , so the product is defined. The resulting matrix has dimensions equal to the number of rows of the first matrix and the number of columns of the second matrix, which is , or in this case.

We use the chain rule: . First, . Second, . Substituting back into the first part, we get . Multiplying the two parts gives .

The function describes how plausible different values of the parameter are, given the fixed observation of 7 heads in 10 tosses. This is the definition of a likelihood function . Probability, in contrast, would describe the chance of observing the data given a fixed, known value of (e.g., if we assume , what is the probability of 7 heads?).

To find the eigenvalues, we solve the characteristic equation . For this matrix, we have . This gives , which simplifies to , or . Factoring this quadratic equation gives . Thus, the eigenvalues are and .

For a continuous random variable, there are infinitely many possible values. The probability of the variable taking on any single, specific value is infinitesimally small, and is defined as zero. Probability is only defined over an interval, calculated as the integral (area under the curve) of the PDF over that interval, i.e., .

The data has three dimensions: the batch size, the image height, and the image width. The resulting tensor would have a shape of (32, 64, 64). Since there are three dimensions (axes), the tensor is of rank 3. If the images were color (e.g., RGB), there would be a fourth dimension for the color channels, resulting in a rank-4 tensor.

Using the properties of variance: . Adding a constant does not change the variance (spread) of the data, so the '' term has no effect. The constant multiplier gets squared. Therefore, .

The gradient vector, , points in the direction of the greatest rate of increase (steepest ascent) of a function. The goal of gradient descent is to minimize a cost function. Therefore, by taking steps in the opposite direction of the gradient, i.e., , we are moving in the direction of the steepest decrease (descent), effectively moving towards a local minimum.

The dot product of two vectors is calculated by multiplying their corresponding components and summing the results. So, .

In Bayes' Theorem: is the Posterior, is the Prior, is the Evidence, and is the Likelihood. It represents the probability of observing the evidence B given that the hypothesis A is true.

The eigenvalues corresponding to the eigenvectors represent the amount of variance in the data along that eigenvector's direction. To capture the most significant patterns in the data first, the principal components (eigenvectors) are ordered from highest variance to lowest. Therefore, they are sorted by their corresponding eigenvalues in descending order.

The time can take any non-negative real value within a range (e.g., 3.14 seconds, 3.141 seconds, etc.). Since the variable can take on an uncountably infinite number of values within an interval, it is a continuous random variable. A discrete variable would have countable values (e.g., number of clicks).

The MSE cost function for linear regression is a quadratic function of the parameters , which results in a bowl-shaped, or convex, surface. A key property of convex functions is that they have a single global minimum and no local minima. This guarantees that an optimization algorithm like gradient descent will eventually converge to the optimal solution.

The definition of statistical independence for two events A and B is that the occurrence of one does not affect the probability of the other. Mathematically, this is expressed as their joint probability being equal to the product of their individual (marginal) probabilities: .

The Rayleigh-Ritz theorem states that the maximum value of the Rayleigh quotient for a symmetric matrix is its largest eigenvalue, . This maximum is achieved when the vector is the eigenvector corresponding to this largest eigenvalue. Similarly, the minimum value is the smallest eigenvalue, .

The gradient is found by differentiating with respect to the vector . The gradient of the first term, , is . The gradient of the second term, , is . By linearity of differentiation, the total gradient is the sum of these two parts: .

This setup describes Linear Discriminant Analysis (LDA). The decision boundary occurs where . Taking the log of the posteriors and simplifying, the quadratic terms in (from the part of the Gaussian PDF) are identical for both classes because the covariance matrix is shared. Therefore, these terms cancel out, leaving a decision boundary that is a linear function of (a hyperplane).

A diagonal covariance matrix means that all off-diagonal elements are zero. The off-diagonal element is the covariance . Therefore, , which is the definition of and being uncorrelated. Independence is a stronger condition; uncorrelated variables are only guaranteed to be independent if they are jointly Gaussian. The variances (diagonal elements) do not have to be equal.

This is a key step in backpropagation. The gradient must have the same dimensions as (). Let , where is an vector and is the element-wise product. By the chain rule, . This is an outer product between the vector and the vector , resulting in the correct dimensions for the gradient.

The likelihood function is numerically equal to the probability of the observed data given the parameter, . However, the crucial distinction is that likelihood is a function of the parameter , with the data held fixed. It measures how plausible a parameter value is. It is not a probability distribution over , and its integral with respect to (from 0 to 1) does not sum to 1. The integral is .

While a PSD matrix must have a non-negative determinant (since the determinant is the product of eigenvalues, which are all non-negative), the converse is not true. A non-negative determinant is a necessary but not sufficient condition. For example, the matrix has a determinant of 1, but its eigenvalues are -1, so it is not PSD. The other three options are all necessary and sufficient conditions for a symmetric matrix to be positive semi-definite.

The original tensors have ranks 3 (indices i, j, k) and 2 (indices j, k). The summation is performed over the repeated index . This operation 'contracts' or removes one index from each tensor. So, the rank of the resulting tensor is the sum of the initial ranks minus twice the number of contracted indices. Here, rank(S) = rank(T) + rank(M) - 2*1 = 3 + 2 - 2 = 3. Wait, let me recheck. The summation is . This is incorrect. A common contraction would be . Let's rephrase the question to a standard operation. Let's use . This is also not a standard single contraction. Let's use a clearer example: . This is matrix multiplication. Let's try . No. Let's stick to the question but fix the explanation. The operation is . The indices being summed over are from and from . The resulting tensor has free indices and . Since there are two free indices, the resulting tensor is of rank 2 (i.e., a matrix). This type of operation is fundamental in tensor networks and computations like attention mechanisms.

First, we must find the normalization constant . . To be a valid PDF, this integral must be 1, so . The joint PDF is . Next, we find the marginal density . Finally, the conditional density is for .

The second partial derivative test for multivariable functions uses the eigenvalues of the Hessian matrix to classify critical points. A critical point is a local minimum if all eigenvalues are positive (positive definite), a local maximum if all are negative (negative definite), and a saddle point if there is at least one positive eigenvalue and at least one negative eigenvalue (indefinite). If the Hessian has zero eigenvalues along with positive or negative ones (semi-definite), the test is inconclusive, though such points are often also saddle points or part of a valley/ridge of minima/maxima.

Let be an eigenvalue of with corresponding eigenvector . Then . Applying again gives . Since , we have , which implies . Since is non-zero, we must have , which means . So, the only possible eigenvalues are 0 and 1. If all eigenvalues were 1, the matrix would be the identity matrix (if diagonalizable). If all were 0, it would be the zero matrix (if diagonalizable). Since it's stated to be neither, it must have a mix of 0 and 1 eigenvalues.

This MGF has the form of a Normal distribution, . By comparing the given MGF with this standard form, we can identify the parameters. The coefficient of is the mean , so . The coefficient of is , so , which implies the variance . Alternatively, one could compute the derivatives of the MGF: , and . Then .

This is a standard result from matrix calculus. The derivation uses Jacobi's formula: . The differential of is . The gradient is the matrix that satisfies . By comparing with , we can identify , which means . For a symmetric A, is also symmetric, so , but the general result involves the transpose.

We use the quotient rule. Let and . Then . Here, and . So, .

The multivariate chain rule states that the Jacobian of a composite function is the product of the Jacobians of the individual functions, evaluated at the appropriate points. The order of multiplication is crucial and follows the order of function composition from outer to inner. The derivative of the outer function with respect to its input is , and the derivative of the inner function with respect to its input is . The resulting Jacobian is . The matrix dimensions also align correctly: gives a matrix, which is the correct shape for the mapping from to .

The distribution of the sum of two independent random variables is given by the convolution of their individual densities. For , . Since the variables are non-negative, the integral becomes . This is the PDF of a Gamma distribution with shape parameter and scale parameter , often written as .

This is a consequence of the 'curse of dimensionality'. In high dimensions, the dot product tends to be close to zero relative to the magnitudes of the vectors, and . The cosine of the angle is . As the dimension increases, the dot product (a sum of i.i.d. random variables with mean 0) grows slower than the vector magnitudes. Consequently, concentrates sharply around 0, meaning the angle concentrates around . Thus, two random vectors in high-dimensional space are almost always nearly orthogonal.

The general formula for the variance of a linear combination of two random variables is . In this case, and . Plugging these values into the formula gives: .

The change of variable formula relates the PDFs of the original and transformed variables. It ensures that the total probability is conserved, i.e., . Rearranging gives . Since , we substitute and the derivative term to get the formula in terms of : . The absolute value of the Jacobian of the inverse transformation is crucial to ensure the PDF is non-negative.

A prior is conjugate to a likelihood if the resulting posterior distribution is in the same probability distribution family as the prior. For example, the Beta distribution is the conjugate prior for the Bernoulli likelihood. This means if our prior is a Beta distribution, observing Bernoulli-distributed data will result in a posterior that is also a Beta distribution. This provides a closed-form, analytical solution for the posterior, avoiding the need for complex numerical integration (like MCMC) to characterize it. While we still need to get the exact posterior (not just its proportionality), the key advantage is the posterior's simple form.