Unit 1 - Practice Quiz

Conventionally, in a data matrix, each row corresponds to a single observation or data point (e.g., a user, a house, an image), while the columns represent the features of that data point.

A scalar (like age) is a 0th-order tensor, a vector (like prices) is a 1st-order tensor, and a matrix (like a grayscale image) is a 2nd-order tensor. A color image has three dimensions (height, width, color channels), making it a 3rd-order tensor.

A scalar is a single numerical value (e.g., temperature), representing only magnitude. A vector is an array of numbers that represents both magnitude and direction in a space.

In the standard notation for matrix dimensions, , always represents the number of rows and represents the number of columns.

A feature vector is a vector whose elements represent the specific features of a single instance or data point. For example, for a house, the feature vector might be [size, number_of_bedrooms, age].

A fundamental property of a vector space is closure. This means that if you add any two vectors from the space, the result is still in the space, and if you multiply any vector by a scalar, the result is also still in the space.

The zero vector is the additive identity in a vector space. In , which represents 3-dimensional space, the zero vector is a vector with three components, all of which are zero.

The dimension of a vector space is defined as the number of linearly independent vectors required to span the entire space. This set of vectors is known as a basis.

A subspace must contain the zero vector and be closed under addition and scalar multiplication. A line through the origin satisfies these conditions, whereas a line not through the origin does not contain the zero vector.

The span of a set of vectors is the set of all possible linear combinations of those vectors. If this span is equal to the entire vector space, the set is said to span the space.

The L2 norm is calculated as the square root of the sum of the squared components: .

The L1 norm calculates distance by summing the absolute differences of the components, which is analogous to moving along a rectangular grid, like the streets of Manhattan.

The L1 norm is the sum of the absolute values of the components: .

L1 regularization adds a penalty equal to the L1 norm of the weights. This has the effect of encouraging some model weights to become exactly zero, leading to a 'sparse' model that performs feature selection.

The L2 norm corresponds to our intuitive understanding of length in a straight line from the origin to the point defined by the vector's coordinates.

Matrix multiplication is the standard way to represent a linear transformation. It maps a vector from one vector space to another while preserving the properties of linearity (additivity and homogeneity).

A scaling matrix has the scaling factors on its main diagonal. To double the size along both axes, the scaling factor for both x (top-left) and y (bottom-right) should be 2.

The identity matrix is the matrix equivalent of the number 1. Multiplying any vector by the identity matrix () results in the original vector unchanged.

This defining equation of linearity combines two properties: additivity () and homogeneity (). Together, this is known as the principle of superposition.

The weights connecting one layer of neurons to the next can be organized into a matrix. Applying this layer to an input vector (of activations from the previous layer) is a linear transformation performed by multiplying the input vector by the weight matrix.

The input matrix has dimensions (batch size input features), which is . The output matrix has dimensions (batch size output features), which is . For the matrix multiplication to be valid, the dimensions must align: . This requires and . Thus, the dimensions of are .

A tensor is a multi-dimensional array. For a batch of images, the standard convention is a 4D tensor. Following the 'channels-last' format, the dimensions are ordered as (batch size, image height, image width, number of color channels). Here, the batch size is 1000, height is 64, width is 64, and there are 3 channels (R, G, B). Therefore, the shape is .

The outer product results in an matrix where every column is a scalar multiple of the vector . Since all columns are linearly dependent (they are all multiples of a single vector), the dimension of the column space is 1. Therefore, the rank of the matrix is 1.

The Hadamard product is calculated by multiplying the corresponding elements of the two matrices. So, . The resulting matrix is .

The data matrix has dimensions , which is . Its transpose, , has dimensions , which is . The matrix multiplication therefore has dimensions , which results in a matrix.

A set is a subspace if it contains the zero vector, is closed under vector addition, and is closed under scalar multiplication. The set represents a plane passing through the origin, which satisfies all three conditions for being a subspace. The other options fail: A) does not contain the zero vector; C) is not closed under multiplication by negative scalars; D) is a single point that is not the origin.

The vector is a scalar multiple of since . This means the two vectors are linearly dependent and point along the same line. The span of a set of linearly dependent vectors is determined by its largest linearly independent subset. In this case, , which is the line passing through the origin in the direction of .

A basis for a vector space must satisfy two conditions: the vectors must be linearly independent, and they must span the entire space. For , any basis must contain exactly 3 vectors. Option C contains only two vectors, so it cannot span the entire 3-dimensional space. Therefore, it cannot be a basis for .

The predicted value vector is computed by multiplying the matrix by the vector . This operation is equivalent to taking a linear combination of the columns of , with the coefficients given by the elements of . By definition, the set of all possible linear combinations of the columns of is the column space of .

The Rank-Nullity Theorem states that for a matrix with columns, the rank of plus the dimension of the null space of (nullity) equals . Here, the transformation is from , so the matrix has 10 columns (). Given that the rank is 3, we have: . Solving for the nullity gives a dimension of 7 for the null space.

L1 regularization corresponds to a constraint region shaped like a diamond (in 2D) or cross-polytope (in higher dimensions). This shape has sharp corners that lie on the axes. The optimal solution is found where the elliptical level curves of the error function first touch this constraint region. It is geometrically probable that this first contact happens at a corner, where one or more coefficients are zero.

The L1 norm (Manhattan distance) is the sum of the absolute values of the components: . The L2 norm (Euclidean distance) is the square root of the sum of the squares of the components: .

The formula for the projection of vector onto vector is . First, calculate the dot product: . Next, calculate the squared L2 norm of : . Finally, compute the projection: .

The projection of onto the column space of is the vector in that space closest to . The vector connecting to its closest point in the subspace, which is the residual , must be orthogonal to the subspace itself. Therefore, the residual vector is orthogonal to every vector in the column space of .

The Frobenius norm of a matrix is the square root of the sum of the squares of all its elements, analogous to the L2 norm for vectors. .

We can find the matrix by tracking the transformation of the basis vectors and . First, reflected across the y-axis becomes . Then, rotating 90 degrees clockwise maps to , resulting in . Second, is unchanged by reflection across the y-axis. Then, rotating 90 degrees clockwise gives . The resulting columns are and , forming the matrix .

In PCA, the eigenvectors of the covariance matrix represent the principal components (the new orthogonal axes of the data). The corresponding eigenvalue for each eigenvector indicates the amount of variance in the original data that is captured when projected onto that eigenvector. Larger eigenvalues correspond to principal components that explain more variance.

The defining equation for an eigenvector, , states that when the matrix transforms the vector , the resulting vector is simply a scalar multiple () of the original vector . This means the transformation only stretches or shrinks the vector along its original direction (and may flip it if ), but it does not change the direction itself.

The Eckart-Young-Mirsky theorem states that the best rank- approximation of a matrix is obtained from its SVD. We form a new diagonal matrix by keeping the largest singular values from and setting all others to zero. The approximation is then calculated as . This effectively captures the most significant components of the original matrix.

The absolute value of the determinant of a transformation matrix gives the scaling factor for areas (in 2D). The determinant of matrix is calculated as: . This means the transformation scales the area of any shape by a factor of 3. Therefore, a unit square with an area of 1 will be transformed into a parallelogram with an area of .

The tensor is indexed as (batch, height, width, channel). Fixing the batch index to i selects the i-th image. Fixing the channel index to j selects the j-th channel (e.g., Red, Green, or Blue). The colons : act as wildcards, selecting all elements along the height and width dimensions. This results in a 2D matrix representing a single color plane of a specific image.

Every column of the matrix is a scalar multiple of the vector . Specifically, the j-th column of is . Since all columns are multiples of a single non-zero vector , the column space is one-dimensional and is spanned by . Therefore, the rank of is 1.

In , a 2D subspace is a plane passing through the origin. Since and are distinct planes through the origin, their intersection must be a line passing through the origin, which is a 1-dimensional subspace. Using the dimension theorem, . Since is a subspace of , . So, , which implies . Since , cannot be 2. Therefore, the dimension must be exactly 1.

The SVD decomposes the transformation into three steps: 1) A rotation/reflection in the domain space (), which maps the unit sphere to itself. 2) A scaling along the new coordinate axes by the singular values (), which transforms the sphere into a hyperellipse. 3) A rotation/reflection in the codomain space (), which orients the hyperellipse in . The final shape is a hyperellipse in .

The residual vector is orthogonal to the projection . By the Pythagorean theorem for vectors, . Therefore, . Let's compute : . Since the columns of are orthonormal, (the identity matrix). So, . Substituting back, we get .

This is a key part of the Fundamental Theorem of Linear Algebra. The null space of , also called the left null space of , contains all vectors such that . This condition means is orthogonal to every row of , which are the columns of . Therefore, any vector in is orthogonal to every vector in the column space . Together, they span the entire space , making them orthogonal complements.

The dual formulation of the SVM optimization problem involves minimizing a quadratic form , where the matrix is constructed from the Gram matrix . For this quadratic program to be convex, the matrix must be positive semi-definite. The structure of is such that it is positive semi-definite if and only if the kernel matrix is. Convexity is essential because it ensures that any local minimum found by an optimization algorithm is also the global minimum.

When has orthonormal columns, . The quadratic term simplifies: . The objective becomes minimizing . This decouples for each component . The minimum for each is found by the soft-thresholding operator, which is given by the formula , applied to each component of .

The Sherman-Morrison formula provides an expression for the inverse of a rank-one update of a matrix: . In our case, and the update is . The formula becomes . The inverse exists if and only if the denominator is non-zero, which is the condition .

The eigenvalues are the roots of the characteristic equation . This gives , which simplifies to , or . Using the quadratic formula, . By Euler's formula, these are and .

To be a subspace, must satisfy three axioms: contain the zero vector, be closed under addition, and be closed under scalar multiplication. contains since . It is also closed under scalar multiplication. However, it is not closed under addition. For example, let and . Both are in since and . Their sum is . For this to be in , we need , but . Thus, is not closed under addition and is not a subspace.

A projection matrix maps a vector space onto a proper subspace (a subspace of lower dimension), unless it is the identity matrix . This means the transformation collapses some dimensions, mapping non-zero vectors (from the orthogonal complement of the subspace) to the zero vector. Therefore, the null space of is non-trivial, which means its determinant is zero and it is not invertible. The only exception is when the subspace is the entire space, in which case , which is invertible. All other properties are true for projection matrices.

The columns of the orthogonal matrix are the eigenvectors of the covariance matrix . These eigenvectors are the principal components, which form a new orthonormal basis for the data. The diagonalization (or equivalently ) shows that in this new basis, the covariance matrix is diagonal. A diagonal covariance matrix means that the new features (the projections of the data onto the principal components) are mutually uncorrelated.

For any vector , we have the inequality . For a unit L2 vector (), this becomes . The lower bound is achieved when all the vector's mass is concentrated in a single entry, which is a standard basis vector (e.g., for , and ). The upper bound is achieved when the mass is spread out as evenly as possible, which occurs when all entries have equal magnitude (e.g., for , and ).

By definition, the predicted vector is a linear combination of the columns of , with the weights as the coefficients. The set of all possible linear combinations of the columns of is its column space, . The goal of ordinary least squares is to minimize the squared L2 distance between the true vector and the prediction , i.e., . The vector in that is closest to is the orthogonal projection of onto .

The regularization problem can be viewed as finding the point where the level sets (contours) of the loss function first touch the constraint region defined by the norm ball (). The L1 norm ball () has sharp corners/vertices at the axes. The elliptical contours of the sum-of-squares error are likely to make first contact with this shape at one of its corners, where one or more weight coordinates are zero. In contrast, the L2 norm ball () is a smooth sphere, and the first point of contact will typically not be on an axis, resulting in small but non-zero weights.

The rank of the matrix is the dimension of its column space (or range). The column space is the set of all possible output vectors. Therefore, the entire input space is mapped into this -dimensional subspace of . Since is not full rank, either or (or both). If , the null space is non-trivial, meaning a subspace of input vectors is mapped to zero. If , the range does not span the entire codomain. In all cases, a dimensionality reduction occurs.

The trace operator has the cyclic property: . This means you can cyclically permute the matrices inside the trace. Following this rule, is equal to and . However, is not generally equal to , which is a non-cyclic permutation. For example, let A, B, C be 1x1 matrices (scalars) a=1, b=2, c=3. Tr(abc)=6 but Tr(bac)=6. Let's try non-square matrices producing a square result. Let , , . Then , which is not square, so trace is not defined. The matrices must result in a square matrix. Let . is valid. requires . For to be valid, and . This is too restrictive. The cyclic property for is the key. . It does not generally equal .

The eigenvalues of the covariance matrix represent the amount of variance in the data along the direction of the corresponding eigenvectors (the principal components). By selecting the eigenvectors associated with the largest eigenvalues, we are choosing the directions of maximum variance. The subspace spanned by these eigenvectors is the lower-dimensional hyperplane that best approximates the data, in the sense that projecting the data onto it preserves the most variance.

Any linear transformation can be represented by a matrix multiplication. A convolution is a linear transformation. When the input image is flattened (vectorized) into a single column vector of size , the 2D convolution operation can be represented as a multiplication with a large transformation matrix of size . Due to the local and repeated nature of the convolution kernel, this matrix has a special structure: it is very sparse (mostly zeros) and exhibits a block Toeplitz structure, where diagonals are constant.