Unit 4 - Practice Quiz

A neural network is composed of interconnected units called neurons or nodes, which process and transmit information, similar to biological neurons in the brain.

A single-layer Perceptron can only learn to separate data that is linearly separable, meaning it can be divided by a single straight line or hyperplane.

MLP stands for Multi-Layer Perceptron, which is a type of feedforward artificial neural network with one or more hidden layers between the input and output layers.

CNNs use special layers called convolutional layers that are highly effective at detecting patterns, features, and spatial hierarchies within images.

RNNs have internal memory (loops) that allow them to process sequences of data, making them ideal for tasks where the order of information is important, like language or stock prices.

The self-attention mechanism is the core component of the Transformer, allowing the model to weigh the importance of different words in the input sequence when processing a specific word.

NLP is a field of AI focused on the interaction between computers and humans using natural language. Its primary goal is to make computers capable of processing and analyzing large amounts of language data.

Syntactic Analysis, or parsing, is the process of analyzing a string of symbols (a sentence) according to the rules of a formal grammar to understand its structure.

Tokenization is a fundamental first step in many NLP tasks. It splits a piece of text into smaller pieces, called tokens, which can be words, characters, or subwords.

Word embeddings capture the semantic meaning and relationships between words by mapping them to vectors of real numbers in a multi-dimensional space.

The attention mechanism allows the model to dynamically weigh the importance of different parts of the input when making a prediction or generating output, improving its performance on long sequences.

BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) are powerful, pre-trained large language models that form the basis for many modern NLP applications.

A chatbot is a software application designed to conduct a conversation with a human user in natural language through text or speech, for purposes like customer service or information retrieval.

Sentiment analysis is the process of computationally identifying and categorizing opinions expressed in a piece of text to determine the writer's attitude towards a particular topic.

Without a non-linear activation function, an MLP, no matter how many layers it has, would behave just like a single-layer perceptron because a series of linear transformations is equivalent to a single linear transformation.

Pooling layers are used to progressively reduce the spatial size of the representation, which helps to decrease the amount of parameters and computation in the network, and also helps to control overfitting.

Machine Translation is a classic NLP use case that involves using software to translate text or speech from a source language to a target language.

Pre-training involves training the model on a vast corpus of text data to learn general language patterns. This pre-trained model can then be adapted (fine-tuned) for more specific downstream tasks with much less data.

Text Summarization is the process of condensing a source text into a shorter version, providing a quick overview of the main points of the original document.

In deep networks or RNNs, gradients can become extremely small as they are propagated backward through time, making it hard for the network to learn long-range dependencies. Architectures like LSTMs and GRUs were developed to mitigate this issue.

A single-layer perceptron can only learn linearly separable patterns. The XOR function's data points (0,0)->0, (0,1)->1, (1,0)->1, (1,1)->0 cannot be separated by a single straight line in a 2D plane. This limitation was crucial in demonstrating the need for multi-layer networks.

A composition of linear functions is itself a linear function. Without non-linear activation functions, the entire MLP, regardless of its depth, can be mathematically simplified to a single linear layer. The non-linearities are essential for learning complex, non-linear patterns.

The Vanishing Gradient Problem occurs when gradients become progressively smaller as they are backpropagated to earlier layers. Activation functions like sigmoid and tanh have derivatives that are less than 1, and multiplying these small numbers repeatedly causes the gradient to vanish. The Rectified Linear Unit (ReLU) has a derivative of 1 for positive inputs, which helps prevent the gradient from shrinking.

The formula for calculating the output size is , where W is input size, K is kernel size, P is padding, and S is stride. For this case: . Since we take the floor of the division, it becomes . So the output is 30x30.

Max-pooling takes the maximum value from a patch of the feature map. This means that even if a feature shifts slightly in position within the patch, the output of the max-pooling layer can remain the same. This makes the network more robust to small translations of features in the input image.

Standard RNNs suffer from the vanishing gradient problem, making it difficult for them to learn dependencies between elements that are far apart in a sequence. LSTMs introduce a 'cell state' and 'gates' (input, forget, output) that regulate what information is stored, forgotten, or outputted, allowing gradients to flow over longer durations and enabling the network to capture long-range dependencies.

In a many-to-one architecture, the RNN processes the entire sequence of inputs (many), and the hidden state at the final time step is expected to have encoded a meaningful summary of the whole sequence. This final hidden state vector is then used as the input to a feed-forward layer (like a softmax classifier) to produce a single output (one), such as a class label.

RNNs must process sequences token by token, as the hidden state of time step 't' depends on the hidden state of 't-1'. In contrast, self-attention calculates a score between every pair of tokens in the sequence simultaneously. This lack of sequential dependency allows for massive parallelization on modern hardware like GPUs, making it much faster to train on large sequences.

The self-attention mechanism treats the input as a set of vectors, meaning it has no inherent sense of order. The sentences "The cat chased the dog" and "The dog chased the cat" would look identical to the attention mechanism without positional information. Positional Encodings are vectors added to the input embeddings to provide the model with a signal about the position of each token in the sequence.

Word-based tokenization fails when it encounters a word not in its vocabulary, mapping it to an 'UNK' token and losing its meaning. Subword tokenization, like BPE, can represent any word by breaking it down into smaller, learned pieces. For example, 'tokenization' might become 'token' and '##ization', allowing the model to reason about novel words and handle OOV issues gracefully.

This phenomenon, known as semantic arithmetic, is a key property of well-trained word embeddings. It shows that the spatial arrangement of vectors is not random but encodes complex semantic and syntactic relationships. The vector difference between 'Paris' and 'France' captures the concept of 'is the capital of', which can then be applied to 'Italy' to find its capital.

Training word embeddings from scratch requires a very large dataset to learn meaningful representations. By using pre-trained embeddings, a model can start with a high-quality understanding of word meanings and relationships. This 'transfer learning' is highly effective for improving performance and convergence speed, especially when the target task has limited training data.

The attention mechanism computes a set of weights for each output step, indicating how much 'attention' to pay to each input word. A high weight means the model has determined that the context and meaning of that specific input word are most important for generating the current word in the translated sequence.

BERT's architecture (based on the Transformer encoder) allows it to see the entire input sentence at once, creating deep bidirectional representations perfect for tasks like sentiment analysis or question answering. GPT's architecture (based on the Transformer decoder) is designed to predict the next word given the previous words, making it a natural fit for generative tasks like writing essays or code.

Unlike traditional language models that predict the next word (left-to-right), MLM allows the model to learn context from both directions. By masking about 15% of the tokens and training the model to predict them, BERT learns rich representations of words based on their full surrounding context.

Lexical, morphological, and syntactic analysis deal with words, their forms, and sentence structure, respectively. Semantic Analysis is the phase concerned with understanding the meaning of the text. Identifying entities like 'parties' and their 'obligations' requires interpreting the meaning and relationships within the text, which is the core of semantic analysis.

The Natural Language Understanding (NLU) component extracts intent and entities. The Dialogue Manager takes this structured information, tracks what has happened so far in the conversation (state tracking), and decides what to do next. This could be asking a clarifying question, querying a database via an API, or providing a final answer.

This task is a direct application of text summarization. A major challenge, especially with abstractive summarization (where the model generates new sentences), is avoiding 'hallucinations'—generating plausible but factually incorrect information that was not in the original article. Ensuring factual consistency is a key area of research.

Sentiment analysis (or opinion mining) is the NLP task focused on identifying and extracting subjective information from source materials. Classifying a text as positive ('thumbs up') or negative ('thumbs down') is a classic binary sentiment classification problem.

First, calculate the pre-activation value, . This is . The ReLU activation function is defined as . Since z is -2.5, the output is .

A composition of linear functions is itself a linear function. No matter how many layers a neural network has, if it only uses linear activation functions, the entire network collapses into a single linear transformation (equivalent to a single layer). Therefore, its representational power is limited to that of a single-layer perceptron, which can only solve linearly separable problems.

A 1x1 convolution operates across all channels at a single pixel location. It's essentially a fully connected layer applied at every spatial position. Its main use is to change the number of channels (depth) of the feature map. By using fewer 1x1 filters than input channels, it performs dimensionality reduction (a 'bottleneck' layer). By using more, it expands the dimensionality. This is done without affecting the spatial dimensions because the kernel size is 1x1 and stride is typically 1.

The core difference is that an LSTM maintains two vectors passed between timesteps: the hidden state () and the cell state (). The cell state acts as a memory conveyor. A GRU combines these into a single hidden state vector (). It uses an update gate to control how much past information to keep versus new information to add, and a reset gate to control how much of the past state to forget. This simplification (fewer gates, one state vector) makes GRUs computationally cheaper, but the dedicated cell state in LSTMs can sometimes offer more powerful modeling capabilities.

While multi-head attention is a core part of the Transformer, simply increasing the number of heads does not change the fundamental complexity with respect to sequence length. Each head still computes an attention matrix. The other options are all valid and popular research directions for creating more efficient Transformers: sliding window (Longformer), sparse patterns (Sparse Transformer), and kernel-based methods (Linear Transformers) all aim to break the quadratic bottleneck.

The key distinction lies in the attention mask. In BERT's self-attention layers, a token can attend to all other tokens in the sequence (both to its left and right), which is why it's called bidirectional. This allows it to build a deep understanding of the full context, perfect for Natural Language Understanding (NLU) tasks like classification or question answering. In GPT, the self-attention is masked so that a token can only attend to previous tokens (and itself). This autoregressive property is essential for generating text one token at a time, making it a natural fit for Natural Language Generation (NLG).

Static embeddings assign a single, fixed vector to each word in the vocabulary (e.g., the vector for 'bank' is the same in 'river bank' and 'investment bank'). Contextualized models are deep neural networks (LSTMs for ELMo, Transformers for BERT) that process the entire input sentence. The embedding for a word is derived from the internal state of the network at that word's position. Since the internal state is influenced by all other words in the sentence, the resulting embedding is context-dependent. Therefore, 'bank' will have different vectors in the two example sentences, resolving the polysemy issue.

The authors of 'Attention Is All You Need' observed that for large values of , the dot products grow large in magnitude. When these large values are fed into the softmax function, the gradients can become vanishingly small, making learning difficult. By scaling the dot products by , the variance of the inputs to the softmax is kept at 1, regardless of . This ensures that the softmax function operates in a region with healthier gradients, stabilizing the training process.

While both are subword tokenization algorithms, their merge criterion differs. BPE is simpler: it iteratively counts all adjacent symbol pairs and merges the most frequent one. WordPiece, used by BERT, is slightly more sophisticated. It builds a vocabulary and then picks the merge that increases the likelihood of the training corpus given a unigram language model. This likelihood-based approach often leads to subwords that align better with meaningful morphological units. For example, WordPiece is more likely to keep word stems intact and segment suffixes like '##ing' or '##ly'.

The number of parameters in a convolutional layer is independent of the input volume's spatial dimensions (). It depends on the filter size, the number of input channels, and the number of filters.

Number of weights = (filter_height filter_width num_input_channels) num_filters
Number of weights = () 32 = 400 32 = 12,800.

Each filter has a single bias term associated with it.
Number of biases = num_filters = 32.

Total learnable parameters = Number of weights + Number of biases = 12,800 + 32 = 12,832.

BLEU (Bilingual Evaluation Understudy) works by matching n-grams (contiguous sequences of n words) in the candidate translation against the n-grams in one or more reference translations. It measures precision—how many of the candidate's n-grams appear in the references. While it includes a penalty for translations that are too short, its core mechanism is lexical overlap. It cannot understand if synonyms are used, if the sentence structure is logical, or if the core meaning is preserved. A translation could have high n-gram overlap with a reference but completely miss the nuance or even state the opposite meaning.

The Perceptron Convergence Theorem guarantees that the learning algorithm will find a separating hyperplane in a finite number of steps if and only if the data is linearly separable. If the data is not linearly separable, there is no solution for the algorithm to converge to. It will continue to find misclassified points and update its weights in an attempt to correct for them, effectively cycling through different decision boundaries forever without ever satisfying the condition of zero misclassifications.

Teacher forcing is a training technique where the model's own output from the previous timestep is replaced with the ground-truth value from the training data as input for the current timestep. This provides a stable, correct signal at each step, preventing the model from propagating its own errors, which makes training much faster and more stable. However, at inference time, there is no ground truth to feed the model; it must use its own (potentially imperfect) predictions as input for the next step. This mismatch between training and inference (exposure bias) can cause the model to perform poorly, as it was never trained to recover from its own mistakes.

An RNN inherently processes a sequence token by token, so the order is naturally encoded in its recurrent state. The self-attention mechanism in a Transformer, however, calculates attention scores between all pairs of tokens in parallel. If you were to shuffle the input tokens, the resulting set of attention scores would be identical, just reordered. The model has no inherent sense of position. Positional encodings are fixed or learned vectors that are added to the token embeddings to give the model explicit information about the position of each token in the sequence, thus breaking the permutation invariance and allowing it to understand word order.

The original NSP task involved predicting if sentence B was the actual sentence following sentence A. Negative examples were created by pairing sentence A with a random sentence from a different document. Researchers found that this task was too simple. The model could often solve it just by checking if the two sentences shared the same topic (e.g., by looking for keyword overlap), without learning about the deeper logical and cohesive relationships between consecutive sentences. The RoBERTa model, for instance, removed the NSP task and showed improved performance on downstream tasks, suggesting NSP's limited utility.

Extractive summarization is akin to highlighting. It's a classification task where the model decides which sentences or phrases from the original document are important enough to be included in the summary. The summary contains only verbatim excerpts. Abstractive summarization is more human-like; it involves 'understanding' the source text and generating a new summary in its own words, potentially using words and phrases not present in the original. This is a sequence generation task, which necessitates an encoder-decoder architecture (like an RNN-based Seq2Seq model or a Transformer) to first encode the meaning of the source text and then decode it into a new sequence of words.

Intent Recognition is a classification task for a single user utterance (e.g., 'I want to fly to Boston' -> intent: book_flight, entity: destination=Boston). Dialogue State Tracking, however, must manage the state of the conversation across multiple turns. If the user then says 'I want to go tomorrow', the DST component must update the dialogue state with date=tomorrow while remembering the destination from the previous turn. It has to accumulate information, handle corrections ('Actually, I meant Boston'), and resolve ambiguities, making it a much more complex and stateful problem than single-turn intent recognition.

While the king-queen analogy is a powerful demonstration, research has shown that this capability is quite brittle. The neat geometric parallels are often an artifact of the statistical patterns of specific, frequent relationships (like country-capital) present in the training data (e.g., Wikipedia). The method often fails on more nuanced or less frequently stated relationships. The success of these analogies is not a universal property of the embedding space but rather a localized phenomenon, making it an unreliable tool for general-purpose analogical reasoning.

As the input passes through the encoder (downsampling path), the network gains semantic information (what is in the image) but loses spatial information (where it is). The decoder (upsampling path) tries to recover this spatial information to produce a high-resolution segmentation map. Skip connections feed feature maps from the encoder directly to corresponding layers in the decoder. This provides the decoder with the fine-grained spatial details from earlier layers that would otherwise be lost, allowing it to produce much more precise and accurate segmentation boundaries.

Instead of performing a single attention function, MHSA projects the queries, keys, and values into multiple, lower-dimensional subspaces and runs the attention mechanism in parallel in each subspace. The outputs are then concatenated and projected again. This allows each 'head' to specialize and learn different kinds of relationships between tokens. For example, one head might learn to track syntactic dependencies, while another might focus on which words refer to the same entity. A single large attention head would average all these different signals, potentially washing out a specific, useful relationship. MHSA allows the model to capture a richer set of features.

This is a classic example of a garden-path sentence. Correct syntactic parsing is crucial for correct semantic interpretation. A naive parser might identify 'The old man' as a noun phrase. However, the correct parse is that '[The] old' is a noun phrase (referring to old people) and 'man' is the main verb. The syntax determines the relationship between words: (Subject: The old) (Verb: man) (Object: the boats). If the parser fails to identify this structure, the semantic analysis stage will be unable to derive the correct meaning (Old people operate the boats) and will instead be left with a nonsensical fragment.