Unit 3: Deep Learning Sequence Models for NLP - Subjective Questions

Sequential text data consists of ordered tokens, such as words, subwords, or characters, in which the interpretation of each token can depend on preceding or succeeding tokens.

Importance of token order:

• Word order carries syntactic and semantic information.
• The same words arranged differently may express different meanings. For example, "The dog chased the cat" differs from "The cat chased the dog".
• Context can resolve ambiguity. The meaning of a word may depend on nearby words.
• Sequence length is variable, so models must process inputs of different sizes.
• Dependencies may be local, such as adjective-noun relationships, or long-range, such as agreement between a subject and a distant verb.

A sequence of tokens can be represented as , where the position is part of the information being modeled.

A recurrent neural network processes a sequence one element at a time while maintaining a hidden state that summarizes previous inputs.

For input at time step , the hidden state is computed as:

The output may be computed as:

where:

• is the current token representation.
• is the previous hidden state.
• , , and are learned weight matrices.
• and are bias vectors.
• The same parameters are reused at every time step.

This recurrence allows information from earlier tokens to influence later predictions. Depending on the task, an RNN may produce an output at every time step or use the final hidden state as a representation of the complete sequence.

Unrolling an RNN represents each time step as a separate computational node while sharing the same parameters across all nodes. For a sequence of length , the recurrence is:

If the total loss is the sum of time-step losses,

then a shared parameter such as receives gradient contributions from every time step:

For an earlier hidden state , gradients from later losses pass through repeated recurrent transitions:

Backpropagation through time (BPTT) therefore applies the chain rule backward across the unrolled network. It calculates gradients for all shared parameters and updates them using an optimizer. The repeated product of Jacobians is also the source of vanishing and exploding gradients.

During BPTT, gradients are repeatedly multiplied by recurrent Jacobian matrices. A simplified gradient contains a product such as:

• If the magnitudes of these factors are mostly less than , the gradient decreases exponentially. This is the vanishing-gradient problem. The model then struggles to learn long-range dependencies.
• If the magnitudes are mostly greater than , the gradient grows rapidly. This is the exploding-gradient problem, which may cause unstable updates or numerical overflow.

Remedies:

• Use LSTM or GRU architectures to improve long-term gradient flow.
• Apply gradient clipping, for example:

where is a threshold.

• Use appropriate initialization, normalization, and learning rates.
• Use truncated BPTT to limit the backward path.
• Employ residual connections or gated recurrent structures in deeper sequence models.

An LSTM maintains a hidden state and a cell state . Its gates regulate the storage, deletion, and exposure of information.

Gate functions:

• Forget gate : decides which old cell-state information to retain.
• Input gate : controls how much candidate information is written.
• Candidate state : contains proposed new information.
• Output gate : controls which cell-state information becomes the hidden output.

The additive cell-state update provides a more direct gradient path than a basic RNN, helping the LSTM learn long-term dependencies.

A GRU is a gated recurrent architecture that combines memory and hidden state into a single state . A common formulation is:

Gate functions:

• Update gate : balances retention of the previous state and adoption of the candidate state.
• Reset gate : controls how much past information is used to construct the candidate state.
• Candidate state : represents possible new content for the hidden state.

Because a GRU uses fewer gates and no separate cell state, it generally has fewer parameters than an LSTM and may train faster while still handling long-range dependencies effectively.

Basic RNN:

• Uses a single recurrent hidden-state update.
• Has the fewest parameters and lowest per-step computational cost.
• Is vulnerable to vanishing and exploding gradients.
• Is suitable for short sequences or problems dominated by local dependencies.

LSTM:

• Uses input, forget, and output gates plus a separate cell state.
• Usually has the largest parameter count among the three.
• Preserves long-term information through its gated additive memory update.
• Is useful when long and complex dependencies are important.

GRU:

• Uses update and reset gates without a separate cell state.
• Has fewer parameters than an LSTM but more than a basic RNN.
• Often trains faster than an LSTM and can achieve comparable performance.
• Is useful when computational efficiency and long-context modeling are both required.

No architecture is universally superior. Selection should be based on validation performance, sequence length, data volume, latency, memory constraints, and the importance of long-range dependencies.

A bidirectional RNN processes a sequence in both temporal directions:

• A forward RNN computes from to .
• A backward RNN computes from to .

The contextual representation at position is commonly formed as:

where denotes concatenation.

Advantages:

• Each output uses both left and right context.
• It is effective for named-entity recognition, part-of-speech tagging, sequence labeling, and offline text classification.
• Ambiguous tokens can be interpreted using words that appear later in the sentence.

Limitations:

• The complete input sequence must normally be available before processing is finished.
• It introduces additional parameters and computation.
• It cannot be directly used for strictly causal generation or real-time streaming tasks where future tokens are unavailable.

Many-to-one:

• Accepts a sequence and produces one output.
• Example: sentiment classification of an entire review.
• A final or pooled hidden representation is passed to a classifier.

One-to-many:

• Uses one initial input or context representation to generate a sequence.
• Example: generating a caption from an encoded image representation.
• Each generated token influences subsequent predictions.

Many-to-many with aligned lengths:

• Produces an output for each input position.
• Example: part-of-speech tagging or named-entity recognition.
• Input and output sequences generally have corresponding positions.

Many-to-many with different lengths:

• Maps one sequence to another sequence whose length may differ.
• Example: machine translation or text summarization.
• It is commonly implemented with an encoder-decoder architecture.

These architectures differ mainly in when outputs are produced and whether input-output positions are directly aligned.

An RNN-based sentiment classification pipeline contains the following stages:

• Preprocessing: tokenize the document and map each token to an integer identifier.
• Embedding: convert token into a dense vector using a learned or pretrained embedding matrix.
• Sequence encoding: pass through an RNN, LSTM, GRU, or bidirectional variant.
• Representation selection: use the final hidden state, pooled hidden states, or an attention-weighted combination as the document representation .
• Classification: for binary sentiment, compute

For multiple sentiment classes, use:

• Training: minimize binary or categorical cross-entropy using labeled reviews.
• Inference: select the class with the highest probability or apply a chosen probability threshold.

Padding masks should be used so that artificial padding tokens do not affect the sequence representation.

A recurrent encoder first transforms a document into a fixed-dimensional representation . This representation is then passed to an output layer.

Multiclass classification:

• Exactly one class is correct.
• The output layer uses softmax:

• Categorical cross-entropy is used:

• Prediction is usually .

Multilabel classification:

• Several labels may be correct simultaneously.
• Each output uses an independent sigmoid:

• Binary cross-entropy is summed or averaged over labels:

• A threshold is applied to each probability.

Thus, softmax models mutually exclusive classes, while independent sigmoids model non-exclusive labels.

An encoder-decoder model maps an input sequence to an output sequence.

Encoder:

• Reads source tokens .
• Produces hidden representations or a final context representation.
• A bidirectional recurrent encoder may be used when the full source text is available.

Decoder:

• Is initialized or conditioned by the encoder representation.
• Predicts output tokens autoregressively:

• Starts with a start-of-sequence token and stops after generating an end-of-sequence token.

For machine translation, the encoder represents the source-language sentence and the decoder generates the target-language sentence. For summarization, the encoder reads the document and the decoder produces a shorter sequence. An attention mechanism can allow the decoder to use different encoder states at each step instead of relying only on one fixed context vector.

Teacher forcing is a training strategy in which the decoder receives the correct previous target token rather than its own previous prediction. At step , training uses to predict .

The sequence loss is commonly:

Advantages:

• Provides a correct history at every training step.
• Speeds up convergence and stabilizes optimization.
• Prevents early incorrect predictions from corrupting all later training inputs.

Limitation: exposure bias

• During inference, the true previous token is unavailable.
• The model must condition on its own predictions.
• A prediction error may alter later inputs and cause errors to accumulate.

Teacher forcing is therefore efficient for maximum-likelihood training, but the difference between training and inference conditions must be considered. Scheduled sampling is one possible technique for gradually introducing model-generated inputs during training.

Teacher forcing:

• Feeds the correct previous target token to the decoder.
• Provides stable and efficient training.
• Creates a mismatch between training and inference.

Free-running decoding:

• Feeds the model's previous prediction back as the next input.
• Matches autoregressive inference conditions.
• Can make training unstable because early mistakes affect later steps, and discrete token selection complicates direct gradient flow.

Scheduled sampling:

• Probabilistically chooses between the correct previous token and the model's prediction.
• The probability of using model-generated tokens is typically increased during training.
• Attempts to reduce exposure bias while retaining some stability from teacher forcing.

A simple schedule may define teacher-forcing probability at epoch , such as:

Scheduled sampling can help in some settings, but it changes the training distribution and does not completely solve sequence-level error accumulation.

Truncated backpropagation through time limits gradient propagation to a fixed window of time steps instead of backpropagating through the complete sequence.

A typical procedure is:

• Process a segment of tokens.
• Compute the loss for that segment.
• Backpropagate only through its unrolled computation graph.
• Update or accumulate parameter gradients.
• Pass the final hidden state to the next segment but detach it from the earlier graph.

Advantages:

• Reduces memory usage from storing activations for very long sequences.
• Lowers the computational cost of each backward pass.
• Makes training on streams or long documents practical.
• Can reduce the severity of exploding gradients.

Trade-off:

Dependencies extending beyond steps receive no direct gradient signal through the truncation boundary. A small is efficient but may prevent learning long-range relationships, whereas a large improves temporal credit assignment at greater memory and computational cost.

Important training techniques include:

• Padding and masking: batch variable-length sequences while preventing padded positions from affecting loss or hidden-state aggregation.
• Length-based batching: group sequences of similar lengths to reduce wasted computation on padding.
• Gradient clipping: limit large gradient norms to control exploding gradients.
• Truncated BPTT: restrict the backward graph for long sequences.
• Dropout: regularize embeddings, outputs, or recurrent connections using suitable recurrent-dropout schemes.
• Learning-rate scheduling: reduce the learning rate when validation performance stops improving.
• Early stopping: stop training when validation loss no longer improves.
• Pretrained embeddings: provide useful lexical representations when labeled data is limited.
• Teacher forcing: stabilize decoder training for sequence generation.
• Checkpointing: preserve the model with the best validation performance.

These techniques address different concerns: computational efficiency, numerical stability, generalization, variable sequence lengths, and the gap between training and inference.

For a target sequence , an autoregressive model assigns probability:

Taking the negative logarithm converts the product into a sum:

The mean token-level cross-entropy is:

Perplexity is the exponential of this average loss:

A lower perplexity means the model assigns higher probability to the observed tokens and is less uncertain on average. For example, a perplexity of can be informally interpreted as uncertainty comparable to choosing among about equally likely alternatives at each step. Perplexity comparisons are valid only when tokenization, vocabulary, data, and evaluation procedures are consistent.

Using true positives , true negatives , false positives , and false negatives :

Interpretation:

• Accuracy measures the proportion of all correct predictions.
• Precision measures how often positive predictions are correct.
• Recall measures how many actual positive cases are detected.
• F1-score is the harmonic mean of precision and recall.

F1-score is more useful when classes are imbalanced or when both false positives and false negatives matter. For multiclass tasks, macro-F1 gives equal weight to every class, while micro-F1 aggregates decisions across classes and is influenced more by frequent classes.

In sequence labeling, the model predicts a label for each token, as in part-of-speech tagging or named-entity recognition.

Token accuracy:

Padding positions must be excluded using a mask. Token accuracy is simple, but it can be misleading when the non-entity label is very frequent.

Entity-level evaluation:

• Predicted and reference tag sequences are converted into entity spans.
• A predicted entity is usually correct only if its type and complete boundary match a reference entity.
• Precision, recall, and F1-score are then calculated from matched spans:

Entity-level F1 is stricter and more informative for named-entity recognition because partially correct spans are normally counted as errors.

BLEU:

• Commonly used for machine translation.
• Measures modified n-gram precision between generated text and one or more references.
• Includes a brevity penalty to discourage overly short outputs.
• Limitation: it may assign a low score to a valid paraphrase that uses different wording.

ROUGE:

• Commonly used for summarization.
• ROUGE-N measures n-gram overlap, while ROUGE-L uses the longest common subsequence.
• It is often recall-oriented and checks how much reference content appears in the output.
• Limitation: lexical overlap does not guarantee factual correctness, coherence, or readability.

Perplexity:

• Evaluates how much probability a model assigns to reference tokens.
• It is derived from average token-level negative log-likelihood.
• Lower perplexity indicates better predictive confidence on the evaluation data.
• Limitation: lower perplexity does not necessarily imply better task-level quality or more useful generated text.

For robust evaluation, automatic metrics should be selected according to the task and may be supplemented with human judgments of fluency, relevance, and factuality.