Unit 4: Sequence-to-Sequence Models and Attention Mechanisms - Subjective Questions

Encoder-decoder architecture is a neural framework used to transform an input sequence into an output sequence of possibly different length.

• The encoder reads the input sequence and converts it into hidden representations.
• A recurrent network, convolutional network, or Transformer can be used as the encoder.
• In a basic recurrent encoder, the hidden state is updated as:
  
• The final hidden state, often called the context vector , summarizes the input sequence.
• The decoder generates the output sequence one token at a time using the context vector and previously generated tokens.
• The probability of an output sequence is modeled as:
  

The main limitation of the basic architecture is that a fixed-length context vector may not preserve all information from a long input sequence. Attention mechanisms address this limitation by allowing the decoder to access all encoder hidden states.

A sequence-to-sequence model performs machine translation by mapping a source-language sentence to a target-language sentence.

• The encoder processes the source sentence, such as an English sentence, and produces hidden states.
• The decoder generates the translated sentence, such as a French sentence, token by token.
• A special <BOS> token indicates the beginning of decoding, while an <EOS> token indicates the end.
• During training, teacher forcing is commonly used. The correct previous target word is provided to the decoder when predicting the next word.
• The training objective is to minimize the negative log-likelihood:
  
• During inference, the actual target sentence is unavailable. The decoder uses its own previously predicted token.
• Greedy decoding selects the most probable token at every step, whereas beam search maintains several possible partial translations.

Attention is particularly important in translation because it enables the decoder to focus on the source words relevant to each target word.

Sequence-to-sequence models are used in abstractive summarization to generate a shorter text that expresses the important content of a document.

• The encoder reads the complete source document and creates contextual representations.
• The decoder generates a summary sequence conditioned on the document representations.
• Attention helps the decoder identify important sentences, phrases, or words while generating each summary token.
• Extractive summarization selects and combines existing sentences or phrases from the source document.
• Abstractive summarization generates new sentences and may use vocabulary or expressions that do not appear exactly in the source.
• Training generally uses pairs of documents and reference summaries.
• The model is optimized using token-level cross-entropy or negative log-likelihood.

A major challenge is that abstractive systems can produce fluent but factually incorrect statements. Therefore, evaluation should consider both linguistic quality and content preservation using metrics such as ROUGE, as well as factuality and human evaluation.

Attention mechanisms improve sequence-to-sequence models by allowing the decoder to dynamically access the encoder's hidden states.

In a classical encoder-decoder model, the entire input sequence is compressed into one fixed-length vector . This causes several problems:

• Information bottleneck: A single vector may not retain all details of a long sentence or document.
• Difficulty with long sequences: Earlier words may be forgotten as the encoder processes later words.
• Poor alignment: The model does not explicitly identify which input words correspond to each output word.
• Weak interpretability: It is difficult to determine which parts of the input influenced a prediction.
• Limited scalability: Performance often decreases as input length increases.

Attention computes a separate context vector for each decoder time step. The decoder assigns different importance weights to encoder states, allowing it to focus on relevant information. This improves translation quality, summarization performance, handling of long sequences, and interpretability.

Soft attention is a differentiable attention mechanism in which the decoder considers all encoder hidden states, assigning each one a continuous importance weight.

Let the encoder produce hidden states . At decoder step , an alignment score is computed between the decoder state and each encoder state:

The scores are normalized using softmax:

The weights satisfy:

The context vector is the weighted sum of the encoder states:

The context vector is then provided to the decoder to generate the next output token. Since all operations are differentiable, soft attention can be trained end-to-end using backpropagation.

An alignment mechanism calculates how relevant each encoder hidden state is to the current decoder state.

• The encoder hidden state represents the input token at position and its surrounding context.
• The decoder state represents the information needed to generate the target token at position .
• An alignment function produces a score:
  
• Higher scores indicate that encoder position is more relevant to the current decoding step.
• The scores are converted into attention weights using softmax:
  
• The weighted encoder representations form the context vector used by the decoder.

For example, while translating a target word corresponding to a particular source word, the alignment mechanism can assign a high weight to that source word. Alignment therefore improves learning of word correspondences, supports variable-length inputs, and provides a partial interpretation of the model's decisions.

Bahdanau attention, also called additive attention, was introduced for neural machine translation to remove the fixed-length context bottleneck.

At decoder step , the alignment score between the previous decoder state and encoder state is computed as:

where , , and are learnable parameters. The scores are normalized:

The context vector is:

The decoder uses along with the previous target token and decoder state to produce the next state and output distribution. Bahdanau attention is called additive because it combines transformed encoder and decoder states through addition before applying a nonlinear activation. It is effective when encoder and decoder state dimensions differ and provides flexible learned alignment.

Luong attention, or multiplicative attention, computes alignment scores using the current decoder state and encoder hidden states. It is generally simpler and computationally more efficient than additive attention.

Common Luong scoring functions include:

• Dot product:
  
• General form:
  
• Concat form:
  

The scores are converted into normalized attention weights:

The context vector is calculated as:

Luong attention commonly uses the current decoder state , whereas Bahdanau attention traditionally uses the previous decoder state . Luong attention is efficient, while Bahdanau attention can provide greater flexibility through its nonlinear additive scoring function.

Bahdanau and Luong attention both compute a weighted combination of encoder states, but they differ in how alignment scores are obtained.

Both methods normalize scores with softmax and compute a context vector. Bahdanau attention may model complex relationships more flexibly, while Luong attention generally has lower computational overhead and is easier to implement efficiently.

Attention is integrated into an encoder-decoder network through the following steps:

• The encoder processes the input sequence and produces hidden states .
• The decoder maintains a state while generating the output sequence.
• At each decoding step, an alignment score is computed between or and every encoder state .
• The scores are converted into attention weights using softmax.
• A dynamic context vector is formed:
  
• The context vector is combined with the decoder state to produce the output distribution:
  
• The selected or reference output token is fed into the next decoder step.

The attention computation is repeated for every output position. Thus, the model can focus on different input regions while generating different output tokens.

Let be the input sequence and be the target sequence. The encoder produces hidden states:

At decoder step , attention produces weights:

and context vector:

The decoder state is updated using the previous target token, previous state, and current context:

The probability of the complete target sequence is factorized as:

The model is trained by minimizing the negative log-likelihood:

Gradients flow through the decoder, attention weights, alignment scores, and encoder, allowing the entire system to learn jointly.

Soft attention assigns a continuous weight to every encoder state and computes a weighted average. Hard attention selects one or a small number of encoder states, usually through a discrete sampling operation.

• Soft attention uses:
  
• Hard attention may select an index and use:
  
• Soft attention is deterministic during a forward pass.
• Hard attention introduces discrete decisions and is therefore difficult to optimize directly with ordinary backpropagation.
• Hard attention may require reinforcement learning, sampling, or other gradient-estimation techniques.
• Soft attention is differentiable, stable, and easy to train end-to-end.
• Hard attention can be more computationally selective because it does not necessarily process every input position.

Soft attention is commonly preferred because it provides reliable gradients and usually achieves strong performance without requiring specialized optimization methods.

BLEU, or Bilingual Evaluation Understudy, measures the similarity between a machine-generated translation and one or more reference translations using modified -gram precision.

The BLEU score is commonly expressed as:

where:

• is the modified precision for -grams.
• is the weight assigned to each -gram order.
• is the brevity penalty.
• is often 4.

The brevity penalty penalizes translations that are much shorter than the reference:

where is candidate length and is reference length.

Limitations:

• It depends on surface-level -gram overlap.
• Valid synonyms and paraphrases may receive low scores.
• It may not reflect fluency, adequacy, or factual correctness well.
• Scores can vary with tokenization and preprocessing.
• A single reference may not capture all valid translations.

ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, measures the overlap between a generated summary and one or more reference summaries. It is widely used for summarization evaluation.

Important variants include:

• ROUGE-N: Measures -gram overlap. ROUGE-1 uses unigrams and ROUGE-2 uses bigrams.
• ROUGE-L: Uses the longest common subsequence between the candidate and reference, capturing sentence-level ordering.
• ROUGE-W: Gives higher credit to consecutive longest common subsequences.
• ROUGE-S: Measures skip-bigram overlap, allowing gaps between words.

ROUGE recall is calculated as:

Precision and F1 can also be reported:

ROUGE is useful because summaries should preserve important content from the reference. However, it may undervalue valid paraphrases, does not fully measure factuality or readability, and can reward lexical copying.

BLEU and ROUGE are automatic evaluation metrics based primarily on overlap between generated text and reference text.

BLEU is useful for comparing translation systems at the corpus level, while ROUGE is useful for checking content coverage in summaries. Neither metric completely evaluates meaning, factuality, coherence, or grammatical quality. Human evaluation and semantic metrics are often needed for a more complete assessment.

Classical sequence-to-sequence models without attention encode the entire input into a single fixed-size vector. This design creates several limitations:

• Fixed-vector bottleneck: The vector has limited capacity for representing long and complex inputs.
• Long-range dependency problems: Important information from early input positions may be lost.
• Poor performance on long sentences: Translation and summarization quality often decreases as sequence length increases.
• Weak word alignment: The model does not explicitly associate output tokens with relevant input tokens.
• Limited interpretability: It is difficult to inspect which input information caused an output.
• Sequential computation: Recurrent encoders and decoders process tokens step by step, reducing parallelism.
• Exposure bias: Training with teacher forcing differs from inference, where the model uses its own predictions.
• Error propagation: An incorrect generated token can affect later decoder states.

Attention reduces the information bottleneck, but recurrent sequence-to-sequence models may still face computational and optimization challenges.

Attention improves long-sequence processing by providing direct access from each decoder step to all encoder hidden states.

Without attention, information must travel through a single context vector or through many recurrent transitions. This can cause vanishing gradients and loss of earlier information. With attention:

• Every encoder position remains available to the decoder.
• The decoder can directly focus on distant input tokens.
• Different output tokens can use different context vectors.
• Alignment weights help identify relevant words even when source and target positions differ.
• Long-distance dependencies can be modeled through learned relationships between decoder states and encoder states.

For example, when translating a sentence, the decoder can assign high attention to a subject noun while generating its corresponding verb, even if many words occur between them. Therefore, attention reduces the burden on the final encoder state and improves both accuracy and interpretability.

Let be the alignment scores at decoder step . Softmax converts each score into an attention weight:

To show that the weights sum to one, add all weights:

The denominator is constant with respect to , so:

Because the exponential function is positive, every attention weight satisfies . Thus, the weights form a probability distribution over encoder positions. The context vector is consequently a convex combination of encoder hidden states:

Teacher forcing is a training strategy in which the correct target token from the previous time step is supplied to the decoder.

• It speeds up training and often improves convergence.
• It provides a reliable input history during learning.
• It creates exposure bias, because inference uses the model's own previous predictions rather than the correct tokens.

Beam search is a decoding algorithm that maintains the most likely partial output sequences.

• At every step, each partial sequence is expanded with possible next tokens.
• Candidate sequences are ranked using cumulative log probability.
• Only the best candidates are retained.
• A beam size of is equivalent to greedy decoding.

Beam search usually finds better sequences than greedy decoding, but it increases computational cost. It can also favor short or generic outputs unless length normalization, coverage penalties, or other decoding adjustments are used.

Attention weights indicate how strongly the model uses each encoder position when constructing a context vector for a particular decoder step.

• A high value of suggests that encoder position contributed strongly to the context used at output step .
• Visualizing the matrix of attention weights can reveal word alignments and translation patterns.
• Attention maps can help diagnose errors such as repeated focus, missing source words, or poor alignment.
• In summarization, attention may show which parts of a document influenced generated content.

However, attention is not necessarily a complete causal explanation:

• Other network components can transform or discard information.
• Different attention distributions may produce similar outputs.
• A high weight does not prove that a token was causally decisive.
• Attention may be diffuse or difficult to interpret for syntactic and semantic relationships.

Therefore, attention visualizations are useful diagnostic evidence, but they should be combined with ablation, gradient-based, or perturbation-based analysis.