Unit 4: Sequence-to-Sequence Models and Attention Mechanisms

CSE472 — Deep Learning For Natural Language Processing 1 min read

I. Orientation

Sequence-to-sequence (seq2seq) learning maps an input sequence to an output sequence whose length and structure may differ. Introduced prominently with neural machine translation using recurrent neural networks, the framework separates input encoding from output generation and later uses attention to overcome fixed-length representation limits.

Governing principle: Estimate the conditional probability of a target sequence (y{1:T}) given a source sequence (x{1:S}):
[
P(y\mid x)=\prod_{t=1}^{T}P(yt\mid y{<t},x)
]
Core convention: The source and target may have different lengths, vocabularies, and languages.
Training assumption: The decoder commonly uses teacher forcing, receiving the true previous token (y_{t-1}) during training.
Generation convention: At inference, the decoder uses its own previously generated token, beginning with <BOS> and ending at <EOS>.
Central limitation: A basic encoder compresses the whole source into one context vector, which can lose information for long sequences.
Attention principle: At each decoding step, dynamically weight encoder states rather than relying only on one fixed vector.

II. Encoder-decoder architectures for NLP

An encoder-decoder architecture contains an encoder that represents the input and a decoder that produces the output token by token.

A. Encoder-decoder architectures for NLP

The architecture is designed for conditional generation tasks in which an output sequence depends on an input sequence.

Encoder: For input tokens (x_1,\ldots,x_S), a recurrent encoder computes:
[
ht=f{\text{enc}}(xt,h{t-1})
]
where (ht) is the hidden state and (f{\text{enc}}) may be an LSTM or GRU.
Context representation: A classical model passes the final encoder state (h_S) to the decoder as a summary of the source.
Decoder: The decoder predicts:
[
st=f{\text{dec}}(y{t-1},s{t-1},c)
]
where (s_t) is the decoder state and (c) is the context vector.
Output distribution: A softmax converts decoder activations into probabilities over the target vocabulary:
[
P(yt\mid y{<t},x)=\operatorname{softmax}(W_os_t+b_o)
]
Concrete structure: For translation, the encoder reads “I like cats,” while the decoder generates an equivalent target sentence one token at a time.

III. Sequence-to-sequence models for machine translation and summarization

Seq2seq models learn to generate a target sequence conditioned on a source sequence, making them suitable for translation and abstraction of text.

A. Sequence-to-sequence models for machine translation and summarization

The same probabilistic framework supports both language conversion and content compression, but the desired output relationship differs.

Machine translation: The source is a sentence in one language and the target is its translation. The model must preserve meaning while changing vocabulary and syntax.
- Example: “She reads books” may become “Elle lit des livres,” requiring reordering and morphological adaptation.
Summarization: The source is a document and the target is a shorter summary. The decoder must select salient information and formulate new sentences.
Training objective: Minimize token-level negative log-likelihood:
[
\mathcal{L}=-\sum_{t=1}^{T}\log P(yt^\ast\mid y{<t}^\ast,x)
]
where (y_t^\ast) is the reference token.
Decoding: Greedy decoding selects the highest-probability token at each step; beam search retains several partial hypotheses to improve sequence-level output.
Task distinction: Translation generally requires semantic faithfulness across languages, whereas summarization permits omission but risks hallucination and unsupported content.

IV. Attention in deep NLP

Attention is a differentiable mechanism that lets a decoder select the most relevant parts of an encoded input at each generation step.

A. Attention in deep NLP

Attention replaces a single static source summary with a context vector that changes according to the current decoder state.

Purpose: At time (t), the decoder should focus on source positions relevant to predicting (y_t); translating a target noun may require a different source position than translating its verb.
Encoder states: Instead of retaining only (h_S), the model stores (h_1,\ldots,h_S).
Dynamic context: The context vector is a weighted sum:
[
ct=\sum{i=1}^{S}\alpha_{t,i}hi
]
where (\alpha{t,i}) is the attention weight assigned to source position (i).
Interpretability: The matrix of weights (\alpha_{t,i}) can indicate source-target correspondence, although it is not a complete explanation of model reasoning.
Benefit: Direct access to all encoder states improves gradient flow and reduces information loss for long sequences.

B. Soft attention

Soft attention assigns continuous, differentiable weights to every encoder state, allowing the entire mechanism to be trained with backpropagation.

Normalization: Scores (e{t,i}) become probabilities through softmax:
[
\alpha{t,i}=\frac{\exp(e{t,i})}{\sum{j=1}^{S}\exp(e_{t,j})}
]
Properties: Each (\alpha_{t,i}\geq0), and (\sumi\alpha{t,i}=1), so (c_t) is a convex combination of encoder states.
Contrast with hard attention: Hard attention samples one or a few positions and is nondifferentiable; soft attention uses all positions with varying influence.
Worked example: If weights for three source tokens are ((0.1,0.7,0.2)), the second token contributes most strongly to the current decoder context.
Computational cost: Every decoder step compares against all (S) encoder positions, producing approximately (O(ST)) attention-score computations.

C. Alignment mechanisms

An alignment mechanism computes how compatible a decoder state is with each encoder state before normalization.

Alignment score:
[
e{t,i}=a(s{t-1},h_i)
]
where (a) is a learned compatibility function.
Meaning of indices: (i) identifies a source position and (t) identifies the target decoding step.
Additive alignment: Uses a feed-forward network to combine (s_{t-1}) and (h_i), useful when their dimensions differ.
Dot-product alignment: Computes (s_t^\top h_i), which is efficient when both vectors have the same dimensionality.
Role in learning: The model is not given word alignments; it learns them indirectly because useful alignments reduce prediction loss on target tokens.

D. Bahdanau attention

Bahdanau attention, introduced for neural machine translation in 2015, uses additive attention and typically conditions alignment on the previous decoder state.

Score function:
[
e_{t,i}=v_a^\top\tanh(Was{t-1}+U_ah_i)
]
where (W_a,U_a,v_a) are learned parameters.
Context construction: Apply softmax to (e_{t,i}), then calculate (c_t=\sumi\alpha{t,i}h_i).
Decoder input: The context is combined with the decoder state to predict the next token, often through:
[
P(y_t\mid\cdot)=\operatorname{softmax}(W_o[s_t;c_t]+b_o)
]
Distinctive feature: Alignment is calculated before or alongside the current decoding operation using (s_{t-1}), allowing a flexible nonlinear comparison.
Practical effect: It performs well when source and target sequences have different lengths or require substantial reordering.

E. Luong attention

Luong attention, proposed in 2015, provides efficient multiplicative attention variants and commonly uses the current decoder state for alignment.

Dot score:
[
e_{t,i}=s_t^\top h_i
]
General score:
[
e_{t,i}=s_t^\top W_ah_i
]
where (W_a) is learned.
Concat score:
[
e_{t,i}=v_a^\top\tanh(W_a[s_t;h_i])
]
Comparison: Dot attention is fastest, while general and concat forms add learned transformations or nonlinear interaction.
Decoder integration: Luong models may combine the attentional hidden state with (s_t) before the final softmax, rather than treating context as only an input.

F. Integrating attention into encoder-decoder networks

Integration requires calculating attention at every decoding step and feeding the resulting context into the decoder’s prediction pathway.

Process: Encode all source tokens, compute scores against the current decoder state, normalize scores, form (c_t), and predict (y_t).

Pseudocode:

TEXT

  H = Encoder(x1, ..., xS)
  s0 = initial_decoder_state
  for t = 1 ... T:
      et[i] = Score(st-1, H[i])
      alpha = softmax(e_t)
      ct = sum_i alpha[i] * H[i]
      st = Decoder(y[t-1], st-1, ct)
      y[t] = softmax(Output(st, ct))

Teacher forcing: During training, y[t-1] is usually the reference token; during inference, it is the model’s previous prediction.
Architectural variants: Bidirectional encoders represent both left and right context, often concatenating forward and backward states.
Operational issue: Beam search can improve output quality but increases decoding time and memory use.

V. Evaluation techniques

Evaluation compares generated sequences with human references while considering adequacy, fluency, informativeness, and task-specific correctness.

A. Evaluation techniques

Reliable evaluation combines automatic metrics with human or task-level assessment because lexical overlap does not fully capture meaning.

Reference-based evaluation: Compare a candidate translation or summary with one or more reference texts.
Corpus-level scoring: Aggregate evidence across many examples rather than interpreting one sentence-level score.
Human evaluation: Translators or annotators may rate adequacy and fluency for translation, or coherence, relevance, and factuality for summaries.
Error analysis: Inspect omissions, repetitions, incorrect word order, untranslated tokens, and hallucinated facts.
Task awareness: A summary with different wording may be semantically excellent but receive a low overlap score, so metrics should be interpreted alongside qualitative checks.

B. BLEU scores

BLEU evaluates machine translation primarily through modified (n)-gram precision and a brevity penalty.

Modified precision: For each (n), candidate (n)-grams are clipped by their maximum reference count:
[
p_n=\frac{\sumg\min(\operatorname{count}{cand}(g),\operatorname{count}_{ref}(g))}
{\sumg\operatorname{count}{cand}(g)}
]
Geometric combination:
[
\operatorname{BLEU}=BP\cdot\exp\left(\sum_{n=1}^{N}w_n\log p_n\right)
]
where (w_n) are weights, commonly (w_n=0.25) for (n=1,\ldots,4).
Brevity penalty:
[
BP=
\begin{cases}
1,& c>r\
e^{1-r/c},& c\le r
\end{cases}
]
where (c) is candidate length and (r) is effective reference length.
Interpretation: Higher BLEU indicates greater (n)-gram overlap, but scores are meaningful mainly under the same dataset, tokenization, and evaluation procedure.
Limitation: It penalizes valid paraphrases and may reward frequent surface phrases without verifying meaning.

C. ROUGE scores

ROUGE measures overlap between generated and reference summaries, especially recall of reference content.

ROUGE-1: Unigram overlap; it indicates shared words and broad content coverage.
ROUGE-2: Bigram overlap; it captures some local fluency and phrase structure.
ROUGE-L: Uses the longest common subsequence (LCS), preserving in-order matching without requiring contiguous words.
Recall, precision, and F-measure:
[
R=\frac{\text{overlap}}{\text{reference units}},\quad
P=\frac{\text{overlap}}{\text{candidate units}},\quad
F_1=\frac{2PR}{P+R}
]
Interpretation: ROUGE recall is useful for checking whether important reference content appears, but high overlap does not guarantee factual correctness or coherence.
Concrete caution: A copied sentence can obtain strong ROUGE while containing a factual error inherited from the source.

VI. Limitations of classical sequence-to-sequence models

Classical recurrent seq2seq systems established the encoder-decoder paradigm but face optimization, representation, and generation problems.

A. Limitations of classical sequence-to-sequence models

A fixed-vector recurrent encoder must compress an arbitrarily long sequence, while recurrent computation also limits parallelism and can weaken long-range dependency learning.

Fixed-length bottleneck: Without attention, every source sentence is represented by one vector (c=h_S); details from early tokens may be lost as (S) grows.
Vanishing and exploding gradients: Repeated multiplication through recurrent transitions can make gradients shrink or grow, although LSTM and GRU gates reduce this risk rather than eliminating it.
Sequential computation: Each state depends on the previous state, so encoding and decoding cannot be fully parallelized across time steps.
Exposure bias: Training conditions on reference history, but inference conditions on generated history; one early error can alter all later predictions.
Repetition and omission: Autoregressive decoders may repeat phrases, skip source content, or stop too early, especially in long summaries.
Weak factual control: Summarization models can generate fluent statements unsupported by the source.
Limited interpretability: Attention patterns show associations, but they do not prove that the model used a particular source token causally.
Evaluation mismatch: BLEU and ROUGE measure surface overlap and may fail to reflect adequacy, semantic equivalence, coherence, or factuality.