Unit 3: Deep Learning Sequence Models for NLP

I. Orientation: Learning from ordered language

Deep learning sequence models represent language as an ordered stream in which each token contributes to the interpretation of later tokens. The central principle is conditional prediction: a model estimates the next token, a label for the whole sequence, or an output sequence using information accumulated from preceding or surrounding tokens.

A. Governing principle

• Sequential dependence: The meaning of a token depends on position and context; “not” can reverse the sentiment of “good.”
• Token representation: A sequence (x_1,x_2,\ldots,x_T) is usually converted into vectors, where (x_t) is the embedding at time (t).
• Hidden state: A state (h_t) summarizes relevant information available at position (t).
• Conditional modeling: Language generation commonly factorizes as:
  
  TEXT

    P(x1, x2, ..., xT) = ∏t P(xt | x1, ..., x(t-1))

  
  Here, (P) is probability and (T) is sequence length.
• Parameter sharing: The same recurrent parameters are applied at every time step, allowing one model to process sequences of different lengths.

II. Sequential text data: Ordered symbolic input

Sequential text data consists of tokens whose order, spacing, and context influence interpretation. Unlike fixed-size tabular input, text may have variable length and long-range dependencies.

A. Sequential text data

Sequential text data is represented as a time-ordered sequence of tokens, usually padded or masked so that batches can be processed efficiently.

• Token sequence: “Deep models learn” becomes ([x_1=\text{Deep}, x_2=\text{models}, x_3=\text{learn}]).
• Vocabulary mapping: Each token is assigned an integer index, then mapped to an embedding (e_t\in\mathbb{R}^d), where (d) is embedding dimension.
• Variable length: A batch may contain lengths (T_1=5) and (T_2=8); padding extends the shorter sequence, while a mask prevents padding from affecting loss.
• Context window: Recurrent models can carry information beyond a fixed window, although the quality of that information depends on the architecture and training.
• Input-output alignment: In next-token prediction, input tokens ((x1,\ldots,x{T-1})) are paired with targets ((x_2,\ldots,x_T)).

III. Recurrent neural networks: State carried through time

A recurrent neural network (RNN) processes one token at a time and updates a hidden state that acts as a running representation of the prefix.

A. Recurrent neural networks

An RNN uses the previous hidden state and current input to compute the next hidden state and output.

• Core recurrence:
  
  TEXT

    h_t = tanh(W_xh x_t + W_hh h_(t-1) + b_h)
    y_t = softmax(W_hy h_t + b_y)

  
  Here, (h_t) is the hidden state, (x_t) the input, (y_t) the output, (W) weight matrices, (b) biases, and (t) the time index.
• Parameter sharing: (W{xh}), (W{hh}), and (W_{hy}) remain unchanged across all (t), limiting parameter growth.
• Output modes: The model can produce one output per token, one output from the final state, or a sequence of outputs.
• Vanishing gradients: Repeated multiplication by derivatives smaller than 1 can make gradients approach zero, weakening learning across long sequences.
• Exploding gradients: Repeated multiplication by large values can produce unstable gradients; gradient clipping constrains the gradient norm, for example to a maximum of 5.

IV. Long short-term memory networks: Controlled memory

Long short-term memory (LSTM) networks extend RNNs with a cell state and gates that regulate information flow. Their design helps preserve useful information over long time intervals.

A. Long short-term memory networks

An LSTM separates persistent memory (c_t) from the exposed hidden state (h_t), using gates with sigmoid values between 0 and 1.

• Forget gate:
  
  TEXT

    f_t = σ(W_f [h_(t-1), x_t] + b_f)

  
  (f_t) determines which parts of the previous cell state are retained.
• Input and candidate states:
  
  TEXT

    i_t = σ(W_i [h_(t-1), x_t] + b_i)
    c~_t = tanh(W_c [h_(t-1), x_t] + b_c)

  
  (i_t) controls writing, while (\tilde{c}_t) is candidate content.
• Cell update:
  
  TEXT

    c_t = f_t ⊙ c_(t-1) + i_t ⊙ c~_t

  
  The operator (\odot) denotes element-wise multiplication.
• Output gate:
  
  TEXT

    o_t = σ(W_o [h_(t-1), x_t] + b_o)
    h_t = o_t ⊙ tanh(c_t)

  
  The output gate controls how much memory becomes visible.
• Concrete behavior: In “The movie was not good,” the forget and input gates can preserve “not” until “good” is processed.

V. Gated recurrent units: Compact recurrent memory

A gated recurrent unit (GRU) is a gated RNN that combines memory and hidden representation into a single state. It generally uses fewer gates and parameters than an LSTM.

A. Gated recurrent units

A GRU uses update and reset gates to decide whether to retain old information or compute a new candidate state.

• Update gate:
  
  TEXT

    z_t = σ(W_z x_t + U_z h_(t-1) + b_z)

  
  (z_t) controls the balance between the previous state and new information.
• Reset gate:
  
  TEXT

    r_t = σ(W_r x_t + U_r h_(t-1) + b_r)

  
  (r_t) controls how strongly the previous state contributes to the candidate.
• Candidate state:
  
  TEXT

    h~_t = tanh(W_h x_t + U_h (r_t ⊙ h_(t-1)) + b_h)

• State interpolation:
  
  TEXT

    h_t = (1 - z_t) ⊙ h_(t-1) + z_t ⊙ h~_t

  
  Some implementations reverse the interpretation of (z_t), so the exact equation should be checked against the framework.
• LSTM comparison: LSTMs have separate cell and hidden states plus three principal gates; GRUs use one state and two gates, often providing faster training with competitive accuracy.

VI. Bidirectional RNNs: Context from both directions

A bidirectional RNN reads a sequence from left to right and right to left, then combines both hidden representations. This is useful when the complete input is available before prediction.

A. Bidirectional RNNs

A bidirectional RNN forms a representation using past and future context at each position.

• Forward recurrence:
  
  TEXT

    h→_t = RNN(x_t, h→_(t-1))

  
  (h^\rightarrow_t) summarizes tokens before and including (t).
• Backward recurrence:
  
  TEXT

    h←_t = RNN(x_t, h←_(t+1))

  
  (h^\leftarrow_t) summarizes tokens after and including (t).
• Combination:
  
  TEXT

    h_t = [h→_t ; h←_t]

  
  The semicolon denotes vector concatenation.
• Advantage: In “bank near the river,” later words help interpret “bank” during tagging or classification.
• Limitation: A bidirectional model cannot directly operate in strictly real-time generation because future tokens are unavailable.

VII. Sequence modeling applications: From token streams to predictions

Sequence models support token-level, sequence-level, and sequence-to-sequence tasks. The required output structure determines where predictions are made.

A. Sequence modeling applications

Sequence modeling applications use recurrent representations to predict labels, tokens, or transformed sequences.

• Language modeling: Predict (x_t) from (x1,\ldots,x{t-1}), usually with cross-entropy loss.
• Sequence labeling: Produce one label per token, as in named-entity recognition: “Ada” (\rightarrow) PERSON.
• Machine translation: Encode a source sequence and decode a target sequence, often with attention in modern architectures.
• Speech and dialogue processing: Map acoustic or conversational sequences to words, intents, or responses.
• Streaming versus offline use: Unidirectional models support streaming; bidirectional models exploit complete input and usually suit offline processing.

B. Sentiment classification

Sentiment classification assigns a document-level polarity or rating by aggregating token information into a sequence representation.

• Input and output: A review (x_1,\ldots,x_T) produces one class, such as positive, negative, or neutral.
• Pooling strategy: Use the final hidden state (h_T), mean pooling (\frac{1}{T}\sum_t h_t), or an attention-weighted sum.
• Classifier:
  
  TEXT

    p = softmax(W_c h_doc + b_c)

  
  (h_{\text{doc}}) is the document vector and (p) contains class probabilities.
• Context effect: “Although slow, the film is excellent” requires integrating contrast across multiple tokens rather than counting positive words.
• Training loss: Categorical cross-entropy penalizes low probability assigned to the correct sentiment label.

C. Text classification

Text classification maps an entire text to a category such as topic, spam status, or intent.

• Feature extraction: The recurrent encoder converts (x_1,\ldots,xT) into (h{\text{text}}).
• Decision layer: A dense layer maps (h_{\text{text}}) to logits; sigmoid is common for binary or multilabel outputs, while softmax is common for mutually exclusive classes.
• Class imbalance: Weighted cross-entropy or resampling can prevent a rare class from being ignored.
• Evaluation split: Documents from the same source or user should be grouped appropriately to reduce train-test leakage.
• Interpretability: Token importance can be inspected through attention weights or perturbation, but neither alone proves causality.

VIII. Sequence training techniques: Optimizing temporal computation

Sequence training techniques control how recurrent models receive inputs and how gradients propagate through long computational graphs.

A. Sequence training techniques

Sequence training techniques are procedures that make optimization computationally feasible while preserving the temporal structure of the task.

• Teacher forcing: During training, the decoder receives the true previous target rather than its own previous prediction:
  
  TEXT

    decoder_input_t = y_(t-1)     # training
    decoder_input_t = ŷ_(t-1)     # inference

  
  (y) is the true token and (\hat{y}) the predicted token.
• Truncated backpropagation through time: The sequence is divided into windows of (K) steps; gradients are backpropagated within each window, while the hidden state may be carried forward without gradient history.
• Batching and padding: Similar-length sequences reduce padding waste; masks ensure padded positions contribute zero loss.
• Gradient clipping: If (|g|>c), rescale gradient (g) to (c g/|g|), where (c) is the clipping threshold.
• Scheduled sampling: The probability of feeding the true previous token can be gradually reduced, exposing the model to its own prediction errors.

B. Teacher forcing

Teacher forcing accelerates supervised sequence learning by supplying the correct history at every training step, but creates a difference between training and inference conditions.

• Benefit: For target sequence “I am ready,” the decoder receives “I” before predicting “am,” even if its earlier prediction was wrong.
• Exposure bias: At inference, one incorrect token can become input for the next step, producing errors not experienced during fully teacher-forced training.
• Use condition: It is especially common in autoregressive decoders where each output depends on the preceding target.
• Control methods: Scheduled sampling, noise injection, and sequence-level objectives can reduce dependence on ideal training histories.

C. Truncated backpropagation through time

Truncated backpropagation through time (TBPTT) limits the gradient path to a fixed number of time steps while retaining forward state across chunks.

• Full BPTT: For (T=10{,}000), storing every activation and differentiating through all steps can exceed memory limits.
• Truncation: With (K=50), process steps (1)-(50), detach the state, then process (51)-(100); the model still receives a state summary, but gradients do not cross the boundary.
• Trade-off: Smaller (K) reduces memory and computation but may prevent learning dependencies longer than (K) steps.
• Boundary handling: Hidden states should be detached between chunks when the intention is to truncate gradients; resetting them entirely discards forward context.

IX. Evaluation metrics for sequence tasks: Measuring appropriate behavior

Evaluation metrics for sequence tasks must match the output type: token labels, classes, generated text, or structured sequences. Accuracy alone can conceal important errors.

A. Evaluation metrics for sequence tasks

Metrics quantify prediction quality against reference labels or sequences, with different metrics emphasizing different forms of correctness.

• Token accuracy: Correct token predictions divided by total evaluated tokens; padding must be excluded from both numerator and denominator.
• Precision, recall, and F1:
  
  TEXT

    Precision = TP / (TP + FP)
    Recall    = TP / (TP + FN)
    F1        = 2PR / (P + R)

  
  (TP), (FP), and (FN) are true positives, false positives, and false negatives.
• Perplexity: For average negative log-likelihood (L), (\text{PPL}=e^L). Lower perplexity indicates better probabilistic prediction, though it does not guarantee better generated text.
• Cross-entropy: Measures the negative log probability assigned to the correct sequence:
  
  TEXT

    L = -(1/T) Σt log P(y_t | y_<t, x)

  
  (y_{<t}) denotes target tokens before (t).
• Sequence-level exact match: Requires the complete predicted sequence to equal the reference; it is strict and useful for exact structured outputs.
• BLEU and ROUGE: BLEU emphasizes modified n-gram precision, while ROUGE commonly emphasizes recall-oriented overlap; both can miss valid paraphrases.
• Classification metrics: Macro-F1 gives each class equal weight, while micro-F1 aggregates decisions and can favor frequent classes.
• Entity-level scoring: Named-entity recognition should often score complete spans, because labeling three of four tokens in one entity is not equivalent to identifying the full entity.
• Calibration and error analysis: Confusion matrices, length-based performance, and confidence calibration reveal whether errors concentrate in rare classes, long sequences, or boundary positions.