Unit 5: Modeling Sequential Data Using Recurrent Neural Networks

I. Orientation — Learning from Ordered Information

Sequential data consists of observations whose order carries information, such as words in a sentence, measurements over time, or frames in a video. A recurrent neural network (RNN) processes such data one step at a time while maintaining a hidden state that summarizes relevant earlier inputs.

• Governing principle: The prediction at time step (t) depends on the current input (xt) and a representation (h{t-1}) of previous inputs.
• Parameter sharing: The same weights are reused at every time step, allowing the model to process variable-length sequences.
• Temporal dependence: Recurrent connections enable information to flow from earlier to later positions.
• Sequence mappings:
  • Many-to-one: A sequence produces one output, as in sentiment classification.
  • One-to-many: One input produces a sequence, as in caption generation.
  • Many-to-many: An input sequence produces an output sequence, as in translation.
• Training convention: The network is unfolded across time and optimized through backpropagation through time (BPTT).
• Main limitation: Ordinary RNNs struggle to preserve information over long intervals because gradients may vanish or explode.
• Major extension: Long short-term memory (LSTM) units use gates and a cell state to regulate information flow.

II. Foundations of Recurrent Neural Networks — Representing Temporal Dependence

An RNN is a parameterized dynamical system in which the hidden state evolves as each sequence element is received. Its recurrence gives the model memory, while shared parameters make learning independent of a fixed sequence length.

A. Modeling sequential data

Modeling sequential data requires preserving order and representing dependencies between observations at different positions.

• Sequence notation: An input sequence is written as
  [
  X=(x_1,x_2,\ldots,x_T),
  ]
  where (x_t) is the input at time (t), and (T) is the sequence length.
• Order sensitivity: The sentences “dog bites man” and “man bites dog” contain similar tokens but express different relationships because their ordering differs.
• Input representation:
  • Numerical measurements may enter as feature vectors.
  • Words may be represented by dense embedding vectors.
  • Categorical events may be encoded using one-hot vectors.
• Probabilistic objective: Sequence generation often factorizes a joint probability as
  [
  P(x_1,\ldots,xT)=\prod{t=1}^{T}P(x_t\mid x1,\ldots,x{t-1}).
  ]
• Variable lengths: Mini-batches commonly use padding, while masks prevent padded positions from affecting loss or state calculations.
• Temporal direction: A unidirectional RNN uses past context; a bidirectional RNN combines forward and backward context when the complete sequence is available.

B. Understanding the structure and flow of an RNN

An RNN repeatedly applies one recurrent cell, passing its hidden state from one time step to the next.

• Core components:
  • (x_t): input vector at time (t).
  • (h_t): hidden-state vector containing the current sequence representation.
  • (y_t): output or prediction at time (t).
• Unrolled structure: The recurrent loop can be represented as copies of one cell connected across (t=1,\ldots,T); all copies share identical weights.
• Information flow:
  [
  (xt,h{t-1})\longrightarrow h_t\longrightarrow y_t.
  ]
• Initial state: (h_0) is usually a zero vector, although it may be learned or supplied by another network.
• Output choice:
  • Use every (h_t) for sequence labeling.
  • Use only (h_T) for many-to-one classification.
  • Feed each output into a decoder for sequence generation.
• Statefulness: A stateful RNN can carry its final state into the next related batch; unrelated sequences require state resetting.

C. Computing activation in an RNN

At each time step, an RNN combines the current input with the previous hidden state and applies a nonlinear activation.

• Hidden-state equation:
  [
  ht=\phi(W{xh}xt+W{hh}h_{t-1}+bh),
  ]
  where (W{xh}) maps input to hidden state, (W_{hh}) is the recurrent weight matrix, (b_h) is a bias, and (\phi) is usually (\tanh).
• Output equation:
  [
  \hat{y}t=g(W{hy}h_t+by),
  ]
  where (W{hy}) maps hidden state to output, (b_y) is an output bias, and (g) may be softmax, sigmoid, or a linear function.
• Classification output: For (K) classes, softmax computes
  [
  P(yt=k)=\frac{e^{z{t,k}}}{\sum{j=1}^{K}e^{z{t,j}}},
  ]
  where (z_{t,k}) is the logit for class (k).
• Parameter sharing: (W{xh}), (W{hh}), and (W_{hy}) remain unchanged across all time steps.
• Training loss: Many-to-many models commonly sum or average the loss over valid time steps; many-to-one models calculate loss from the final sequence representation.

D. Challenges of learning long-range interactions

Long-range learning is difficult because BPTT repeatedly multiplies derivatives through many recurrent steps.

• Vanishing gradients: If recurrent Jacobian magnitudes are mostly below (1), gradient products approach zero, so early states receive negligible updates.
• Exploding gradients: If those magnitudes exceed (1), gradients may grow exponentially and destabilize optimization.
• Activation saturation: Large positive or negative inputs push (\tanh) derivatives toward zero, worsening gradient decay.
• Practical consequences: A basic RNN may forget an early subject before predicting a verb appearing many words later.
• Mitigation methods:
  • Apply gradient clipping, such as limiting the global gradient norm to (1) or (5).
  • Use careful initialization, normalization, or truncated BPTT.
  • Replace simple recurrent cells with LSTM or gated recurrent unit cells.
• Trade-off: Truncated BPTT reduces memory and computation but prevents direct gradient propagation beyond the chosen truncation window.

III. TensorFlow RNN Construction — Building Deep Recurrent Models

TensorFlow and Keras provide recurrent layers that accept tensors shaped approximately as (batch, time_steps, features) and automatically perform recurrence across the time dimension.

A. Implementing a multilayer RNN for sequence modeling in TensorFlow

A multilayer RNN stacks recurrent layers so that one layer’s output sequence becomes the next layer’s input sequence.

• Stacking rule: Every recurrent layer except the last must normally specify return_sequences=True.
• Example architecture:

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.SimpleRNN(
        64, return_sequences=True,
        input_shape=(None, feature_count)
    ),
    tf.keras.layers.SimpleRNN(32),
    tf.keras.layers.Dense(class_count, activation="softmax")
])

model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

• Defined values: feature_count is the number of features per time step; class_count is the number of target categories; 64 and 32 are hidden-state dimensions.
• Tensor flow: The first RNN returns (batch, time, 64), the second returns (batch, 32), and the dense layer returns class probabilities.
• Regularization: dropout affects input connections, while recurrent_dropout affects recurrent connections; both can reduce overfitting but increase training cost.
• Masking: Embedding(mask_zero=True) or a Masking layer allows later recurrent layers to ignore padded positions.
• Limitation: More layers increase representational capacity but also computation, optimization difficulty, and overfitting risk.

IV. Sequence-Learning Applications — Classification, Generation, and Forecasting

The RNN architecture and output layer must match the task: classification predicts labels, generation predicts the next symbol, and forecasting predicts future numerical values.

A. Text classification with an RNN

Text classification uses a recurrent representation of a token sequence to predict a category such as topic, intent, or sentiment.

• Processing pipeline: Text is tokenized, converted to integer IDs, embedded, passed through an RNN, and mapped to class probabilities.
• Typical model:

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, 128, mask_zero=True),
    tf.keras.layers.SimpleRNN(64),
    tf.keras.layers.Dense(1, activation="sigmoid")
])

• Binary objective: Sigmoid output (\hat y) is trained with
  [
  L=-[y\log \hat y+(1-y)\log(1-\hat y)],
  ]
  where (y\in{0,1}) is the true label.
• Representation choice: The final hidden state summarizes the sequence, while pooling over all states may preserve more distributed evidence.
• Constraint: Basic RNNs may lose decisive words appearing far from the end; bidirectional or LSTM models usually handle such context better.

B. Text generation with an RNN

Text generation trains an RNN to predict the next token from preceding tokens and then repeatedly feeds generated tokens back into the model.

• Training pairs: For “deep learning works,” the input might be “deep learning,” with targets “learning works,” shifted by one position.
• Output distribution: A softmax layer gives (P(x{t+1}\mid x{\leq t})) over the vocabulary.
• Generation loop:

state = initial_state
token = start_token
repeat:
    probabilities, state = model(token, state)
    token = sample(probabilities)
    append token to output

• Sampling temperature:
  [
  P_i=\operatorname{softmax}(z_i/\tau),
  ]
  where (z_i) is a token logit and (\tau) is temperature. Low (\tau) produces conservative text; high (\tau) increases diversity.
• Exposure problem: Training uses true previous tokens, whereas generation uses the model’s own outputs, allowing errors to accumulate.

C. Time series forecasting

Time series forecasting predicts future numerical observations from an ordered window of past measurements.

• Windowing: Given window length (w), inputs ((x_{t-w+1},\ldots,xt)) may predict (x{t+1}) or several future values.
• Regression output: A dense layer with linear activation is appropriate for unrestricted continuous targets.
• Loss functions:
  [
  \text{MSE}=\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat yi)^2,
  \qquad
  \text{MAE}=\frac{1}{N}\sum{i=1}^{N}|y_i-\hat y_i|.
  ]
• Data preparation: Training, validation, and test sets must preserve chronological order; random splitting can leak future information.
• Scaling: Features are commonly standardized using statistics calculated only from the training period.
• Forecast modes: Direct multi-step forecasting predicts all horizons together; recursive forecasting repeatedly reuses earlier predictions and may accumulate error.

V. Long Short-Term Memory Networks — Controlled Information Retention

LSTM networks replace the basic recurrent update with a memory cell and learned gates. This creates additive state transitions that improve gradient flow and support longer temporal dependencies.

A. LSTM units

An LSTM unit controls what information is forgotten, written, and exposed through sigmoid gates.

• Gate equations:
  [
  f_t=\sigma(W_f[xt,h{t-1}]+b_f),\quad
  i_t=\sigma(W_i[xt,h{t-1}]+b_i),
  ]
  [
  \tilde c_t=\tanh(W_c[xt,h{t-1}]+b_c),\quad
  o_t=\sigma(W_o[xt,h{t-1}]+b_o).
  ]
• State updates:
  [
  c_t=ft\odot c{t-1}+i_t\odot\tilde c_t,\qquad
  h_t=o_t\odot\tanh(c_t).
  ]
• Symbol meanings: (f_t), (i_t), and (o_t) are forget, input, and output gates; (c_t) is cell state; (\tilde c_t) is candidate memory; (\odot) denotes element-wise multiplication.
• Gradient advantage: The additive cell-state path can retain information without repeatedly replacing the entire state.
• Cost: Four learned transformations make an LSTM more computationally expensive than a simple RNN.

B. Sequence classification with LSTM

LSTM classification uses the final hidden state or pooled output states as a feature representation for predicting a sequence label.

• Keras form:

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, 128, mask_zero=True),
    tf.keras.layers.LSTM(64),
    tf.keras.layers.Dense(class_count, activation="softmax")
])

• Classification logic: The LSTM output has shape (batch, 64) and the dense layer converts it into class_count probabilities.
• Strength: Gated memory can preserve earlier evidence, such as a negation that changes sentiment later in a sentence.
• Bidirectional option: Bidirectional(LSTM(64)) combines past-to-future and future-to-past representations when future context is available.

C. Stacked LSTM for sequence classification

A stacked LSTM places multiple LSTM layers in depth to learn increasingly abstract temporal features.

• Implementation:

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, 128, mask_zero=True),
    tf.keras.layers.LSTM(128, return_sequences=True),
    tf.keras.layers.LSTM(64),
    tf.keras.layers.Dense(class_count, activation="softmax")
])

• Layer roles: The first LSTM supplies a hidden vector for every time step; the second condenses that sequence into one classification representation.
• Potential benefit: Lower layers can capture local patterns, while higher layers combine them into broader sequence-level structure.
• Limitations: Stacking increases parameter count, latency, and overfitting risk; dropout, early stopping, and validation monitoring are therefore important.