Unit 5: Modeling Sequential Data Using Recurrent Neural Networks - Subjective Questions

Sequential data consists of observations arranged in a meaningful order, where the position and context of an observation influence its interpretation. Examples include words in a sentence, daily stock prices, speech signals, and sensor readings.

Unlike ordinary fixed-size data, sequential data has the following characteristics:

• Order dependence: Changing the order of observations can change the meaning.
• Variable length: Different sequences may contain different numbers of elements.
• Temporal or contextual relationships: Earlier observations may influence later ones.
• Non-independent observations: Consecutive values are often correlated.

For example, in the sentence "The dog chased the cat," the word order affects the meaning. Similarly, the temperature recorded today may depend on temperatures from previous days. Recurrent neural networks are designed to model such dependencies by processing one element at a time while maintaining a hidden state.

A recurrent neural network processes a sequence one time step at a time. At time step , it receives the current input and the hidden state from the previous time step, .

The hidden state is computed as:

The output is then calculated using:

where:

• is the input at time step .
• is the current hidden state or memory.
• carries information from the previous time step.
• maps the input to the hidden layer.
• is the recurrent weight matrix.
• maps the hidden state to the output.
• and are bias vectors.
• and are activation functions.

The same weights are reused at every time step. This allows an RNN to handle sequences of different lengths while sharing learned temporal patterns.

For a simple RNN, the activation at time step is obtained by combining the current input with the previous hidden state. The pre-activation is:

Applying a nonlinear activation function gives:

For a scalar example, let , , , , and . Using the hyperbolic tangent activation function:

Therefore:

The new hidden state, approximately , summarizes the current input together with information retained from the previous time step. In vector form, the same computation is performed using matrix multiplication and vector addition.

Backpropagation through time (BPTT) is the training method used to optimize an RNN over a sequence. The recurrent network is conceptually unfolded across all time steps, producing a chain of repeated network cells.

The procedure involves:

• Processing the sequence in the forward direction.
• Computing the loss at one or more time steps.
• Propagating gradients backward from later time steps toward earlier time steps.
• Accumulating gradients for shared recurrent weights.
• Updating the parameters using an optimization algorithm such as gradient descent.

If the total loss is , the gradient with respect to a recurrent parameter includes contributions from multiple time steps:

BPTT is necessary because the current output depends not only on the current input but also on previous hidden states. It allows the model to learn temporal relationships. However, when sequences are long, repeated multiplication of Jacobian matrices can produce vanishing or exploding gradients, making long-range learning difficult.

During BPTT, gradients are repeatedly propagated through the recurrent connections. The gradient at an early time step contains products of terms involving the recurrent weight matrix and derivatives of the activation function.

A simplified form is:

Two major problems occur:

• Vanishing gradients: If the repeated factors have magnitudes less than one, the product becomes extremely small. Early time steps receive almost no useful learning signal, so the RNN cannot remember information over long intervals.
• Exploding gradients: If the factors have magnitudes greater than one, the product becomes extremely large. Training becomes unstable and parameter updates may diverge.

These problems are caused by long chains of multiplication, saturating activation functions, and unsuitable parameter initialization. Common remedies include gradient clipping, careful initialization, truncated BPTT, normalization, and gated architectures such as LSTM and GRU. LSTM units are particularly effective because their cell state provides a controlled path for information and gradients.

Unrolling and stacking represent two different dimensions of an RNN architecture.

• Unrolling across time: A single RNN cell is repeated for each time step in a sequence. The same parameters are reused, and the hidden state flows from one time step to the next. This captures temporal dependencies.
• Stacking layers: Multiple recurrent layers are placed vertically. The output sequence from one recurrent layer becomes the input sequence to the next layer. Each layer can learn a different level of representation.

For a two-layer stacked RNN, the computation can be represented as:

Unrolling is essential for processing time. Stacking increases model depth and representational power. A shallow RNN may be sufficient for simple patterns, while stacked RNNs are useful for complex language, speech, or sensor data. However, increasing depth also increases computational cost and the risk of overfitting.

A multilayer RNN can be implemented in TensorFlow using recurrent layers such as tf.keras.layers.SimpleRNN, tf.keras.layers.LSTM, or tf.keras.layers.GRU.

A typical architecture is:

• An input layer with shape (time_steps, features).
• A first recurrent layer with return_sequences=True so that it produces an output for every time step.
• A second recurrent layer. Its return_sequences value depends on the task.
• A dense output layer suitable for regression or classification.

For example, the conceptual code is:

model = tf.keras.Sequential([

tf.keras.layers.Input(shape=(time_steps, features)),

tf.keras.layers.SimpleRNN(64, return_sequences=True),

tf.keras.layers.SimpleRNN(32),

tf.keras.layers.Dense(num_classes, activation="softmax")

])

For sequence-to-sequence prediction, the final recurrent layer should usually use return_sequences=True. For sequence classification, only the final representation may be needed, so the last recurrent layer can return a single hidden vector. The loss function and final activation must match the task, such as softmax with categorical cross-entropy for multiclass classification.

Text classification with an RNN generally follows these stages:

• Text cleaning: Normalize case and handle punctuation, special symbols, or missing text.
• Tokenization: Split each document into tokens such as words or subwords.
• Vocabulary construction: Assign an integer index to each token.
• Sequence conversion: Replace tokens with their integer indices.
• Padding and truncation: Convert sequences to a common length for batch processing.
• Embedding: Map each token index to a dense vector using an embedding layer.
• Sequence processing: Pass the embedding sequence through an RNN, LSTM, or GRU.
• Classification: Feed the final hidden representation into a dense output layer.

For binary classification, the output may be:

where is the sigmoid function. Binary cross-entropy is commonly used as the loss. For multiclass classification, a softmax output and categorical cross-entropy are appropriate. Padding masks should be handled correctly so that artificial padding tokens do not influence the learned representation.

In text generation, an RNN predicts the next token based on the preceding tokens. Given a sequence , the model estimates:

During training, the input sequence is shifted by one position to create target labels. For example, the input may contain "deep learn" and the target may contain "eep learn" at the character level, or the next word at each position at the word level. The model is trained using categorical cross-entropy.

During generation:

• A seed text initializes the hidden state.
• The RNN predicts a probability distribution for the next token.
• A token is selected using greedy selection or random sampling.
• The selected token is appended to the input.
• The process repeats until a stopping condition is reached.

Temperature modifies the probability distribution. A low temperature makes generation more predictable, while a high temperature produces more diverse but potentially less coherent text. Sampling should use a valid vocabulary and include safeguards such as maximum length and an end-of-sequence token.

For time series forecasting, an RNN receives a window of previous observations and predicts one or more future values. If a series is , a window of length can be used as:

with target:

The main steps are:

• Sort observations chronologically.
• Handle missing values and outliers.
• Normalize or standardize the training data.
• Create sliding input windows and corresponding targets.
• Split data chronologically into training, validation, and test sets.
• Train an RNN or LSTM using a regression loss such as mean squared error.
• Invert normalization before interpreting predictions.

Common metrics include mean absolute error, mean squared error, and root mean squared error:

Chronological splitting is important because randomly mixing future observations into training data causes data leakage.

An LSTM uses a cell state and three gates to regulate information flow. Given input and previous hidden state , the equations are:

Here:

• is the forget gate, which controls information removed from the cell state.
• is the input gate, which controls new information written to memory.
• is the candidate cell content.
• is the updated long-term memory.
• is the output gate.
• is the exposed hidden state.
• denotes element-wise multiplication.

The additive cell-state update helps gradients flow over long sequences and reduces the limitations of a basic RNN.

A basic RNN and an LSTM both process sequences recurrently, but they differ in how they preserve information.

An RNN update is approximately:

An LSTM selectively decides what to forget, store, and expose. This improves performance when relevant information is separated by many time steps. The trade-off is increased memory usage, computation, and training time.

In sequence classification, the input is a complete sequence and the output is one class label. Examples include sentiment classification, activity recognition, and medical record classification.

A typical LSTM classifier works as follows:

• Each input sequence is represented as a matrix of features or token embeddings.
• The LSTM processes the sequence in temporal order.
• The final hidden state or a pooled representation summarizes the sequence.
• A dense layer converts the representation into class scores.
• A sigmoid or softmax layer produces class probabilities.

For multiclass classification:

The predicted class is usually the index with the largest probability. The model is trained by comparing with the true label using categorical cross-entropy. Padding must be masked when sequences have different lengths. Dropout, recurrent dropout, and early stopping can be used to reduce overfitting.

A stacked LSTM contains multiple LSTM layers arranged vertically. The first layer processes the input sequence and returns a hidden vector at every time step. This complete output sequence is passed to the next LSTM layer.

For two layers:

The first layer may learn local or lower-level temporal patterns, while higher layers can combine them into more abstract patterns. In TensorFlow, intermediate LSTM layers need return_sequences=True so that the next layer receives a sequence rather than only the final vector.

Advantages include:

• Greater representational capacity.
• Ability to learn hierarchical temporal features.
• Improved performance on complex sequence classification tasks.

Disadvantages include increased computational cost, training difficulty, and overfitting risk. Dropout, regularization, validation monitoring, and appropriate hidden sizes are important when designing a stacked model.

RNN applications can be categorized according to the number of input and output elements:

• One-to-one: A single input produces a single output. This is the ordinary feed-forward setting and is not inherently sequential.
• One-to-many: A single input produces a sequence of outputs. Image captioning is an example, where an image representation generates a sentence.
• Many-to-one: A sequence produces one output. Sentiment analysis maps a sentence to a sentiment class.
• Many-to-many: A sequence produces an output sequence. Machine translation, part-of-speech tagging, and next-token prediction are examples.

In a many-to-one classifier, the final hidden state may be connected to a classification layer. In a many-to-many model, output layers may be applied at every time step. For example, language modeling predicts:

The arrangement determines the shape of the target data, the required recurrent layer configuration, and where the loss is calculated.

Teacher forcing trains a sequence model by providing the true previous token as input when predicting the next token, rather than using the model's previous prediction.

For target sequence , training uses:

where the previous values are the ground-truth tokens. Its advantages are:

• Faster convergence.
• More stable training.
• Clear learning targets at every time step.
• Efficient parallel computation of training losses in some architectures.

The main limitation is exposure bias. During training, the model sees correct previous tokens, but during inference it must use its own generated tokens. An early incorrect prediction can therefore cause later errors to accumulate. Possible solutions include scheduled sampling, where the model gradually replaces true tokens with generated tokens, sequence-level objectives, beam search, and improved decoding strategies. The probability of selecting the true token in scheduled sampling can be decreased gradually during training.

An embedding layer maps each discrete token index to a dense, trainable vector. If the vocabulary size is and the embedding dimension is , the embedding matrix is:

A token with index is represented by the row .

Embeddings are preferred over one-hot vectors because:

• Lower dimensionality: A one-hot vector has dimension , whereas an embedding may have dimension where .
• Similarity representation: Similar words can acquire similar vector representations.
• Learnability: The vectors are adjusted during training for the task.
• Computational efficiency: Dense matrix operations are more useful than extremely sparse vectors.

The embedding sequence is supplied to the RNN, which learns how word representations interact over time. Pretrained embeddings may provide useful general semantic information, while task-specific embeddings can adapt to the vocabulary and domain of the dataset.

Suppose an RNN receives input windows and predicts a continuous target . Let the prediction be , where represents all trainable parameters.

The mean squared error over examples is:

For a single example, the derivative with respect to the prediction is:

This error is propagated through the output layer, recurrent states, and input window using BPTT. The optimizer then updates the parameters:

where is the learning rate. MSE strongly penalizes large errors, making it useful when accurate numerical predictions are required. For noisy series or outlier-prone data, mean absolute error or a robust loss may be preferable.

Sequences in a batch usually need a common length. Shorter sequences are extended with a special padding token, while longer sequences may be truncated. For example, sequences can be padded using a token index such as zero.

Padding can cause problems because the recurrent model may interpret padded positions as real input. Masking solves this by marking padded positions as invalid and preventing them from affecting recurrent computations or loss calculations.

Important considerations include:

• Place padding consistently, usually at the beginning or end of a sequence.
• Add a masking layer or configure the embedding layer with mask_zero=True when supported.
• Ensure that subsequent layers can propagate masks.
• Exclude padded positions from sequence-level metrics and losses.
• Use attention or pooling operations that respect the mask.

Correct masking is especially important for text classification and sequence labeling. Without it, the model may learn artifacts related to sequence length or padding patterns instead of meaningful temporal information.

In sequence classification, an entire input sequence is mapped to one label. The network may use the final hidden state :

The loss is computed once per sequence, commonly using categorical cross-entropy.

In sequence-to-sequence prediction, the model produces an output at every time step. For example:

The loss is usually summed or averaged over valid time steps:

A sequence classification model generally needs only the final representation, so the final recurrent layer can return one vector. A sequence-to-sequence model requires the recurrent layer to return the complete output sequence. Examples include sentiment classification for sequence classification and part-of-speech tagging or language modeling for sequence-to-sequence prediction.