Unit 4: Introduction to deep neural networks; Modern NLP - Subjective Questions

An Artificial Neural Network (ANN) is a computational model inspired by the structure and functioning of biological neural networks in the human brain. It consists of interconnected processing units called neurons that work together to learn patterns from data.

Biological Inspiration:

• Biological neurons receive signals through dendrites, process them in the cell body, and transmit output through the axon.
• ANNs mimic this by receiving inputs, applying weights, summing them, and passing the result through an activation function.

Components of an Artificial Neuron:

• Inputs (): Feature values fed into the neuron.
• Weights (): Represent the strength/importance of each input.
• Bias (): A constant that shifts the activation.
• Summation function: Computes the weighted sum .
• Activation function (): Introduces non-linearity, producing output .

Common activation functions include Sigmoid, ReLU, and Tanh. ANNs learn by adjusting weights through training algorithms like backpropagation.

The Perceptron is the simplest type of artificial neural network, introduced by Frank Rosenblatt (1958). It is a binary linear classifier used for supervised learning.

Working:

• It computes a weighted sum of inputs: .
• Applies a step activation function:
  

Weight Update Rule (Derivation):
During training, weights are updated to reduce classification error. If is the target output and is the predicted output:


where is the learning rate. The term is the error:

• If prediction is correct, , no update.
• If wrong, weights shift toward correct classification.

Limitations:

• Can only solve linearly separable problems.
• Cannot solve the XOR problem (famously shown by Minsky & Papert).
• Uses a hard step function, making it non-differentiable and unable to output probabilities.

These limitations led to the development of Multi-Layer Perceptrons (MLPs).

A Multi-Layer Perceptron (MLP) is a feedforward neural network consisting of multiple layers of neurons that can learn non-linear relationships.

Architecture:

• Input Layer: Receives feature values.
• Hidden Layer(s): One or more layers where non-linear transformations occur using activation functions.
• Output Layer: Produces the final prediction (classification/regression).
• All neurons between adjacent layers are fully connected.

Backpropagation Algorithm:
Backpropagation trains the MLP by minimizing a loss function using gradient descent.

1. Forward Pass: Inputs propagate through the network to compute the output and loss .
2. Backward Pass: Compute the gradient of the loss with respect to each weight using the chain rule:
   
3. Weight Update: Adjust weights in the direction that reduces loss:
   

Key Points:

• Uses differentiable activation functions (e.g., Sigmoid, ReLU).
• Solves non-linearly separable problems like XOR.
• Prone to issues like vanishing gradients in deep networks.

A Convolutional Neural Network (CNN) is a specialized deep learning architecture designed to process grid-like data, such as images. It automatically learns spatial hierarchies of features.

Key Layers:

• Convolutional Layer: Applies learnable filters/kernels that slide over the input to detect features like edges and textures. Produces feature maps via the operation:
  
• Activation Layer (ReLU): Introduces non-linearity by setting negative values to zero.
• Pooling Layer: Reduces spatial dimensions (e.g., Max Pooling), providing translation invariance and reducing computation.
• Fully Connected Layer: Flattens features and performs final classification.

Why Suited for Images:

• Parameter sharing: The same filter is applied across the image, reducing parameters.
• Local connectivity: Captures spatial locality of pixels.
• Translation invariance: Recognizes features regardless of position.
• Hierarchical feature learning: Early layers detect simple features, deeper layers detect complex patterns.

CNNs power applications like image classification, object detection, and facial recognition.

A Recurrent Neural Network (RNN) is a type of neural network designed to process sequential data such as text, speech, and time-series by maintaining a memory of previous inputs.

Architecture & Working:

• Unlike feedforward networks, RNNs have loops that allow information to persist.
• At each time step , the hidden state is computed from the current input and the previous hidden state :
  
  
• The same weights are shared across all time steps (parameter sharing over time).

Problem It Solves:

• Handles variable-length sequences and captures temporal dependencies where the order of data matters.

Main Challenges:

• Vanishing Gradient Problem: Gradients shrink during backpropagation through time, making it hard to learn long-term dependencies.
• Exploding Gradients: Gradients can grow uncontrollably.
• Short-term memory: Difficulty remembering information from far back.

These issues led to improved variants like LSTM and GRU, which use gating mechanisms to control information flow.

The Transformer, introduced in the paper "Attention Is All You Need" (Vaswani et al., 2017), is a deep learning architecture that relies entirely on attention mechanisms, eliminating recurrence and convolution.

Overall Structure:
It consists of an Encoder and a Decoder, each composed of stacked identical layers.

Encoder:

• Processes the input sequence into rich contextual representations.
• Each layer has a Multi-Head Self-Attention sublayer and a Feed-Forward Network.

Decoder:

• Generates output sequences one token at a time.
• Contains masked self-attention (prevents looking at future tokens), encoder-decoder attention, and a feed-forward network.

Self-Attention Mechanism:
Self-attention allows each token to attend to all other tokens. It uses Query (Q), Key (K), and Value (V) matrices:

• The dot product measures relevance between tokens.
• Scaling by stabilizes gradients.

Other Key Components:

• Positional Encoding: Injects word order information since there is no recurrence.
• Multi-Head Attention: Captures different types of relationships in parallel.

Advantages: Parallel processing, better handling of long-range dependencies, and scalability, forming the foundation of models like BERT and GPT.

The Self-Attention mechanism enables a model to weigh the importance of different words in a sequence relative to each other, capturing contextual relationships regardless of distance.

Mathematical Formulation:
For each input token, three vectors are derived using learned weight matrices:

• Query (Q): What the token is looking for.
• Key (K): What the token contains.
• Value (V): The actual information carried.

The attention output is computed as:

Step-by-step:

1. Compute similarity scores between the query and all keys via dot product .
2. Scale by to prevent large values that destabilize softmax.
3. Apply softmax to get attention weights (probabilities summing to 1).
4. Multiply weights with Value vectors to get a weighted sum.

Multi-Head Attention:
Multiple attention heads run in parallel to capture diverse relationships, then results are concatenated.

Why It Is Powerful:

• Captures long-range dependencies without recurrence.
• Parallelizable, enabling faster training than RNNs.
• Context-aware: The same word gets different representations based on surrounding words (e.g., "bank" of a river vs. "bank" for money).

This makes it the core of modern NLP models like Transformers, BERT, and GPT.

Natural Language Processing (NLP) is a branch of Artificial Intelligence that enables computers to understand, interpret, generate, and respond to human language in a meaningful way. It combines linguistics, computer science, and machine learning.

Phases of NLP:

• Lexical Analysis (Morphological Analysis): Breaking text into words/tokens and analyzing word structure. Example: Splitting "running" into root "run" + suffix "ing".
• Syntactic Analysis (Parsing): Analyzing grammatical structure to check if a sentence is well-formed. Example: "The cat sat" is valid; "Cat the sat" is not.
• Semantic Analysis: Deriving the meaning of the text. Example: Understanding that "hot dog" refers to a food, not a warm animal.
• Discourse Integration: Interpreting meaning based on surrounding sentences/context. Example: Resolving "He" in "John went home. He was tired."
• Pragmatic Analysis: Understanding intended meaning based on real-world knowledge and context. Example: "Can you pass the salt?" is a request, not a question about ability.

These phases work together to convert raw text into machine-understandable meaning, powering applications like chatbots and translation systems.

Tokenization is the process of breaking down text into smaller units called tokens, which can be words, subwords, or characters. It is a fundamental preprocessing step in NLP.

Types of Tokenization:

• Word Tokenization: Splits text into individual words. Example: "I love AI" → ["I", "love", "AI"]. Limitation: Struggles with out-of-vocabulary words and large vocabularies.
• Character Tokenization: Splits text into individual characters. Example: "AI" → ["A", "I"]. Handles rare words but produces very long sequences.
• Subword Tokenization: Breaks words into meaningful subunits. Used by modern models. Example: "unhappiness" → ["un", "happi", "ness"]. Common algorithms include Byte Pair Encoding (BPE), WordPiece (used in BERT), and SentencePiece.
• Sentence Tokenization: Splits text into sentences. Example: "Hello. How are you?" → ["Hello.", "How are you?"].

Importance:

• Converts unstructured text into a structured format for models.
• Subword tokenization balances vocabulary size and handles rare/unknown words effectively.
• Directly affects model performance and efficiency.

Tokenization is the first step before converting tokens into numerical embeddings.

Word Embeddings are dense, low-dimensional vector representations of words that capture their semantic meaning and relationships. Words with similar meanings have similar vectors.

One-Hot Encoding (Traditional):

• Each word is represented as a sparse vector with a single 1 and rest 0s.
• Example: In a vocabulary of 10,000 words, each word is a 10,000-dimensional vector.
• Drawbacks:
  • Very high-dimensional and sparse.
  • No notion of similarity; "king" and "queen" are equally distant as "king" and "apple".
  • Memory inefficient.

Dense Embeddings (Word2Vec):

• Represents words as dense, low-dimensional vectors (e.g., 100–300 dimensions).
• Learned from context using models like:
  • CBOW (Continuous Bag of Words): Predicts a word from its context.
  • Skip-gram: Predicts context words from a target word.
• Captures semantic relationships:
  

Comparison Summary:

Other popular embeddings include GloVe and contextual embeddings from BERT.

BERT and GPT are both Transformer-based language models but differ significantly in design and purpose.

BERT (Bidirectional Encoder Representations from Transformers):

• Architecture: Uses only the encoder stack of the Transformer.
• Directionality: Bidirectional — reads context from both left and right simultaneously.
• Training Objective:
  • Masked Language Modeling (MLM): Predicts randomly masked words.
  • Next Sentence Prediction (NSP): Predicts if two sentences follow each other.
• Use Cases: Understanding-based tasks like classification, sentiment analysis, question answering, named entity recognition (NER).

GPT (Generative Pre-trained Transformer):

• Architecture: Uses only the decoder stack of the Transformer.
• Directionality: Unidirectional (left-to-right) / autoregressive.
• Training Objective: Causal Language Modeling — predicts the next word given previous words.
• Use Cases: Generation-based tasks like text generation, chatbots, summarization, code generation.

Comparison Table:

In essence, BERT excels at understanding language, while GPT excels at generating language.

A chatbot is a software application that simulates human conversation through text or voice, using NLP and AI techniques.

Types of Chatbots:

• Rule-Based (Menu/Button-Based): Follow predefined rules and decision trees. Simple but rigid. Example: FAQ bots.
• Retrieval-Based: Select the best response from a predefined set using intent matching.
• Generative (AI-Based): Generate responses dynamically using deep learning (e.g., GPT-based bots). More flexible and human-like.

Key Components:

• Natural Language Understanding (NLU): Interprets user input by identifying:
  • Intent: The user's goal (e.g., "book a flight").
  • Entities: Key information (e.g., date, location).
• Dialogue Management: Tracks conversation context and decides the next action.
• Natural Language Generation (NLG): Produces human-like responses.
• Backend/Integration: Connects to databases, APIs, and business logic.

Building Process:

1. Define purpose and scope of the chatbot.
2. Collect and prepare training data (intents, entities, sample utterances).
3. Choose a platform/framework (e.g., Rasa, Dialogflow, or custom LLM).
4. Train the NLU model to recognize intents and entities.
5. Design conversation flows and responses.
6. Integrate with channels (web, WhatsApp, etc.).
7. Test, deploy, and continuously improve using user feedback.

Modern chatbots increasingly use Large Language Models for natural, context-aware conversations.

Sentiment Analysis (also called opinion mining) is an NLP task that determines the emotional tone or polarity (positive, negative, or neutral) expressed in a piece of text.

Approaches:

• Lexicon-Based (Rule-Based): Uses dictionaries of words with predefined sentiment scores. Example: "good" = +1, "terrible" = -2. Simple but context-limited.
• Machine Learning-Based: Trains classifiers (e.g., Naive Bayes, SVM) on labeled data using features like bag-of-words or TF-IDF.
• Deep Learning-Based: Uses RNNs, LSTMs, or Transformer models (BERT) to capture context and achieve high accuracy.

Levels of Analysis:

• Document level: Overall sentiment of an entire document.
• Sentence level: Sentiment of individual sentences.
• Aspect level: Sentiment toward specific aspects (e.g., "The food was great but service was slow").

Applications:

• Brand monitoring and social media analysis.
• Product review analysis for e-commerce.
• Customer feedback and market research.
• Financial market prediction from news sentiment.

Challenges:

• Sarcasm and irony: "Great, another delay!" is negative despite "great".
• Context and domain dependence.
• Negation handling: "not good".
• Ambiguity and mixed sentiments in a single text.

Modern transformer models handle these challenges better due to contextual understanding.

CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) are two major deep learning architectures designed for different data types.

CNN:

• Designed for: Spatial/grid data like images.
• Working: Uses convolutional filters to detect local spatial features.
• Strengths: Excellent at capturing spatial hierarchies; parameter sharing; parallelizable.
• Weaknesses: Not naturally suited for sequential/temporal dependencies.

RNN:

• Designed for: Sequential/temporal data like text, speech, time-series.
• Working: Maintains a hidden state that carries information across time steps.
• Strengths: Captures temporal dependencies and handles variable-length sequences.
• Weaknesses: Suffers from vanishing/exploding gradients; slow due to sequential processing (not parallelizable).

Comparison Table:

Application Areas:

• CNN: Image classification, object detection, facial recognition, medical imaging.
• RNN: Language modeling, speech recognition, machine translation, stock prediction.

Interestingly, CNNs are sometimes used in NLP for text classification, and both have largely been complemented by Transformers for sequence tasks.

Machine Translation (MT) and Text Summarization are two important NLP applications that transform text.

1. Machine Translation:
MT automatically translates text from one language to another.

Techniques:

• Rule-Based MT (RBMT): Uses linguistic rules and dictionaries. Rigid and labor-intensive.
• Statistical MT (SMT): Uses probabilistic models learned from bilingual corpora.
• Neural MT (NMT): Uses deep learning (seq2seq with attention, Transformers). Produces fluent, context-aware translations. Example: Google Translate.

2. Text Summarization:
Condenses a long text into a shorter version while retaining key information.

Two Main Types:

• Extractive Summarization: Selects and combines important sentences directly from the source text. Uses ranking algorithms (e.g., TextRank). Simpler and factually safe.
• Abstractive Summarization: Generates new sentences that paraphrase the content, similar to how humans summarize. Uses seq2seq models and Transformers (e.g., BART, T5, GPT). More natural but can introduce errors.

Common Foundation:
Both tasks are often modeled as sequence-to-sequence (seq2seq) problems, greatly improved by attention mechanisms and Transformer architectures.

Challenges:

• MT: handling idioms, grammar, low-resource languages.
• Summarization: maintaining coherence, avoiding factual hallucinations.

Activation Functions introduce non-linearity into neural networks, enabling them to learn complex patterns. Without them, a network would behave like a simple linear model regardless of depth.

Common Activation Functions:

1. Sigmoid:

• Output range: (0, 1).
• Used in binary classification output layers.
• Drawbacks: Vanishing gradients for large |x|; not zero-centered.

2. Tanh (Hyperbolic Tangent):

• Output range: (-1, 1).
• Zero-centered, generally better than sigmoid.
• Still suffers from vanishing gradients.

3. ReLU (Rectified Linear Unit):

• Output range: [0, ∞).
• Advantages: Computationally efficient; mitigates vanishing gradient; fast convergence.
• Drawback: "Dying ReLU" — neurons can get stuck at 0.

Variants: Leaky ReLU, ELU, and Softmax (for multi-class output).

Summary Table:

Choosing the right activation function significantly affects training speed and performance.

The vanishing and exploding gradient problems occur during the training of deep neural networks using backpropagation, affecting the ability to learn effectively.

Vanishing Gradient Problem:

• Occurs when gradients become extremely small as they propagate backward through many layers.
• Caused by repeated multiplication of small derivatives (e.g., from sigmoid/tanh, whose derivatives are < 1).
• Effect: Early layers learn very slowly or stop learning; long-term dependencies are lost (especially in RNNs).

Exploding Gradient Problem:

• Occurs when gradients become excessively large, causing unstable updates.
• Caused by repeated multiplication of large values.
• Effect: Weights oscillate or diverge; loss becomes NaN.

Mathematical Insight:
During backpropagation, gradients are products of many terms:

If terms are < 1 → vanishing; if > 1 → exploding.

Solutions:

• ReLU activation: Avoids saturating gradients.
• Proper weight initialization: Xavier/He initialization.
• Batch Normalization: Normalizes layer inputs.
• Gradient Clipping: Caps gradients to a threshold (fixes exploding).
• Residual/Skip connections (ResNet): Allow gradients to flow directly.
• LSTM/GRU: Use gating to preserve gradients in sequential models.

These techniques enable stable training of very deep networks.

A Language Model (LM) is a probabilistic model that assigns probabilities to sequences of words, predicting the likelihood of a word given its context. Formally, it models:

1. Statistical Language Models (N-gram):

• Estimate probabilities based on word frequency counts from a corpus.
• Use the Markov assumption — a word depends only on the previous words.
• Example (bigram): .
• Advantages: Simple, fast, interpretable.
• Drawbacks:
  • Data sparsity: Many word combinations never appear.
  • Limited context: Only captures short dependencies.
  • Requires smoothing techniques (e.g., Laplace smoothing).

2. Neural Language Models:

• Use neural networks (RNNs, LSTMs, Transformers) to learn distributed representations.
• Represent words as dense embeddings, capturing semantic relationships.
• Advantages:
  • Capture long-range dependencies and context.
  • Generalize better to unseen sequences.
  • Power modern models like BERT and GPT.
• Drawbacks: Computationally expensive; require large data and compute.

Comparison:

Neural LMs, especially Transformer-based, now dominate modern NLP.

Digital Assistants (or virtual assistants) are AI-powered systems that interact with users through voice or text to perform tasks, answer questions, and control devices. Examples include Amazon Alexa, Apple Siri, and Google Assistant.

Role and Capabilities:

• Answering questions and providing information.
• Setting reminders, alarms, and calendar events.
• Controlling smart home devices.
• Playing media and performing online tasks.
• Hands-free, conversational interaction.

Processing Pipeline (Voice Input → Response):

1. Wake Word Detection: Listens for a trigger word (e.g., "Hey Siri").
2. Automatic Speech Recognition (ASR): Converts spoken audio into text.
3. Natural Language Understanding (NLU): Extracts intent (user's goal) and entities (key details) from the text.
4. Dialogue Management: Maintains context and decides the appropriate action, possibly querying databases or APIs.
5. Task Execution/Fulfillment: Performs the requested action (e.g., fetch weather, play music).
6. Natural Language Generation (NLG): Formulates a human-like text response.
7. Text-to-Speech (TTS): Converts the text response back into natural-sounding speech.

Underlying Technologies:

• Deep learning for ASR and TTS.
• Transformer-based NLP models for understanding and generation.
• Cloud computing for processing and integration.

Modern assistants increasingly leverage Large Language Models for more natural, context-aware conversations.

The Attention mechanism allows a model to dynamically focus on the most relevant parts of the input sequence when producing each part of the output, rather than relying on a single fixed representation.

Problem with Traditional Seq2Seq Models:

• Classic encoder-decoder (RNN/LSTM) models compress the entire input into a single fixed-length context vector.
• Bottleneck: For long sequences, this vector cannot retain all information, causing performance to drop.
• Difficulty capturing long-range dependencies.

How Attention Solves It:

• Instead of one context vector, attention computes a weighted combination of all encoder hidden states for each decoder step.
• Attention weights determine how much focus to place on each input word:
  
  where are attention weights (summing to 1) and are encoder states.
• Weights are computed via a scoring function followed by softmax.

Why It Was a Breakthrough:

• Handles long sequences effectively — no information bottleneck.
• Interpretability: Attention weights show which input words influenced the output (e.g., in translation).
• Better long-range dependency modeling.
• Led to the fully attention-based Transformer architecture, revolutionizing NLP.

Types: Additive (Bahdanau) attention and Multiplicative (Luong/dot-product) attention. Self-attention (used in Transformers) applies attention within a single sequence.

Attention is the foundation of modern models like BERT, GPT, and machine translation systems.