Unit4 - Subjective Questions

Definition:
A Perceptron is the simplest type of artificial neural network and a fundamental building block of deep learning. It is a binary classifier that maps input vectors to an output value using weights and a bias.

Mathematical Model:

1. Inputs (): The perceptron takes inputs .
2. Weights (): Each input is multiplied by a corresponding weight , representing the importance of the input.
3. Weighted Sum (): The weighted inputs are summed up along with a bias term ().
   
4. Activation Function: A step function (usually Heaviside or Sign function) is applied to to determine the output ().
   

Diagram Description:
Input nodes feed into a summation node (neuron), which passes the result through an activation function to produce the output.

Architecture of MLP:
An MLP is a feedforward artificial neural network consisting of at least three layers of nodes:

1. Input Layer: Receives the initial data signal.
2. Hidden Layer(s): One or more layers where computational processing and feature extraction occur via weighted sums and non-linear activation functions.
3. Output Layer: Produces the final prediction or classification.

Working Mechanism:

• Forward Propagation: Data flows from input to output. Each neuron calculates , where is a non-linear activation function (like Sigmoid or ReLU).
• Backpropagation: The network calculates the error (Loss) between the predicted output and actual target. This error is propagated backward to update weights using an optimization algorithm (like Gradient Descent) to minimize the loss.

Solving Limitations:
A single Perceptron can only classify linearly separable data (solving the XOR problem is impossible). MLP introduces non-linearity through hidden layers and activation functions, allowing it to learn complex boundaries and solve non-linearly separable problems.

Comparison:

1. Sigmoid Function:

• Formula:
• Range: (0, 1)
• Usage: Often used in the output layer for binary classification. It suffers from the vanishing gradient problem in deep networks.

2. ReLU (Rectified Linear Unit):

• Formula:
• Range: [0, )
• Usage: Most popular for hidden layers. It is computationally efficient and reduces the vanishing gradient problem, though it can suffer from the "dying ReLU" problem (neurons outputting zero forever).

3. Tanh (Hyperbolic Tangent):

• Formula:
• Range: (-1, 1)
• Usage: Zero-centered, making it generally better than Sigmoid for hidden layers as optimization is easier, but still susceptible to vanishing gradients.

CNN Architecture:
CNNs are specialized neural networks used primarily for processing grid-like data, such as images. The architecture typically consists of a stack of layers:

1. Convolutional Layer (The Core):

   • Role: Feature extraction. It uses a set of learnable filters (kernels) that slide (convolve) over the input image.
   • Operation: Performs element-wise multiplication and summation to produce a Feature Map. This allows the network to detect edges, textures, and patterns.
   • Key Concepts: Stride (step size of the filter), Padding (adding borders to preserve dimension).

2. Pooling Layer (Downsampling):

   • Role: Dimensionality reduction. It reduces the spatial size of the representation to decrease the number of parameters and computation in the network.
   • Types:
     • Max Pooling: Takes the maximum value from a specific window (captures most prominent features).
     • Average Pooling: Takes the average value.

3. Fully Connected (FC) Layer:

   • Role: Classification. After feature extraction and downsampling, the data is flattened into a vector and passed through standard dense layers to output the final class probabilities.

CNNs are preferred over MLPs for image tasks due to:

1. Spatial Invariance: CNNs can recognize features (like a cat's ear) regardless of where they appear in the image, thanks to weight sharing in filters. MLPs treat every pixel as a separate input tied to a specific location.
2. Parameter Efficiency: MLPs connect every input pixel to every neuron in the hidden layer, leading to an explosion of parameters for large images (e.g., a image has 1 million inputs). CNNs share weights (filters), drastically reducing the parameter count.
3. Local Connectivity: CNNs capture local spatial dependencies (neighboring pixels usually relate to each other), whereas MLPs ignore the 2D structure of the image by flattening it immediately.

Recurrent Neural Networks (RNNs):
RNNs are designed for sequential data (time series, text, audio) where the order matters. Unlike feedforward networks, RNNs have a "memory" loop.

• The output of a hidden layer at time step depends not only on the input at () but also on the hidden state from the previous time step ().
• Formula:

The Vanishing Gradient Problem:

• Cause: During training (Backpropagation through Time), gradients are calculated by applying the chain rule across many time steps.
• Effect: If the weights are small (less than 1) or the activation function (like Sigmoid/Tanh) has small derivatives, the gradients shrink exponentially as they propagate back to earlier time steps.
• Result: The weights in the earlier layers do not update effectively, causing the RNN to forget long-range dependencies (e.g., forgetting the subject of a sentence appearing at the start of a long paragraph).

Definition:
Natural Language Processing (NLP) is a branch of Artificial Intelligence that gives computers the ability to understand, interpret, and generate human language (text or speech) in a way that is meaningful.

Major Phases of NLP:

1. Lexical Analysis: The study of the structure of words (morphology) and breaking text into paragraphs, sentences, and words (tokenization).
2. Syntactic Analysis (Parsing): Analyzing the grammatical structure of the sentence to check for conformity to the rules of grammar (e.g., creating parse trees).
3. Semantic Analysis: Determining the meaning of the words and sentences. It focuses on the literal meaning irrespective of context.
4. Discourse Integration: Analyzing the meaning of a sentence based on the sentences preceding it (contextual flow).
5. Pragmatic Analysis: Deriving knowledge from external scenarios and understanding the intent or effect of the language in a real-world situation (reading between the lines).

Tokenization:
Tokenization is the process of breaking down a stream of text into smaller units called tokens. These tokens can be words, characters, or subwords. It is the first step in most NLP pipelines.

Difference:

• Word Tokenization:

  • Splits text based on whitespace or punctuation.
  • Example: "I'm running" ["I", "'m", "running"]
  • Limitation: Large vocabulary size; struggles with out-of-vocabulary (OOV) words.

• Subword Tokenization (e.g., BPE, WordPiece):

  • Splits words into frequent sub-units.
  • Example: "running" ["run", "##ning"]
  • Advantage: Handles rare words by breaking them into known parts; balances vocabulary size and semantic representation. Used in modern models like BERT and GPT.

Word Embeddings:
Word embeddings are dense vector representations of words where words with similar meanings are located close to each other in the vector space. Examples include Word2Vec, GloVe, and FastText.

Superiority over One-Hot Encoding:

1. Dimensionality:
   • One-Hot: Produces high-dimensional sparse vectors (size equals vocabulary size, e.g., 50,000 dimensions).
   • Embeddings: Produces low-dimensional dense vectors (e.g., 100-300 dimensions).
2. Semantic Meaning:
   • One-Hot: Vectors are orthogonal; there is no mathematical relationship between "King" and "Queen".
   • Embeddings: Capture semantic relationships. Operations like are possible.
3. Generalization: Embeddings allow models to generalize better to unseen words if they are semantically similar to known words.

Transformer Architecture:
Introduced in the paper "Attention is All You Need" (2017), the Transformer relies entirely on attention mechanisms, discarding recurrence and convolutions.

1. Encoder Block:

   • Purpose: Processes the input sequence to create a deep understanding of the context.
   • Components: A stack of identical layers. Each layer has two sub-layers:
     • Multi-Head Self-Attention: allows the model to associate each word with every other word in the input.
     • Feed-Forward Network: processes the attention outputs.
   • Includes residual connections and layer normalization.

2. Decoder Block:

   • Purpose: Generates the output sequence one element at a time.
   • Components: Similar to the encoder but with an extra sub-layer:
     • Masked Self-Attention: Ensures the prediction for position can only depend on known outputs at positions less than (prevents looking ahead).
     • Encoder-Decoder Attention: Helps the decoder focus on relevant parts of the input sequence (from the Encoder stack).

3. Positional Encoding: Since Transformers process data in parallel, positional encodings are added to embeddings to retain the order of words.

Attention Mechanism:
The Attention mechanism mimics human cognitive attention. Instead of processing the whole input as a fixed context vector (as in older RNNs), it allows the model to assign different "weights" or importance to different parts of the input sequence when generating a specific part of the output.

Self-Attention:
Self-attention (or intra-attention) relates different positions of a single sequence to compute a representation of the sequence.

• Example: In the sentence "The animal didn't cross the street because it was too tired", self-attention allows the model to associate "it" strongly with "animal" rather than "street".
• Mechanism: It computes three vectors for each word: Query (Q), Key (K), and Value (V). The attention score is derived by comparing the Query of one word with the Keys of others.

BERT:
BERT is a transformer-based machine learning technique for NLP pre-training. It uses only the Encoder stack of the Transformer architecture. Its key innovation is bidirectionality, meaning it looks at the text context from both left-to-right and right-to-left simultaneously.

Training Objectives:
BERT is pre-trained on two unsupervised tasks:

1. Masked Language Modeling (MLM): Randomly masks 15% of the words in the input (e.g., "The [MASK] sat on the mat") and asks the model to predict the masked word based on context.
2. Next Sentence Prediction (NSP): The model receives pairs of sentences and predicts if the second sentence logically follows the first. This helps in understanding relationships between sentences (crucial for QA and summarization).

Comparison:

Building a modern AI chatbot involves the following workflow:

1. Define Scope: Determine the purpose (e.g., Customer Service, Booking) and domain.
2. Data Collection: Gather conversation logs, FAQs, and intent examples.
3. Preprocessing: Clean text, tokenize, and remove stop words.
4. NLU (Natural Language Understanding) Component:
   • Intent Recognition: Classify what the user wants (e.g., "BookFlight", "CheckBalance").
   • Entity Extraction (NER): Extract specific variables (e.g., "New York", "tomorrow").
5. Dialogue Management: Maintain the state of the conversation and determine the next action based on the intent and context.
6. Response Generation:
   • Retrieval-based: Select a pre-written response.
   • Generative: Use LLMs (like GPT) to generate a dynamic response.
7. Testing & Deployment: Test for edge cases and deploy on platforms (Web, WhatsApp, Slack).

Sentiment Analysis:
Also known as opinion mining, it is the process of determining the emotional tone behind a body of text. It classifies text as Positive, Negative, or Neutral (and sometimes intensity).

Implementation:

1. Preprocessing: Tokenization, stop-word removal, lemmatization.
2. Feature Extraction: Convert text to numbers using Bag of Words, TF-IDF, or Word Embeddings.
3. Model Classification:
   • Traditional: Use Naive Bayes, SVM, or Logistic Regression on extracted features.
   • Deep Learning: Feed embeddings into RNNs (LSTMs) or Transformers (BERT) to capture context and sarcasm.
4. Output: The model outputs a probability score for each sentiment class.

1. Extractive Summarization:

• Method: Selects important sentences or phrases directly from the original text and stitches them together.
• Analogy: Like highlighting key sentences in a book.
• Technique: Uses statistical scoring (e.g., TF-IDF, Graph-based ranking) to identify high-importance sentences.
• Pros/Cons: Grammatically safer but may lack coherence or flow.

2. Abstractive Summarization:

• Method: Generates new sentences that capture the core meaning of the source text using new words and phrasing.
• Analogy: Like reading a book and writing a summary in your own words.
• Technique: Requires advanced Deep Learning models (Seq2Seq RNNs, Transformers like T5 or BART) to understand semantics and generate language.
• Pros/Cons: More human-like and concise but computationally expensive and prone to "hallucinations" (inventing facts).

NLP is difficult because human language is ambiguous and complex. Common challenges include:

1. Ambiguity:
   • Lexical: Words with multiple meanings (e.g., "Bank" - river vs. finance).
   • Syntactic: Sentences with multiple parse structures (e.g., "I saw the man with the telescope").
2. Sarcasm and Irony: Machines struggle to detect when the literal meaning is opposite to the intended meaning.
3. Slang and Idioms: Informal language and expressions (e.g., "It's raining cats and dogs") are hard to translate literally.
4. Contextual Coference: Understanding "He" or "It" refers to which noun mentioned sentences ago.
5. Domain Specificity: A model trained on news articles may fail to understand medical or legal documents.

Loss Function (Cost Function):

• It is a mathematical function that quantifies the error between the model's prediction () and the actual target ().
• Goal: The objective of training is to minimize this value.
• Examples:
  • Mean Squared Error (MSE): Used for regression.
  • Cross-Entropy Loss: Used for classification.

Optimizer:

• It is an algorithm used to update the weights of the neural network to minimize the Loss Function.
• It decides how much to change the weights and in which direction based on the gradients calculated via backpropagation.
• Examples:
  • Gradient Descent: Updates weights by moving in the opposite direction of the gradient.
  • Adam (Adaptive Moment Estimation): A popular optimizer that adjusts learning rates adaptively for each parameter, combining advantages of other extensions like RMSprop and Momentum.