Unit 4: Introduction to deep neural networks; Modern NLP

I. Introduction to Neural Networks

Artificial neural networks (origins in McCulloch–Pitts, 1943; backpropagation popularised 1986) are computational systems of interconnected units that learn a mapping from inputs to outputs by adjusting numeric weights. Everything below rests on the same skeleton and vocabulary.

• Neuron (unit): computes a weighted sum plus bias, then applies a non-linearity: y = f(Σ wᵢxᵢ + b), where wᵢ are weights, xᵢ inputs, b bias, f the activation.
• Activation functions: ReLU(z)=max(0,z), sigmoid(z)=1/(1+e⁻ᶻ), tanh, softmax for probability vectors. Non-linearity lets networks approximate arbitrary functions.
• Layers: input, hidden, output; "deep" means two or more hidden layers.
• Learning by gradient descent: minimise a loss L (e.g. cross-entropy) by updating w ← w − η·∂L/∂w, with learning rate η.
• Backpropagation: the chain rule propagates the loss gradient backward through layers to obtain every ∂L/∂w.

II. Feedforward Networks — Perceptron and MLP

A. Perceptron

The single-layer perceptron is the simplest trainable classifier, drawing a linear decision boundary.

• Model: ŷ = step(Σ wᵢxᵢ + b), output 1 if the sum ≥ 0 else 0.
• Update rule: wᵢ ← wᵢ + η(y − ŷ)xᵢ, applied per misclassified example.
• Limitation: solves only linearly separable problems; famously cannot represent XOR (Minsky & Papert, 1969).

B. MLP

The multilayer perceptron stacks fully connected layers with non-linear activations to overcome the XOR limitation.

• Structure: input → one or more hidden layers → output; every neuron connects to all neurons in the next layer ("dense").
• Universal approximation: one sufficiently wide hidden layer can approximate any continuous function.
• Training: forward pass computes outputs, backpropagation computes gradients, gradient descent updates weights over many epochs.
• Worked idea (XOR): two hidden ReLU units let the MLP carve two half-planes whose combination reproduces XOR, impossible for a single perceptron.

III. Convolutional Neural Networks — grids and images

A. CNN

CNNs exploit spatial locality, sharing weights so nearby pixels are processed by the same small filter.

• Convolution layer: slides a kernel (e.g. 3×3) across the image, computing Σ (kernel · patch) to produce a feature map; the same weights are reused everywhere (parameter sharing).
• Pooling: downsamples, e.g. max-pooling takes the maximum in each 2×2 window, giving translation invariance and fewer parameters.
• Hierarchy: early filters detect edges, deeper filters detect shapes then objects.
• Applications: image classification, object detection, medical imaging; landmark model AlexNet (2012).

IV. Recurrent Neural Networks — sequences

A. RNN

RNNs process ordered data by carrying a hidden state that summarises past inputs.

• Recurrence: hₜ = f(Wₓxₜ + Wₕhₜ₋₁ + b), where hₜ is the state at step t; the same weights apply at every step.
• Vanishing/exploding gradients: long sequences shrink or blow up gradients, so plain RNNs forget distant context.
• LSTM / GRU: gated cells (input, forget, output gates) regulate what to keep or discard, enabling long-range memory.
• Applications: speech recognition, time-series forecasting, early machine translation.

V. Transformer Architecture and Applications

The Transformer (Vaswani et al., "Attention Is All You Need", 2017) replaces recurrence with attention, processing all tokens in parallel.

A. Transformer Architecture

Its core is stacked self-attention and feedforward blocks operating on the whole sequence at once.

• Encoder–decoder: encoder builds contextual representations; decoder generates output autoregressively.
• Self-attention: each token attends to every other token, capturing long-range dependencies directly.
• Positional encoding: since there is no recurrence, sinusoidal or learned vectors inject word order.
• Multi-head attention: several attention computations run in parallel to capture different relations.
• Residual connections + layer normalisation: stabilise training of deep stacks.

B. Applications

The parallelism and scalability of Transformers made them the backbone of modern AI.

• NLP: translation, summarisation, question answering, code generation.
• Beyond text: Vision Transformers for images, speech models, protein folding (AlphaFold).
• Foundation models: pretrained once on massive corpora, then fine-tuned for many tasks.

VI. Introduction to NLP

Natural Language Processing enables computers to read, interpret and generate human language, bridging unstructured text and structured computation.

• Goal: map ambiguous, context-dependent language to representations machines can act on.
• Core challenges: ambiguity ("bank" = river/finance), sarcasm, coreference, morphology, and world knowledge.
• Paradigm shift: rule-based systems → statistical models → neural embeddings → large pretrained Transformers.

A. NLP phases

Classical NLP analyses text through successive linguistic layers.

• Lexical / morphological: break text into tokens and analyse word structure (stems, affixes).
• Syntactic: parse grammatical structure (part-of-speech tagging, parse trees).
• Semantic: assign meaning to words and sentences (word sense disambiguation).
• Discourse: link sentences, resolve references across a passage.
• Pragmatic: interpret intended meaning in context (a question functioning as a request).

VII. NLP Building Blocks — Tokenization, Embeddings, Attention

A. Tokenization

Tokenization splits raw text into the units a model processes.

• Word-level: splits on whitespace/punctuation; suffers from huge vocabularies and unknown words.
• Subword (BPE, WordPiece): merges frequent character pairs so "playing" → "play" + "##ing"; handles rare words with a compact vocabulary.
• Special tokens: [CLS], [SEP], [MASK], <eos> mark structure and tasks.

B. Embeddings

Embeddings map discrete tokens to dense vectors that encode meaning geometrically.

• Idea: similar words sit close in a high-dimensional space; "distributional hypothesis" — words in similar contexts have similar meaning.
• Static embeddings: Word2Vec, GloVe give one fixed vector per word; captures analogies like king − man + woman ≈ queen.
• Contextual embeddings: BERT/GPT produce a different vector per occurrence, so "bank" differs by sentence.

C. Attention

Attention lets a model weight the relevance of every token when representing another.

• Query–Key–Value: Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V, where Q,K,V are learned projections and dₖ the key dimension (scaling prevents oversized dot products).
• Intuition: the query for one word scores its match with every key; high scores pull in that word's value.
• Self- vs cross-attention: self-attention relates a sequence to itself; cross-attention lets a decoder attend to encoder outputs.

VIII. Language models (BERT, GPT)

Language models assign probabilities to sequences and generate text, pretrained on large corpora then adapted downstream.

A. BERT

BERT (Bidirectional Encoder Representations from Transformers, Google 2018) reads context from both directions for understanding tasks.

• Architecture: Transformer encoder stack only.
• Masked language modelling: randomly hide ~15% of tokens and predict them, forcing bidirectional context.
• Next-sentence prediction: learns relationships between sentence pairs.
• Use: fine-tuned for classification, named-entity recognition, QA — not for free-form generation.

B. GPT

GPT (Generative Pretrained Transformer, OpenAI) is a decoder-only model built to generate text.

• Architecture: Transformer decoder with causal (left-to-right) masking.
• Objective: next-token prediction — maximise P(wₜ | w₁…wₜ₋₁).
• Scaling & prompting: large versions perform few-shot/zero-shot tasks from instructions alone.
• Contrast: BERT is bidirectional and understanding-oriented; GPT is unidirectional and generation-oriented.

IX. Building chatbots and digital assistants

Conversational systems combine NLP components into an interactive pipeline that understands, decides and responds.

• Intent recognition: classify what the user wants ("book_flight").
• Entity extraction (slots): pull parameters like date, destination.
• Dialogue management: track state across turns and choose the next action.
• Response generation: template-based (rule-driven) or generative (LLM-driven).
• Two design styles:
  1. Retrieval-based: selects the best reply from a fixed set — safe, predictable.
  2. Generative: composes new replies with an LLM — fluent but can hallucinate.
• Digital assistants (Alexa, Siri): add speech-to-text, wake-word detection and integration with external APIs and devices.

X. NLP use cases

A. Sentiment analysis

Determines the emotional polarity of text.

• Task: classify as positive, negative or neutral; may extend to aspect-level ("battery bad, screen good").
• Method: fine-tune a model like BERT on labelled reviews; output via softmax.
• Uses: brand monitoring, product-review mining, market feedback.

B. Translation

Converts text from a source language to a target language.

• Neural machine translation: encoder–decoder Transformer maps source to target sequence.
• Advance: attention removed the fixed-length bottleneck of earlier sequence models, sharply improving fluency and long-sentence accuracy.
• Uses: real-time translation, subtitling, cross-lingual search.

C. Summarization

Compresses a document into a shorter form preserving key information.

• Extractive: selects and stitches the most important sentences verbatim.
• Abstractive: generates new sentences paraphrasing content, using generative Transformers.
• Uses: news digests, meeting notes, document triage.