Unit 2: Word Embeddings and Vector Representations

I. Orientation — Language as Geometry

Word representation converts discrete linguistic symbols into numerical vectors that neural networks can process. The governing principle is the distributional hypothesis: words occurring in similar contexts tend to have related meanings. A representation therefore derives semantic information from patterns of word usage rather than from dictionary definitions alone.

A. Defining Foundations

The unit rests on a small set of assumptions about words, contexts, and geometric structure.

• Tokens: A token is an observed unit, such as learning, while a type is its vocabulary entry; preprocessing determines whether Learn and learn share one type.
• Vocabulary: A vocabulary (V) contains the retained word types, and its size (|V|) determines the dimensions of one-hot and model output layers.
• Context: Context commonly means words within a fixed window; with radius (m=2), the context of fox in the quick brown fox jumps is quick, brown, and jumps.
• Distributional evidence: Words such as doctor and nurse frequently occur near terms including hospital, patient, and treatment, producing related representations.
• Geometric interpretation: Distance, direction, and neighborhood in vector space are treated as indicators of linguistic relationships.
• Learned parameters: Embedding coordinates are optimized from data; an individual dimension usually has no predefined meaning.
• Static representation: Word2Vec and GloVe assign one vector per vocabulary item, so bank has the same vector in river bank and central bank.

II. Vector Space Models — Representing Linguistic Distribution

A. Vector space models

Vector space models represent linguistic units as points in a multidimensional space so that mathematical operations can compare them.

• One-hot vectors: Each word receives a vector of length (|V|) containing one 1 and otherwise zeros. For (V={\text{cat},\text{dog},\text{fish}}), dog is ([0,1,0]).
• Limitation of one-hot encoding: Any two distinct one-hot vectors are orthogonal, so their dot product is zero regardless of semantic relationship.
• Count-based vectors: A word can instead be represented by its co-occurrence counts with context terms; rows of a word-context matrix then encode distributional profiles.
• Weighting: Raw counts may be replaced by TF-IDF or positive pointwise mutual information to reduce the influence of universally frequent words.
• Similarity measure: Cosine similarity compares vector direction while reducing sensitivity to magnitude.

cos(u, v) = (u · v) / (||u||₂ ||v||₂)

Here (u) and (v) are word vectors, (u\cdot v) is their dot product, and (|u|_2) is Euclidean length.

• Interpretation: Cosine values near (1) indicate similar directions, values near (0) indicate weak relation, and negative values indicate opposing directions.
• Sparsity problem: Count and one-hot vectors may have thousands of dimensions but relatively few nonzero entries, increasing memory requirements and weakening generalization.

III. Dense Representations — Learned Distributed Features

A. Dense word embeddings

Dense word embeddings map words to short, real-valued vectors whose coordinates are learned jointly from linguistic evidence.

• Embedding matrix: A matrix (E\in\mathbb{R}^{|V|\times d}) stores one (d)-dimensional row per word, where typically (d\ll|V|).
• Lookup operation: Multiplying one-hot vector (x_w) by (E) selects the embedding of word (w).

e_w = x_wᵀE

Here (x_w\in\mathbb{R}^{|V|}) is the one-hot vector and (e_w\in\mathbb{R}^{d}) is the resulting dense embedding.

• Distributed representation: Meaning is encoded across many coordinates, and each coordinate contributes to representations of many words.
• Training modes: Pretrained embeddings can be frozen, updated during downstream training, or initialized randomly and learned entirely for a task.
• Benefits: Dense vectors reduce dimensionality, allow graded similarity, and transfer statistical knowledge to words occurring in related contexts.
• Limitations: Static embeddings merge multiple senses, inherit social biases from training text, and provide unreliable vectors for rare or unseen words.

IV. Word2Vec — Predictive Learning from Local Context

A. Word2Vec

Word2Vec is a family of shallow neural models introduced by Mikolov and collaborators in 2013 to learn embeddings through word-context prediction.

• Training data: A sliding window converts a corpus into target-context observations; window radius (m) controls how local the evidence is.
• Parameters: Word2Vec learns an input matrix (W) and an output-context matrix (W'); rows of (W) are commonly retained as embeddings.
• Objective: Training increases the probability of observed target-context pairs and decreases that of unobserved or sampled negative pairs.
• Softmax probability:

P(o | c) = exp(v'ₒ · v_c) / Σᵢ exp(v'ᵢ · v_c)

Here (c) is the input word, (o) is the predicted word, (v_c) is its input vector, (v'_i) is output vector (i), and the sum covers the vocabulary.

• Efficiency techniques: Hierarchical softmax replaces full-vocabulary prediction with a tree path, while negative sampling trains against a small set of noise words.
• Subsampling: Randomly discarding very frequent words such as the reduces computation and prevents them from dominating context evidence.

B. CBOW

Continuous Bag of Words predicts a target word from the embeddings of its surrounding context words.

• Input construction: Context vectors are summed or averaged, discarding their order; cats ___ milk provides cats and milk as predictors.
• Hidden representation:

h = (1/C) Σⱼ v_cⱼ

Here (C) is the number of context words, (v_{c_j}) is context word (j)'s embedding, and (h) is their mean.

• Prediction: The hidden vector (h) is passed to the output layer to estimate the missing target, such as drink.
• Bag-of-words assumption: dog bites man and man bites dog can supply the same context set, so local word order is not represented.
• Strength: Combining several context words creates a stable signal and makes CBOW relatively fast, especially for frequent vocabulary.
• Weakness: Averaging can blur informative context distinctions and usually gives weaker representations for rare words than Skip-Gram.

C. Skip-Gram models

Skip-Gram reverses CBOW by using one center word to predict the words appearing around it.

• Training pairs: With center fox and radius (2), separate pairs may include (fox, quick), (fox, brown), and (fox, jumps).
• Objective:

L = -Σₜ Σ₋ₘ≤ⱼ≤ₘ, j≠0 log P(wₜ₊ⱼ | wₜ)

Here (L) is loss, (t) indexes center positions, (m) is window radius, (wt) is the center word, and (w{t+j}) is a context word.

• Negative sampling: For an observed pair ((c,o)), training raises (\sigma(v'_o\cdot v_c)) and lowers it for sampled noise words, where (\sigma(z)=1/(1+e^{-z})).
• Strength: Each center word generates multiple updates, allowing Skip-Gram to learn useful representations for less frequent words.
• Cost: Producing several target pairs per center word generally makes training slower than CBOW.
• Comparison: CBOW performs context-to-word prediction and favors speed; Skip-Gram performs word-to-context prediction and favors detailed rare-word representations.

V. GloVe — Learning from Global Co-occurrence Statistics

A. GloVe embeddings

Global Vectors for Word Representation, introduced by Pennington, Socher, and Manning in 2014, learns vectors from aggregated corpus-wide co-occurrence counts.

• Co-occurrence matrix: (X_{ij}) records how often word (j) appears in the context of word (i), possibly with distance-based weighting.
• Core objective:

J = Σᵢ,ⱼ f(Xᵢⱼ)(wᵢ · w̃ⱼ + bᵢ + b̃ⱼ - log Xᵢⱼ)²

Here (J) is loss, (w_i) and (\tilde w_j) are word and context vectors, (b_i) and (\tilde b_j) are biases, and (f) limits the influence of extreme counts.

• Statistical principle: Dot products are trained to approximate logarithmic co-occurrence counts, linking vector geometry to global corpus statistics.
• Ratio information: Relative probabilities can distinguish concepts; ice co-occurs more characteristically with solid, whereas steam favors gas.
• Final representation: Implementations commonly combine (w_i) and (\tilde w_i), for example by addition, because both encode useful information.
• Comparison with Word2Vec: GloVe factorizes information from an explicit global matrix, while Word2Vec learns through sampled local prediction events.
• Constraints: Constructing the co-occurrence matrix can consume substantial memory, and corpus bias or polysemy remains embedded in the result.

VI. Semantic Geometry — Meaning and Relational Structure

A. Capturing semantic similarity

Embedding similarity emerges when words receive comparable updates because they occur in overlapping linguistic environments.

• Semantic neighbors: The nearest vectors to car may include vehicle, truck, and automobile because these words share contexts involving roads, driving, and transport.
• Cosine ranking: A query word is compared with every candidate using cosine similarity, and candidates are sorted from highest to lowest score.
• Similarity versus association: coffee and cup may be close because they co-occur, although they are associated rather than synonymous.
• Evaluation: Intrinsic evaluation correlates model similarities with human word-pair judgments; extrinsic evaluation measures effects on tasks such as classification or named-entity recognition.
• Dependence on data: Domain-specific corpora alter neighborhoods; in medical text, positive may lie near diagnoses rather than generally favorable terms.

B. Analogy relationships

Analogy methods test whether relational differences between word vectors are approximately consistent directions in embedding space.

• Vector-offset method:

answer = argmaxₓ cos(vₓ, v_b - v_a + v_c)

Here (a:b::c:x) is the analogy, (v) denotes an embedding, and (x) excludes the supplied words.

• Worked relation: For man : king :: woman : ?, the query (v{\text{king}}-v{\text{man}}+v_{\text{woman}}) may retrieve queen.
• Encoded patterns: Offsets can reflect semantic relations such as country-capital and syntactic relations such as adjective-comparative.
• Approximation: Relations are not perfectly linear; performance depends on corpus coverage, dimension, training objective, and nearest-neighbor method.
• Bias exposure: Analogy results can reveal stereotypes learned from text, so geometric regularity does not imply factual or ethical validity.

VII. Embedding Visualization — Projecting High-Dimensional Structure

A. Visualizing embedding spaces using PCA or t-SNE

Dimensionality-reduction methods project embeddings into two or three dimensions so that selected neighborhood patterns can be inspected visually.

1. PCA: Principal Component Analysis linearly projects centered vectors onto orthogonal directions of maximum variance.
   • Computation: The first principal component captures the greatest variance, and the second captures the greatest remaining variance subject to orthogonality.
   • Interpretation: PCA preserves broad global structure reasonably well and gives deterministic results for fixed data, but a two-dimensional view may discard substantial information.
2. t-SNE: t-distributed Stochastic Neighbor Embedding constructs a nonlinear map designed to preserve local neighborhood probabilities.
   • Behavior: Nearby words often form visible clusters, but distances between clusters, cluster sizes, and empty regions should not be interpreted literally.
   • Parameters: Perplexity influences neighborhood scale, while initialization and random seed can change the displayed arrangement.

• Procedure: Select a meaningful vocabulary subset, normalize or center vectors as appropriate, fit the reducer, and plot each projected point with its word label.
• Reading a plot: A cluster containing Paris, Rome, and Madrid suggests shared distributional behavior, not proof that the model stores a formal category called “capital.”
• Responsible comparison: Use identical words and settings when comparing models, report the reduction parameters, and confirm apparent patterns with original-space cosine similarities.