Unit 2: Word Embeddings and Vector Representations - Subjective Questions

A vector space model represents textual objects such as documents, sentences, or words as numerical vectors in a multidimensional space.

• Each dimension usually corresponds to a term, word, or learned feature.
• A document can be represented as a vector of word counts, binary indicators, or weighted values such as TF-IDF.
• Similarity between two vectors is commonly measured using cosine similarity:
  
• A value close to indicates similar directions, while a value close to indicates little similarity.

Traditional vector space models are easy to interpret, but they often produce very high-dimensional and sparse vectors. They also have difficulty recognizing that words with similar meanings, such as "car" and "automobile," are semantically related unless they share exact features.

In a one-hot representation, a vocabulary of size is represented using a vector of length . A word has value at its own index and everywhere else.

Major limitations include:

• High dimensionality: The vector size grows with the vocabulary.
• Sparsity: Almost every component is zero.
• No semantic relationship: The vectors for "king" and "queen" are as distant as the vectors for "king" and "table" under ordinary one-hot distance measures.
• Poor generalization: The representation does not share information between related words.

Dense word embeddings represent words using low-dimensional, real-valued vectors learned from language data. Words occurring in similar contexts receive similar vectors. This allows the representation to encode semantic and syntactic information, reduce memory requirements, and support operations such as similarity search and analogy solving.

Dense word embeddings are compact vectors in which most or all components contain non-zero real-valued numbers. A word is represented as a vector , where is usually much smaller than the vocabulary size.

The distributional hypothesis states that words appearing in similar linguistic contexts tend to have similar meanings. For example, "doctor" and "nurse" may occur near words such as "hospital," "patient," and "treatment."

Embedding algorithms use these contextual patterns to learn vector coordinates. Consequently:

• Semantically related words tend to be close in the embedding space.
• Syntactic patterns can be encoded in vector directions.
• The vectors can be used as input features for classification, translation, question answering, and language modeling.

The learned dimensions generally do not have simple individual interpretations; meaning is distributed across many dimensions.

Word2Vec is a predictive framework that learns word embeddings from local context windows. Its main architectures are Continuous Bag of Words (CBOW) and Skip-Gram.

During training, the model uses word-context pairs and learns an embedding matrix. Given an input word or context, it predicts another word. The model typically contains:

• An input embedding matrix .
• An output embedding matrix .
• A scoring function based on the dot product of input and output vectors.
• A probability objective, often optimized with negative sampling or hierarchical softmax.

Word2Vec is efficient because it learns from local prediction tasks instead of explicitly storing and factorizing a huge vocabulary-by-vocabulary co-occurrence matrix. Negative sampling further reduces the cost by updating only the observed word and a small number of randomly selected negative words for each training example.

The CBOW model predicts a target word from the words surrounding it. Suppose the context is and the target is .

The model performs the following steps:

• It looks up an embedding for every context word.
• It combines the embeddings, commonly by averaging them:
  
• It uses the hidden vector to predict the target word.
• The training objective maximizes:
  

The output probability can be computed with softmax:

CBOW is generally faster and often works well for frequent words because it uses multiple context words to predict one target.

The Skip-Gram model reverses the CBOW prediction task. It uses a center word to predict each surrounding context word. For a center word and context window size , the objective is:

Using softmax, the conditional probability is:

Here, is the input vector of the center word and is the output vector of a context word.

Advantage: Skip-Gram performs well for rare words because each occurrence creates several training pairs.

Limitation: It creates more training examples and is therefore usually slower than CBOW, especially with large context windows.

CBOW is useful when training speed is important and the corpus is large. Skip-Gram is often preferred when high-quality representations for infrequent words are required.

The standard softmax requires calculating scores for every word in the vocabulary, which is expensive when the vocabulary size is large. Negative sampling replaces this multiclass problem with several binary classification problems.

For a positive center-context pair and negative samples , the objective is:

where:

• is the sigmoid function.
• The positive pair is encouraged to have a large dot product.
• Negative pairs are encouraged to have a small or negative dot product.

Only output vectors are updated instead of all output vectors. Therefore, the computational cost changes approximately from dependence on to dependence on the small value , making training much faster.

The context window size specifies how many words before and after a target word are used to create training examples.

• A small window focuses on nearby grammatical relationships. It tends to capture syntactic similarity, such as relationships between words that can appear in similar local grammatical positions.
• A large window captures broader topic and semantic relationships because it includes more surrounding content.
• Increasing the window produces more training pairs and can increase computational cost.
• The best window size depends on the task. For part-of-speech or grammatical tasks, a smaller window may be useful. For topic similarity, document retrieval, or semantic clustering, a larger window may work better.

The window is therefore an important hyperparameter because it determines which notion of similarity the embedding model emphasizes.

GloVe, or Global Vectors for Word Representation, learns embeddings from global word-context co-occurrence statistics. It first constructs a co-occurrence matrix , where records how often word occurs in the context of word .

A common GloVe objective is:

Here, limits the influence of extremely frequent co-occurrences.

Comparison:

• Word2Vec learns by predicting words in local contexts through individual training examples.
• GloVe directly models the logarithm of global co-occurrence counts.
• Word2Vec is naturally online and predictive, while GloVe explicitly uses corpus-wide statistics.
• Both methods produce dense vectors in which related words tend to have similar positions or directions.

Thus, Word2Vec emphasizes predictive context relationships, whereas GloVe combines global count information with a vector-based objective.

The GloVe objective models relationships between word vectors and co-occurrence counts using rather than the raw count .

• Logarithmic scaling: Raw counts can vary by several orders of magnitude. Taking compresses this range and prevents extremely frequent word pairs from dominating the objective.
• Meaningful ratios: Ratios of co-occurrence probabilities often carry semantic information. Logarithms convert ratios into differences, which are easier to represent with vector dot products.
• Weighting function: The function reduces the effect of very frequent pairs and prevents rare pairs from producing unstable updates. A typical function is:
  

These choices make training more balanced and allow embeddings to capture useful statistical relationships without being controlled entirely by common words.

For word vectors and , cosine similarity is defined as:

It measures the cosine of the angle between the two vectors.

• A value close to means the vectors point in similar directions.
• A value close to means they are approximately unrelated in direction.
• A negative value indicates opposing directions, although many embedding spaces primarily use positive similarity values.

Cosine similarity normalizes each vector by its magnitude. This is useful because vector length may reflect frequency, confidence, or other training effects rather than meaning. By focusing on direction, cosine similarity emphasizes the relative pattern of dimensions that represents semantic information.

Analogy relationships can appear as approximately consistent vector offsets. For the analogy "king is to man as queen is to woman," the relationship can be expressed as:

To find the missing word, calculate:

The predicted answer is the vocabulary word whose vector has the greatest cosine similarity with , excluding the three known words.

This method works because training can encode relationships such as gender, tense, plurality, or country-capital distinctions as directions or offsets. However, analogy accuracy is not guaranteed; it depends on corpus quality, embedding method, vocabulary, and possible biases in the training data.

Semantic similarity concerns related meaning or topic. Examples include:

• "car" and "automobile"
• "doctor" and "hospital"
• "cat" and "dog"

These words may be related because they refer to similar concepts or commonly occur in the same topics.

Syntactic similarity concerns grammatical behavior or function. Examples include:

• "run" and "walk" as verbs
• "quickly" and "slowly" as adverbs
• "beautiful" and "powerful" as adjectives

A word embedding may encode both kinds of information. Smaller context windows often emphasize local syntactic patterns, while larger context windows tend to capture broader semantic or topical associations. The distinction is not absolute because meaning and grammatical behavior can overlap in real language.

PCA projects high-dimensional vectors into a lower-dimensional space while preserving as much variance as possible.

Procedure:

• Select a set of word vectors and arrange them in a matrix .
• Center the matrix by subtracting the mean vector.
• Compute the covariance matrix or perform singular value decomposition.
• Select the first two or three principal components.
• Project each word vector onto these components:
  
  where contains the selected principal directions.
• Plot the projected coordinates and label the points with their words.

PCA preserves directions containing the greatest overall variance. It may reveal clusters and broad relationships, but it loses information contained in the discarded dimensions. A two-dimensional plot is therefore an approximation and does not necessarily preserve exact nearest-neighbor relationships.

t-SNE, or t-distributed Stochastic Neighbor Embedding, is a nonlinear dimensionality-reduction method designed to preserve local neighborhoods.

It works by:

• Converting distances between high-dimensional points into probabilities representing neighborhood similarity.
• Creating low-dimensional points whose neighborhood probabilities resemble the original probabilities.
• Minimizing a divergence, commonly the Kullback-Leibler divergence, between the two distributions.

Strengths of t-SNE:

• Reveals local clusters that may not be visible with linear PCA.
• Often produces visually separated groups of related words.

Weaknesses of t-SNE:

• Global distances and cluster sizes may be misleading.
• Results depend on parameters such as perplexity, initialization, and random seed.
• It can be computationally expensive.

PCA is deterministic, faster, and easier to interpret as a linear projection. t-SNE is better for exploring local structure but should not be treated as a precise map of the entire embedding geometry.

PCA should be preferred for a quick, reproducible overview or when global structure matters. t-SNE is useful for exploratory analysis of local neighborhoods, provided that plots are interpreted cautiously and results are checked across multiple settings.

A typical Word2Vec training process includes:

• Preprocessing: Tokenize the corpus, normalize case when appropriate, handle punctuation, and remove or retain stop words based on the task.
• Vocabulary construction: Count word frequencies and discard extremely rare words if necessary.
• Training pair generation: Use a sliding context window to create center-context pairs for Skip-Gram or context-target examples for CBOW.
• Model initialization: Initialize input and output embedding matrices with small random values.
• Optimization: Update the parameters using stochastic gradient descent or a related optimizer. Negative sampling or hierarchical softmax is commonly used to make training efficient.
• Iteration: Repeat over the corpus for several epochs, possibly using subsampling for very frequent words.
• Vector extraction: Use the learned input embeddings, or combine input and output embeddings, as the final word representations.
• Evaluation: Test nearest neighbors, analogy performance, and downstream NLP tasks.

Embeddings capture semantic similarity because the training objective encourages words that occur in comparable contexts to have comparable vector relationships. If two words frequently appear near similar verbs, nouns, and modifiers, their vectors are adjusted in similar directions.

For example, "physician" and "doctor" may both occur near "patient," "clinic," and "treatment," causing them to become neighbors in the embedding space.

Similar words can still receive different embeddings because:

• They may have different frequencies.
• They may occur in different grammatical constructions.
• They may have different senses or domains.
• The corpus may contain limited evidence for one word.
• The learned space is affected by initialization, hyperparameters, and optimization.
• A model may distinguish usage patterns even when dictionary meanings are similar.

Therefore, embeddings represent distributional usage rather than a perfect dictionary definition.

A static word embedding assigns one vector to each vocabulary item. The word "bank" therefore receives the same vector in "river bank" and "bank loan."

A contextual representation generates a vector that depends on the sentence in which the word appears. The representations of "bank" in the two examples can therefore differ according to their meanings.

Static embeddings are inadequate for polysemous words because:

• Multiple senses are compressed into one vector.
• The vector often reflects the dominant sense in the training corpus.
• Similarity searches may mix unrelated meanings.
• Downstream models must infer the intended sense from context using the same fixed word vector.

Static embeddings remain useful because they are compact, reusable, and computationally inexpensive, but contextual models provide a more precise representation when word meaning changes across contexts.