Week 4

Chapter 4: Word Embeddings

Learn how words are represented as dense vectors that capture semantic meaning, from Word2Vec and GloVe to subword methods like FastText and BPE. Master the algorithms, training objectives, and evaluation methods behind modern word representations.

Chapter Overview

Word embeddings are one of the most important breakthroughs in NLP. Instead of treating words as arbitrary symbols (one-hot vectors), embeddings map each word to a dense, low-dimensional vector where geometric relationships encode semantic meaning. Words with similar meanings cluster together, and directions in the vector space correspond to linguistic relationships like gender, tense, and geography.

The key insight is that you can learn these vectors from raw text alone. By training a model to predict words from their contexts (or vice versa), the model is forced to encode meaning into its weight matrices. The resulting vectors transfer to downstream tasks, providing a powerful initialization for everything from sentiment analysis to machine translation.

This chapter covers the major embedding algorithms --- Word2Vec (CBOW and Skip-gram), GloVe, and FastText --- along with subword tokenization methods (BPE, SentencePiece) that handle out-of-vocabulary words. You will also learn how to evaluate embedding quality using analogy tasks, similarity benchmarks, and downstream performance.

Chapter Roadmap

Click any topic to jump in

Word2Vec CBOW

Predicting a center word from context — learning dense embeddings by averaging surrounding word vectors.

CBOW Objective FunctionContext Window and SlidingEmbedding AveragingTraining and Loss

Skip-gram

Predicting context from a center word — better for rare words via negative sampling.

Skip-gram ObjectiveNegative SamplingNoise Distribution

Bridging predictive and count-based

Global co-occurrence and subword morphology

GloVe

Factorizing the global co-occurrence matrix — combining count-based and predictive approaches.

Co-occurrence MatrixGloVe Objective FunctionWeighting FunctionCo-occurrence Probability Ratios

FastText

Character n-gram embeddings — handling OOV words and morphology with subword information.

Character N-gram DecompositionOut-of-Vocabulary HandlingHashing Trick for Memory Efficiency

Measuring quality

Embedding Evaluation

Analogy tasks, similarity benchmarks, and downstream metrics — measuring what embeddings capture.

Word Analogy TasksCosine Similarity and Distance MetricsIntrinsic vs. Extrinsic EvaluationWord Similarity Benchmarks

Modern tokenization foundation

Subword Embeddings

BPE and SentencePiece — tokenization-level solutions that feed modern transformers.

Byte Pair Encoding (BPE)SentencePieceMorphological AwarenessVocabulary Size Trade-offs

The Continuous Bag of Words (CBOW) model predicts a target word from its surrounding context words. Proposed by Mikolov et al. (2013), Word2Vec demonstrated that simple, shallow neural networks trained on massive corpora produce word vectors with remarkable semantic properties.

CBOW works by averaging the embedding vectors of all context words within a fixed window, then passing this average through a linear layer and softmax to predict the center word. Despite its simplicity --- just two weight matrices and no hidden nonlinearity --- CBOW learns vectors where semantic similarity corresponds to cosine similarity, and linear algebra on the vectors captures analogical relationships.

The key advantage of CBOW over earlier methods is scalability. Because the architecture is shallow, it can be trained on billions of words in hours rather than days. The resulting vectors serve as features for downstream NLP tasks, dramatically improving performance on everything from named entity recognition to question answering.

In this topic

1CBOW Objective Function

2Context Window and Sliding

3Embedding Averaging

4Training and Loss

1 of 4

CBOW Objective Function

$P(w_t \mid w_{t-c}, \ldots, w_{t+c}) = \text{softmax}(W' \cdot \bar{h})$

CBOW maximizes the probability of the center word $w_t$ given its context. The context is represented by averaging the input embedding vectors of surrounding words to produce $\bar{h} = \frac{1}{2c} \sum_{-c \leq j \leq c, j \neq 0} v_{w_{t+j}}$ . This average vector is projected through a second weight matrix $W'$ and passed through softmax over the entire vocabulary. The model has two sets of embeddings: input embeddings $W$ (used for context words) and output embeddings $W'$ (used for prediction). After training, usually only the input embeddings $W$ are kept.

Mathematical Intuition

CBOW maximizes $P(w_t \mid w_{t-c}, \ldots, w_{t+c}) = \text{softmax}(W' \bar{h})$ where $\bar{h}$ is the average of context embeddings. The softmax denominator $\sum_{w=1}^{V} \exp(\mathbf{v}'_w \cdot \bar{h})$ sums over all $|V|$ words — $O(|V|)$ per training step. With $|V| = 100{,}000$ and billions of training positions, this is the computational bottleneck that negative sampling and hierarchical softmax were designed to solve.

Example:

Given the sentence "the quick brown fox jumps" with window size 2, what is the training example when "brown" is the center word?

2 of 4

Context Window and Sliding

The window size $c$ controls how many words on each side of the center word are included as context. A window of 2 means 4 context words total (2 left, 2 right). The window slides across the entire corpus, generating one training example per position. Smaller windows capture syntactic patterns (part of speech, word order), while larger windows capture semantic/topical similarity. Word2Vec typically uses $c \in [2, 10]$ .

Mathematical Intuition

The context window of size $c$ produces $2c$ context words per position. Larger $c$ captures topical similarity (words about the same subject cluster together), while smaller $c$ captures syntactic similarity (words with the same part of speech cluster). Empirically, $c = 5$ works well for most tasks. The window slides $T$ times across the corpus, generating $T$ training examples — the entire training set is just the corpus itself.

Example:

For the sentence "I love natural language processing" with window size 1, list all (context, target) training pairs.

3 of 4

Embedding Averaging

$\bar{h} = \frac{1}{2c} \sum_{j=-c, j \neq 0}^{c} v_{w_{t+j}}$

CBOW averages the input embedding vectors of all context words to produce a single fixed-length representation $\bar{h}$ . This averaging is the "bag of words" part --- it discards word order within the context window. Despite this simplification, the averaging operation works well because the model learns to encode complementary information across context positions. The resulting $\bar{h}$ is a composite representation that captures the gist of the local context.

Mathematical Intuition

Embedding averaging $\bar{h} = \frac{1}{2c} \sum_{j} v_{w_{t+j}}$ is a lossy compression: $2c$ vectors are reduced to one. The information loss is bounded by the variance of the context embeddings: if all context vectors are similar (semantically coherent context), the average preserves most information. If they are dissimilar (diverse context), averaging destroys individual word signals. This is why CBOW struggles with rare words — their contexts are diverse.

Example:

If the embeddings for "the" = [0.1, 0.3], "cat" = [0.5, -0.1], "on" = [0.2, 0.4], "mat" = [-0.1, 0.2], what is $\bar{h}$ ?

4 of 4

Training and Loss

$\mathcal{L} = -\frac{1}{T} \sum_{t=1}^{T} \log P(w_t \mid w_{t-c}, \ldots, w_{t+c})$

The training objective is to minimize the negative log-likelihood averaged over all $T$ positions in the corpus. Computing the full softmax over a vocabulary of $V$ words is expensive (requires summing over all words for the normalizing constant), so practical implementations use hierarchical softmax or negative sampling to approximate the gradient. Stochastic gradient descent with linear learning rate decay is standard. Training typically requires one or two passes over the corpus.

Mathematical Intuition

The cross-entropy loss $\mathcal{L} = -\frac{1}{T} \sum_t \log P(w_t \mid \text{context})$ decreases as the model learns to predict center words. The gradient $\nabla_{\bar{h}} \mathcal{L} = \hat{y} - y$ (predicted minus true distribution) has magnitude proportional to the prediction error. After convergence, the embedding matrix $W \in \mathbb{R}^{|V| \times d}$ encodes each word's meaning in $d$ dimensions, where $d = 100\text{--}300$ is typical.

Example:

Why is computing the full softmax expensive for large vocabularies?

Theory Exercise

Problem:

Explain why CBOW tends to perform better on frequent words while Skip-gram performs better on rare words. What property of the CBOW architecture causes this?

Hints:

Think about how averaging affects rare vs. frequent words

Papers

Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013)Distributed Representations of Words and Phrases (Mikolov et al., 2013)

Blogs

The Illustrated Word2vec - Jay Alammar Word2Vec Explained - Hugging Face

Coding Exercise

Problem:

Implement a minimal CBOW model using NumPy. Create the forward pass that takes context word indices, looks up embeddings, averages them, and computes the softmax output.

Hints:

Use a random embedding matrix of shape (vocab_size, embed_dim)

Text Representation

Sequence Models