Loading...
Learn how words are represented as dense vectors that capture semantic meaning, from Word2Vec and GloVe to subword methods like FastText and BPE. Master the algorithms, training objectives, and evaluation methods behind modern word representations.
Word embeddings are one of the most important breakthroughs in NLP. Instead of treating words as arbitrary symbols (one-hot vectors), embeddings map each word to a dense, low-dimensional vector where geometric relationships encode semantic meaning. Words with similar meanings cluster together, and directions in the vector space correspond to linguistic relationships like gender, tense, and geography.
The key insight is that you can learn these vectors from raw text alone. By training a model to predict words from their contexts (or vice versa), the model is forced to encode meaning into its weight matrices. The resulting vectors transfer to downstream tasks, providing a powerful initialization for everything from sentiment analysis to machine translation.
This chapter covers the major embedding algorithms --- Word2Vec (CBOW and Skip-gram), GloVe, and FastText --- along with subword tokenization methods (BPE, SentencePiece) that handle out-of-vocabulary words. You will also learn how to evaluate embedding quality using analogy tasks, similarity benchmarks, and downstream performance.
Click any topic to jump in
Predicting a center word from context — learning dense embeddings by averaging surrounding word vectors.
Predicting context from a center word — better for rare words via negative sampling.
Global co-occurrence and subword morphology
Factorizing the global co-occurrence matrix — combining count-based and predictive approaches.
Character n-gram embeddings — handling OOV words and morphology with subword information.
Analogy tasks, similarity benchmarks, and downstream metrics — measuring what embeddings capture.
BPE and SentencePiece — tokenization-level solutions that feed modern transformers.
The Continuous Bag of Words (CBOW) model predicts a target word from its surrounding context words. Proposed by Mikolov et al. (2013), Word2Vec demonstrated that simple, shallow neural networks trained on massive corpora produce word vectors with remarkable semantic properties.
CBOW works by averaging the embedding vectors of all context words within a fixed window, then passing this average through a linear layer and softmax to predict the center word. Despite its simplicity --- just two weight matrices and no hidden nonlinearity --- CBOW learns vectors where semantic similarity corresponds to cosine similarity, and linear algebra on the vectors captures analogical relationships.
The key advantage of CBOW over earlier methods is scalability. Because the architecture is shallow, it can be trained on billions of words in hours rather than days. The resulting vectors serve as features for downstream NLP tasks, dramatically improving performance on everything from named entity recognition to question answering.
CBOW maximizes the probability of the center word given its context. The context is represented by averaging the input embedding vectors of surrounding words to produce . This average vector is projected through a second weight matrix and passed through softmax over the entire vocabulary. The model has two sets of embeddings: input embeddings (used for context words) and output embeddings (used for prediction). After training, usually only the input embeddings are kept.
CBOW maximizes where is the average of context embeddings. The softmax denominator sums over all words — per training step. With and billions of training positions, this is the computational bottleneck that negative sampling and hierarchical softmax were designed to solve.
Given the sentence "the quick brown fox jumps" with window size 2, what is the training example when "brown" is the center word?
The window size controls how many words on each side of the center word are included as context. A window of 2 means 4 context words total (2 left, 2 right). The window slides across the entire corpus, generating one training example per position. Smaller windows capture syntactic patterns (part of speech, word order), while larger windows capture semantic/topical similarity. Word2Vec typically uses .
The context window of size produces context words per position. Larger captures topical similarity (words about the same subject cluster together), while smaller captures syntactic similarity (words with the same part of speech cluster). Empirically, works well for most tasks. The window slides times across the corpus, generating training examples — the entire training set is just the corpus itself.
For the sentence "I love natural language processing" with window size 1, list all (context, target) training pairs.
CBOW averages the input embedding vectors of all context words to produce a single fixed-length representation . This averaging is the "bag of words" part --- it discards word order within the context window. Despite this simplification, the averaging operation works well because the model learns to encode complementary information across context positions. The resulting is a composite representation that captures the gist of the local context.
Embedding averaging is a lossy compression: vectors are reduced to one. The information loss is bounded by the variance of the context embeddings: if all context vectors are similar (semantically coherent context), the average preserves most information. If they are dissimilar (diverse context), averaging destroys individual word signals. This is why CBOW struggles with rare words — their contexts are diverse.
If the embeddings for "the" = [0.1, 0.3], "cat" = [0.5, -0.1], "on" = [0.2, 0.4], "mat" = [-0.1, 0.2], what is ?
The training objective is to minimize the negative log-likelihood averaged over all positions in the corpus. Computing the full softmax over a vocabulary of words is expensive (requires summing over all words for the normalizing constant), so practical implementations use hierarchical softmax or negative sampling to approximate the gradient. Stochastic gradient descent with linear learning rate decay is standard. Training typically requires one or two passes over the corpus.
The cross-entropy loss decreases as the model learns to predict center words. The gradient (predicted minus true distribution) has magnitude proportional to the prediction error. After convergence, the embedding matrix encodes each word's meaning in dimensions, where is typical.
Why is computing the full softmax expensive for large vocabularies?
Explain why CBOW tends to perform better on frequent words while Skip-gram performs better on rare words. What property of the CBOW architecture causes this?
Implement a minimal CBOW model using NumPy. Create the forward pass that takes context word indices, looks up embeddings, averages them, and computes the softmax output.