Chapter 11: NLP Fundamentals

Word2Vec learns embeddings from a prediction task on raw text. Skip-gram takes each word and predicts the words within a small window around it; CBOW does the reverse, predicting the center word from the average of its neighbors. Instead of a full softmax over the vocabulary, negative sampling trains a logistic classifier to tell real word-context pairs from random ones, with noise words drawn in proportion to frequency to the 3/4 power. Words that appear in similar contexts receive similar gradients and end up with similar vectors, usually 100 to 300 dimensions. Each word gets exactly one vector, so all senses of an ambiguous word are blended.

GloVe, Global Vectors, starts from the corpus-wide co-occurrence matrix $X$, where $X_{ij}$ counts how often word $j$ appears near word $i$. It then fits vectors so that the dot product of two word vectors plus two biases approximates $\log X_{ij}$, using weighted least squares. A weighting function caps very frequent pairs and gives zero weight to pairs that never co-occur, so only the nonzero entries are visited. The motivation is that ratios of co-occurrence probabilities carry meaning: 'ice' co-occurs with 'solid' far more than 'steam' does. GloVe and Word2Vec reach similar quality; Levy and Goldberg showed that skip-gram implicitly factorizes a shifted PMI matrix.

An embedding layer is a trainable matrix $E$ with one row per vocabulary entry and one column per dimension. The forward pass simply looks up row $i$ for token id $i$, which is mathematically the same as multiplying a one-hot vector by $E$ but far cheaper. During backpropagation only the rows of tokens present in the batch receive gradient, so updates are sparse. The layer can start random and learn with the task, or start from pretrained vectors such as GloVe and then be fine-tuned or frozen. Its parameter count, vocabulary size times dimension, is often a large share of a small model.

Static embeddings give every occurrence of a word the same vector, so 'bank' in 'river bank' and 'bank account' are identical. Contextual embeddings come from a model that reads the whole sentence. ELMo used bidirectional LSTMs; BERT and GPT use Transformer layers, where self-attention mixes each token's vector with those of its neighbors at every layer. The output vector for a token is therefore a function of the entire input, so different senses land in different regions of the space. These vectors are not stored in a table: they are computed on the fly, which costs a forward pass but resolves ambiguity.

Bag of words represents a document as a vector with one entry per vocabulary word, holding its count or presence; word order is discarded. TF-IDF reweights each count by the log of the number of documents divided by the number containing the word, so common words fade and distinctive ones stand out. The vectors are high-dimensional and over 99 percent zeros, stored sparsely, and a linear model such as logistic regression or a linear SVM on top is fast, strong, and interpretable. The weakness is order: 'dog bites man' and 'man bites dog' get identical vectors. Adding n-grams, adjacent word pairs, recovers some local order.

A recurrent classifier embeds each token, runs an LSTM or GRU across the sequence, and feeds a summary vector to a softmax layer. The simplest summary is the final hidden state, which must compress the whole text; max pooling or attention over all hidden states usually works better on long inputs. A bidirectional LSTM runs a second LSTM from right to left and concatenates both directions, so every position sees its full context. Recurrent models capture order and negation that bag of words misses, but they process tokens one at a time, which limits parallelism, and they were largely replaced by Transformers after 2018.

Fine-tuning BERT for classification adds one linear layer that maps the final [CLS] vector, 768 numbers in BERT-base, to $K$ class logits, then trains the whole network on labeled data with a small learning rate, typically 2e-5 to 5e-5 for 2 to 4 epochs. Pretraining already taught the network syntax and word meaning, so a few thousand labels often beat models trained from scratch on far more. With very little data or compute, a cheaper option is to freeze the encoder and train only the head, or the head plus the top layers. Too high a learning rate can erase pretrained knowledge, sometimes called catastrophic forgetting.

In multi-label classification a text can carry any subset of $K$ labels: a film can be funny and romantic at once. Softmax is the wrong output, because it forces the $K$ probabilities to sum to 1, so raising one label lowers the others. Instead each label gets its own sigmoid, an independent yes or no probability, and the loss is binary cross-entropy summed over labels. At prediction time each label is thresholded separately, often at 0.5, though tuning one threshold per label on validation data usually raises F1, especially for rare labels. Evaluation uses micro- or macro-averaged F1 rather than plain accuracy.

BERT is a stack of Transformer encoder layers; BERT-base has 12 layers, hidden size 768, 12 heads, and 110 million parameters. It is pretrained with masked language modeling: 15 percent of input tokens are selected, and of those 80 percent become [MASK], 10 percent a random token, and 10 percent stay unchanged; the model must recover the originals. Because attention is bidirectional, each prediction uses context on both sides. The original recipe also had next-sentence prediction, which RoBERTa later showed was unnecessary. BERT produces excellent representations for classification, tagging, and extractive question answering, but it has no natural left-to-right way to generate text.

GPT is a stack of Transformer decoder layers with a causal mask, so position $t$ attends only to positions up to $t$. It is pretrained to predict the next token, which uses every position of every text as a training example with no labels needed. Generation is a loop: run the model, pick a token from the output distribution by greedy choice or sampling, append it, and repeat; a key-value cache avoids recomputing earlier positions. Scaling this recipe from GPT-2's 1.5 billion parameters to GPT-3's 175 billion produced few-shot learning from prompts alone. The cost is that every output token needs its own forward pass.

T5, the Text-to-Text Transfer Transformer, keeps the original encoder-decoder layout: an encoder reads the input bidirectionally, and a decoder generates the output causally while cross-attending to the encoder's states. Every task becomes text in and text out, marked by a prefix such as 'translate English to German:' or 'summarize:', and even labels are generated as words, so one model, one loss, and one decoding procedure cover translation, summarization, classification, and question answering. Pretraining uses span corruption: random spans are replaced by sentinel tokens and the decoder reproduces them. The cost is two stacks, and generating labels as text can occasionally produce invalid outputs.

Adapting a pretrained Transformer ranges from cheap to expensive. Feature extraction freezes the backbone and trains only a new head. Full fine-tuning updates every weight; it is usually most accurate but stores a full model copy per task. Parameter-efficient methods train a tiny add-on. LoRA freezes each weight matrix $W$ and learns an update $BA$, where $B$ is $d \times r$ and $A$ is $r \times d$ with rank $r$ much smaller than $d$; $B$ starts at zero, so training begins from the original model. Adapters and prompt tuning are relatives. LoRA updates can be merged into $W$ after training, adding no inference latency.