Chapter 7: Text Classification

The log-space computation $\log P(c \mid d) \propto \log P(c) + \sum_i \log P(w_i \mid c)$ converts products of small probabilities into sums of log-probabilities, avoiding numerical underflow. The naive independence assumption $P(d \mid c) = \prod_i P(w_i \mid c)$ is clearly wrong (words are correlated), yet the classifier works because the decision boundary $\log P(c_1 \mid d) > \log P(c_2 \mid d)$ only requires the correct ranking of classes, not calibrated probabilities. Laplace smoothing $P(w \mid c) = (\text{count}(w,c) + \alpha) / (\sum_{w'} \text{count}(w',c) + \alpha V)$ prevents zero probabilities by adding pseudo-counts, effectively applying a Dirichlet prior with concentration $\alpha$.

TF-IDF weights $w_{t,d} = \text{TF}(t,d) \cdot \log(N / \text{DF}(t))$ balance two signals: how important a term is to a specific document (TF) and how distinctive it is across the corpus (IDF). A word appearing 10 times in a document but in every document (like "the") gets high TF but low IDF, resulting in low weight. A word appearing 3 times but in only 5 of 10,000 documents gets moderate TF but very high IDF ($\log(10000/5) \approx 7.6$), making it highly discriminative. The IDF term is essentially the Shannon information content of observing a word: $-\log P(\text{word in random doc})$.

In hierarchical classification, the label space forms a tree or DAG where each node is a category. A local approach trains one classifier per internal node to distinguish among its children. A global approach flattens the hierarchy but constrains predictions to be consistent: if the model predicts "Genetics", the parent "Biology" and grandparent "Science" must also be predicted. The constraint can be enforced via masking: at inference time, only labels whose parents are already predicted (or above a threshold) are considered. The evaluation metric must account for partial credit — predicting "Biology" when the gold label is "Genetics" is partially correct.

When class frequencies differ by 10x or more, the majority class dominates the cross-entropy gradient because it contributes more terms to the loss sum. Cost-sensitive learning assigns class weights inversely proportional to frequency: $w_c = N / (C \cdot n_c)$ where $N$ is total samples, $C$ is number of classes, and $n_c$ is the count of class $c$. This makes each class contribute equally to the total loss in expectation. Focal loss $(1 - p_t)^\gamma \cdot (-\log p_t)$ takes a different approach: it down-weights easy (well-classified) examples regardless of class, focusing the model on the hard examples that tend to be from rare classes.

Binary relevance trains $C$ independent binary classifiers, one per label: $P(y_c = 1 \mid x) = \sigma(w_c^\top x + b_c)$. Each classifier uses binary cross-entropy $\mathcal{L}_c = -[y_c \log \hat{p}_c + (1-y_c)\log(1-\hat{p}_c)]$ and the total loss is $\sum_c \mathcal{L}_c$. The independence assumption means the model cannot capture label correlations — if labels A and B always co-occur, each classifier must learn this pattern independently from the features rather than from the other classifier's output. Despite this limitation, binary relevance is simple, parallelizable, and works well when label correlations are weak.

A classifier chain orders labels as $y_1, y_2, \ldots, y_C$ and trains each classifier to predict $P(y_c \mid x, y_1, \ldots, y_{c-1})$, conditioning on the true labels of all previous classifiers in the chain. This captures label dependencies: if $y_1 = \text{"politics"}$ is predicted, the classifier for $y_2 = \text{"election"}$ can use this as a feature. The chain approximates the joint distribution $P(y_1, \ldots, y_C \mid x) = \prod_c P(y_c \mid x, y_{<c})$ via the chain rule. Order matters: placing highly predictive labels earlier in the chain gives downstream classifiers more useful conditioning information.

Unlike multiclass classification where the prediction is always $\text{argmax}_c P(c \mid x)$, multi-label requires choosing a threshold $\tau$: predict label $c$ if $P(y_c = 1 \mid x) > \tau$. The default $\tau = 0.5$ is rarely optimal. Threshold tuning optimizes $\tau$ on a validation set to maximize F1 score: $F_1(\tau) = 2 \cdot P(\tau) \cdot R(\tau) / (P(\tau) + R(\tau))$. Per-label thresholds $\tau_c$ further improve performance because different labels may have different calibration. A rare label might achieve best F1 at $\tau = 0.3$ while a common label peaks at $\tau = 0.6$.

A 1D convolutional filter $w \in \mathbb{R}^{k \times d}$ slides over consecutive $k$-grams in the embedding matrix $X \in \mathbb{R}^{n \times d}$. At position $i$, the output is $c_i = \text{ReLU}(w \cdot X_{i:i+k-1} + b)$ where $X_{i:i+k-1}$ is a $k \times d$ submatrix. This is a pattern detector: the filter learns to fire when a specific $k$-gram pattern appears, regardless of where it appears in the sequence. Unlike RNNs that process sequentially, all positions are convolved in parallel, giving $O(1)$ depth for pattern detection versus $O(n)$ for an RNN to propagate information across the sequence.

Max pooling $\hat{c} = \max(c_1, c_2, \ldots, c_{n-k+1})$ extracts the strongest activation of each filter across the entire sequence. This makes the representation position-invariant: whether a bigram "not good" appears at position 3 or position 30, the same filter fires and the max pool captures it equally. The resulting feature is a single scalar per filter, creating a fixed-length representation regardless of input length. The gradient flows only through the argmax position, meaning the model learns from the single strongest match of each pattern, ignoring weaker matches elsewhere in the text.

Using filters of sizes $k \in \{3, 4, 5\}$ captures different n-gram scales: trigrams, 4-grams, and 5-grams. Each filter size produces a feature map of different length ($n-k+1$), but max pooling reduces each to a single value. With $m$ filters per size and 3 sizes, the final representation is a $3m$-dimensional vector. This multi-scale approach captures both local patterns ("very good") and slightly longer phrases ("not at all bad") simultaneously. The total parameter count is $m \times (3d + 4d + 5d) = 12md$ — much smaller than an LSTM's $4d(d + d_e)$ because CNN filters are small and shared across positions.

TextCNN composes embedding, parallel convolutions, max-over-time pooling, dropout, and a softmax layer, so sequence length matters only inside the convolutions. Kim (2014) compared four embedding setups. On the MR movie-review dataset, random embeddings scored 76.1 percent, frozen word2vec 81.0, fine-tuned word2vec 81.5, and a two-channel model with one frozen and one fine-tuned copy 81.1. Pretrained vectors gave the large jump, while the multichannel design did not consistently help. Kim's filters, 100 each of heights 3, 4, and 5 on 300-dimensional vectors, hold $100 \times 12 \times 300 = 360{,}000$ weights, and every window is convolved in parallel rather than in an RNN's sequential steps.

The classification head maps the [CLS] token representation $h_{\text{CLS}} \in \mathbb{R}^d$ to class logits: $z = W h_{\text{CLS}} + b$ where $W \in \mathbb{R}^{C \times d}$. The [CLS] representation aggregates information from the entire sequence via self-attention — it attends to all tokens and encodes a summary. For BERT-base ($d = 768$) with 3 classes, the head adds only $768 \times 3 + 3 = 2,307$ parameters on top of 110M pretrained parameters. This extreme parameter efficiency is why fine-tuning works: the pretrained model already has rich linguistic representations, and only a thin task-specific layer needs to be learned.

Discriminative fine-tuning uses different learning rates for different layers: lower layers get smaller rates (e.g., $2 \times 10^{-5}$) because their general linguistic features need minimal adjustment, while the classification head gets a larger rate (e.g., $1 \times 10^{-3}$) because it starts from random initialization. The common heuristic is $\eta_l = \eta_{\text{base}} \cdot \alpha^{L-l}$ for layer $l$ with decay factor $\alpha \approx 0.9$. Warmup (linearly increasing $\eta$ from 0 to $\eta_{\text{max}}$ over the first 5--10% of steps) prevents large early updates from destroying pretrained features before the optimizer has a good gradient estimate.

Catastrophic forgetting occurs when fine-tuning on a small task dataset overwrites the pretrained representations, degrading performance on the pretraining distribution and potentially reducing generalization. Mathematically, the fine-tuned parameters $\theta^*_{\text{ft}}$ move far from the pretrained parameters $\theta_{\text{pt}}$ in a direction that optimizes task loss but damages the general features. L2 regularization toward the pretrained weights $(\theta - \theta_{\text{pt}})^2$ keeps the model close. Elastic Weight Consolidation (EWC) goes further by weighting each parameter's penalty by its Fisher information — parameters that are important for the pretrained knowledge get stronger regularization.

LoRA (Low-Rank Adaptation) freezes the pretrained weights $W$ and adds a low-rank update $\Delta W = BA$ where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times d}$ with rank $r \ll d$. For $d = 768$ and $r = 8$, each adapted layer adds only $768 \times 8 \times 2 = 12,288$ parameters versus $768^2 = 589,824$ for full fine-tuning. The low-rank constraint acts as an inductive bias: task-specific adaptations tend to lie in a low-dimensional subspace of the full parameter space. Prompt tuning prepends $k$ learnable "soft prompt" tokens to the input, adding only $k \times d$ parameters while keeping the entire model frozen.

Precision $P = \text{TP}/(\text{TP} + \text{FP})$ measures what fraction of positive predictions are correct (penalizes false alarms). Recall $R = \text{TP}/(\text{TP} + \text{FN})$ measures what fraction of actual positives are found (penalizes misses). F1 $= 2PR/(P+R)$ is the harmonic mean, which is always less than or equal to the arithmetic mean — it heavily penalizes imbalanced P/R. If $P = 0.99$ and $R = 0.01$, the arithmetic mean is 0.50 (looks decent) but $F_1 = 0.02$ (correctly identifies the problem). This harsh penalty is why F1 is the standard metric for imbalanced classification.

Macro-average computes the metric per class then averages: $F_1^{\text{macro}} = \frac{1}{C} \sum_c F_{1,c}$, treating all classes equally regardless of size. Micro-average pools all TP/FP/FN across classes: $P^{\text{micro}} = \sum_c \text{TP}_c / \sum_c (\text{TP}_c + \text{FP}_c)$, weighting by class size. Weighted-average multiplies each class's metric by its support: $F_1^{\text{weighted}} = \sum_c (n_c / N) \cdot F_{1,c}$. For imbalanced datasets (95% class A, 5% class B), micro-F1 $\approx$ accuracy $\approx$ A's F1, while macro-F1 averages A and B equally, exposing poor performance on the rare class.

The confusion matrix $M_{ij} = |\{x : \hat{y}(x) = i, y(x) = j\}|$ shows which classes are confused with which. The diagonal contains correct predictions; off-diagonal entries reveal error patterns. Normalizing each row by its sum gives $P(\text{predicted} = i \mid \text{true} = j)$, making it easy to spot classes that are systematically misclassified. For instance, if the (negative, neutral) cell is large, the model struggles to distinguish mildly negative from neutral — a signal to improve feature engineering or add more training data for these borderline cases.

The ROC curve plots true positive rate ($\text{TPR} = \text{TP}/(\text{TP} + \text{FN})$) versus false positive rate ($\text{FPR} = \text{FP}/(\text{FP} + \text{TN})$) at every possible threshold. AUC is the integral under this curve, with 1.0 being perfect and 0.5 being random. Probabilistically, AUC equals $P(\text{score}(x^+) > \text{score}(x^-))$ — the probability that a randomly chosen positive example scores higher than a randomly chosen negative. This threshold-independent property is valuable when the operating threshold will be chosen later based on deployment constraints (e.g., "we can tolerate 5% false positive rate").