Week 3

Chapter 3: Text Representation

Transform raw text into numerical vectors that machine learning models can process. Master classical techniques from simple count-based methods to dimensionality reduction, building the foundation for modern embeddings.

Chapter Overview

Before any NLP model can process text, words and documents must be converted into numerical representations. This chapter covers the classical methods that dominated NLP for decades and still serve as strong baselines today.

The central challenge is capturing meaning in numbers. A naive approach (assigning word 1 = 1, word 2 = 2) imposes a false ordering. Instead, we need representations where similar texts map to nearby vectors. The methods here range from simple count-based approaches (Bag of Words) to statistically weighted schemes (TF-IDF) to techniques that capture word relationships through co-occurrence patterns.

Understanding these foundations is essential because:

Bag of Words and TF-IDF remain competitive for many classification tasks
Co-occurrence matrices inspired the word embeddings revolution (Word2Vec, GloVe)
Dimensionality reduction techniques (SVD, PCA) are used everywhere in modern NLP pipelines
Every modern embedding method can be understood as improving upon these classical approaches

This chapter covers:

Bag of Words: Converting documents to count vectors
TF-IDF: Weighting terms by importance across a corpus
N-grams: Capturing local word order and phrases
One-Hot Encoding: Binary word representations and their limitations
Co-occurrence Matrices: Learning from word context patterns
Dimensionality Reduction: Compressing sparse representations into dense vectors

Chapter Roadmap

Click any topic to jump in

Bag of Words

Count-based document vectors — ignoring word order to capture which words appear and how often.

Vocabulary ConstructionCount Vector RepresentationTerm Frequency (TF)Limitations of Bag of Words

Improving on raw counts

Weighting by importance and capturing local order

TF-IDF

Weighting terms by local frequency and global rarity — the workhorse of information retrieval.

TF-IDF FormulaInverse Document Frequency (IDF)TF-IDF VariantsTF-IDF for Document Retrieval

N-grams

Capturing local word order as token tuples — bigrams, trigrams, and language model probabilities.

N-gram DefinitionN-gram Language ModelsSkip-gramsData Sparsity and Smoothing

From sparse to distributional

Simple indicators and context-based statistics

One-Hot Encoding

Binary indicator vectors — the simplest representation and why it fails at scale.

One-Hot Vector DefinitionDimensionality ProblemEmbedding Layer as One-Hot Lookup

Co-occurrence Matrices

Counting word-context pairs and PMI weighting — the statistical foundation of word embeddings.

Word-Context Co-occurrence MatrixPointwise Mutual Information (PMI)PPMI (Positive PMI)SVD on Co-occurrence Matrices

Compressing into dense representations

Dimensionality Reduction

PCA, SVD, and LSA — compressing sparse representations into dense vectors that generalize.

Principal Component Analysis (PCA)Singular Value Decomposition (SVD)Latent Semantic Analysis (LSA)Truncated SVD for Sparse Matrices

The Bag of Words (BoW) model is the simplest and most intuitive method for converting text into numerical vectors. Each document is represented as a vector of word counts, completely ignoring grammar and word order.

Despite its simplicity, BoW is surprisingly effective for many tasks like spam detection, sentiment analysis, and document classification. The key insight is that which words appear often matters more than how they are arranged.

Formally, given a vocabulary $V$ of $|V|$ unique words, each document $d$ is represented as a vector $\mathbf{x} \in \mathbb{R}^{|V|}$ where $x_i$ is the count of the $i$ -th vocabulary word in $d$ .

In this topic

1Vocabulary Construction

2Count Vector Representation

3Term Frequency (TF)

4Limitations of Bag of Words

1 of 4

Vocabulary Construction

$V = \{w_1, w_2, \ldots, w_n\} = \bigcup_{d \in D} \text{unique}(d)$

The vocabulary is the set of all unique words across the entire corpus $D$ . Each word gets a fixed index position. The vocabulary size $|V|$ determines the dimensionality of every document vector. In practice, vocabularies are pruned by removing rare words (appearing in fewer than $k$ documents) and very common words (stop words) to reduce dimensionality.

Mathematical Intuition

The vocabulary $V = \bigcup_{d \in D} \text{unique}(d)$ grows sublinearly with corpus size by Heaps' law: $|V| \approx k N^\beta$ . Each document vector $\mathbf{x} \in \mathbb{R}^{|V|}$ has at most $|d|$ non-zero entries out of $|V|$ dimensions. For a typical corpus with $|V| = 50{,}000$ and average document length 200 words, each vector is $99.6\%$ zeros — sparse matrix storage is essential.

Example:

Build the vocabulary for the corpus: ["the cat sat", "the dog ran", "a cat ran"]. What is the dimensionality?

2 of 4

Count Vector Representation

$\mathbf{x}_d = [\text{count}(w_1, d), \text{count}(w_2, d), \ldots, \text{count}(w_{|V|}, d)]$

Each document $d$ becomes a vector where the $i$ -th element is the number of times word $w_i$ appears in $d$ . This creates a document-term matrix $M \in \mathbb{R}^{|D| \times |V|}$ where each row is a document and each column is a word. The matrix is typically very sparse since most documents use only a small fraction of the vocabulary.

Mathematical Intuition

The document-term matrix $M \in \mathbb{R}^{|D| \times |V|}$ has $M_{ij} = \text{count}(w_j, d_i)$ . Two documents sharing $k$ words have dot product $\mathbf{x}_1 \cdot \mathbf{x}_2 = \sum_{j: w_j \in d_1 \cap d_2} c_{1j} \cdot c_{2j}$ . Cosine similarity normalizes this by vector magnitudes, removing the length bias: longer documents do not automatically appear more similar to everything.

Example:

Using vocabulary [a, cat, dog, ran, sat, the], represent "the cat sat" and "a cat ran" as count vectors.

3 of 4

Term Frequency (TF)

$\text{TF}(t, d) = \frac{f_{t,d}}{\sum_{t' \in d} f_{t',d}}$

Raw counts can be misleading: a longer document naturally has higher counts. Term Frequency normalizes by dividing each word count by the total number of words in the document. This gives a proportion rather than an absolute count. Variants include binary TF (1 if present, 0 otherwise), logarithmic TF ( $1 + \log f_{t,d}$ ), and augmented TF that prevents bias toward longer documents.

Mathematical Intuition

Term frequency $\text{TF}(t,d) = f_{t,d} / \sum_{t'} f_{t',d}$ normalizes raw counts to a probability distribution over the vocabulary. The sum $\sum_t \text{TF}(t,d) = 1$ for every document, regardless of length. This means TF treats a 10-word tweet and a 10,000-word article on equal footing — the representation captures word proportions, not absolute counts.

Example:

Compute TF for every word in "the cat sat on the mat" (6 words total).

4 of 4

Limitations of Bag of Words

BoW has three fundamental limitations: (1) No word order -- "dog bites man" and "man bites dog" produce identical vectors. (2) High dimensionality -- vocabulary sizes of 50K-100K create very sparse vectors. (3) No semantics -- "happy" and "joyful" are treated as completely unrelated dimensions. These limitations motivate TF-IDF (for importance weighting), n-grams (for local order), and word embeddings (for semantics).

Mathematical Intuition

BoW is invariant to word permutation: any reordering of words produces the same vector. Formally, for any permutation $\pi$ , $\text{BoW}(w_1, \ldots, w_n) = \text{BoW}(w_{\pi(1)}, \ldots, w_{\pi(n)})$ . This means $n!$ distinct sentences map to the same representation. For a 10-word sentence, that is $10! = 3{,}628{,}800$ distinct orderings collapsed into one vector — a massive information loss that n-grams partially recover.

Example:

Show that BoW cannot distinguish "the cat chased the dog" from "the dog chased the cat".

Theory Exercise

Problem:

A corpus has 10,000 documents with an average of 200 words each. The vocabulary after preprocessing has 25,000 unique words. (a) What is the shape of the document-term matrix? (b) Estimate the sparsity (percentage of zero entries). (c) Why is sparsity a problem for machine learning?

Hints:

Each document is a row, each word is a column

Papers

Speech and Language Processing Ch. 6 (Jurafsky & Martin)

Blogs

Bag of Words Explained - Scikit-learn

Coding Exercise

Problem:

Use scikit-learn's CountVectorizer to convert a list of 3 sentences into a document-term matrix. Print the vocabulary, the matrix shape, and the dense matrix.

Hints:

Import CountVectorizer from sklearn.feature_extraction.text

Text Preprocessing

Word Embeddings