Loading...
Transform raw text into numerical vectors that machine learning models can process. Master classical techniques from simple count-based methods to dimensionality reduction, building the foundation for modern embeddings.
Before any NLP model can process text, words and documents must be converted into numerical representations. This chapter covers the classical methods that dominated NLP for decades and still serve as strong baselines today.
The central challenge is capturing meaning in numbers. A naive approach (assigning word 1 = 1, word 2 = 2) imposes a false ordering. Instead, we need representations where similar texts map to nearby vectors. The methods here range from simple count-based approaches (Bag of Words) to statistically weighted schemes (TF-IDF) to techniques that capture word relationships through co-occurrence patterns.
Understanding these foundations is essential because:
This chapter covers:
Click any topic to jump in
Count-based document vectors — ignoring word order to capture which words appear and how often.
Weighting by importance and capturing local order
Weighting terms by local frequency and global rarity — the workhorse of information retrieval.
Capturing local word order as token tuples — bigrams, trigrams, and language model probabilities.
Simple indicators and context-based statistics
Binary indicator vectors — the simplest representation and why it fails at scale.
Counting word-context pairs and PMI weighting — the statistical foundation of word embeddings.
PCA, SVD, and LSA — compressing sparse representations into dense vectors that generalize.
The Bag of Words (BoW) model is the simplest and most intuitive method for converting text into numerical vectors. Each document is represented as a vector of word counts, completely ignoring grammar and word order.
Despite its simplicity, BoW is surprisingly effective for many tasks like spam detection, sentiment analysis, and document classification. The key insight is that which words appear often matters more than how they are arranged.
Formally, given a vocabulary of unique words, each document is represented as a vector where is the count of the -th vocabulary word in .
The vocabulary is the set of all unique words across the entire corpus . Each word gets a fixed index position. The vocabulary size determines the dimensionality of every document vector. In practice, vocabularies are pruned by removing rare words (appearing in fewer than documents) and very common words (stop words) to reduce dimensionality.
The vocabulary grows sublinearly with corpus size by Heaps' law: . Each document vector has at most non-zero entries out of dimensions. For a typical corpus with and average document length 200 words, each vector is zeros — sparse matrix storage is essential.
Build the vocabulary for the corpus: ["the cat sat", "the dog ran", "a cat ran"]. What is the dimensionality?
Each document becomes a vector where the -th element is the number of times word appears in . This creates a document-term matrix where each row is a document and each column is a word. The matrix is typically very sparse since most documents use only a small fraction of the vocabulary.
The document-term matrix has . Two documents sharing words have dot product . Cosine similarity normalizes this by vector magnitudes, removing the length bias: longer documents do not automatically appear more similar to everything.
Using vocabulary [a, cat, dog, ran, sat, the], represent "the cat sat" and "a cat ran" as count vectors.
Raw counts can be misleading: a longer document naturally has higher counts. Term Frequency normalizes by dividing each word count by the total number of words in the document. This gives a proportion rather than an absolute count. Variants include binary TF (1 if present, 0 otherwise), logarithmic TF (), and augmented TF that prevents bias toward longer documents.
Term frequency normalizes raw counts to a probability distribution over the vocabulary. The sum for every document, regardless of length. This means TF treats a 10-word tweet and a 10,000-word article on equal footing — the representation captures word proportions, not absolute counts.
Compute TF for every word in "the cat sat on the mat" (6 words total).
BoW has three fundamental limitations: (1) No word order -- "dog bites man" and "man bites dog" produce identical vectors. (2) High dimensionality -- vocabulary sizes of 50K-100K create very sparse vectors. (3) No semantics -- "happy" and "joyful" are treated as completely unrelated dimensions. These limitations motivate TF-IDF (for importance weighting), n-grams (for local order), and word embeddings (for semantics).
BoW is invariant to word permutation: any reordering of words produces the same vector. Formally, for any permutation , . This means distinct sentences map to the same representation. For a 10-word sentence, that is distinct orderings collapsed into one vector — a massive information loss that n-grams partially recover.
Show that BoW cannot distinguish "the cat chased the dog" from "the dog chased the cat".
A corpus has 10,000 documents with an average of 200 words each. The vocabulary after preprocessing has 25,000 unique words. (a) What is the shape of the document-term matrix? (b) Estimate the sparsity (percentage of zero entries). (c) Why is sparsity a problem for machine learning?
Use scikit-learn's CountVectorizer to convert a list of 3 sentences into a document-term matrix. Print the vocabulary, the matrix shape, and the dense matrix.