Week 2

Chapter 2: Text Preprocessing

Master the essential techniques for cleaning and transforming raw text into structured input for NLP models. From tokenization and stemming to normalization and encoding, these preprocessing steps form the foundation of every NLP pipeline.

Chapter Overview

Raw text is messy. It contains inconsistent casing, special characters, HTML artifacts, contractions, and encoding issues. Before any NLP model can extract meaning from text, we need a systematic preprocessing pipeline to clean, normalize, and structure the input.

Text preprocessing is not a one-size-fits-all process. The right pipeline depends on your task: sentiment analysis benefits from keeping exclamation marks and emoticons, while topic modeling usually removes stopwords and punctuation. A search engine needs stemming for recall, but a chatbot needs lemmatization for grammatical responses.

This chapter covers the six core preprocessing techniques every NLP practitioner must master:

Tokenization: splitting text into meaningful units (words, subwords, characters)
Stemming & Lemmatization: reducing words to their base forms
Stopword Removal: filtering out low-information words
Regular Expressions: pattern-based text extraction and cleaning
Text Normalization: standardizing text format (casing, unicode, contractions)
Text Encoding: handling character sets, UTF-8, and multilingual text

Chapter Roadmap

Click any topic to jump in

Tokenization

Splitting text into tokens — word, subword (BPE), and character strategies and their trade-offs.

Word TokenizationSubword Tokenization (BPE & WordPiece)Sentence TokenizationCharacter-Level Tokenization

Reducing vocabulary

Normalizing word forms and removing noise

Stemming & Lemmatization

Reducing words to base forms — fast heuristic stemming vs. linguistically correct lemmatization.

Porter StemmerSnowball StemmerWordNet LemmatizerStemming vs Lemmatization: When to Use Which

Stopword Removal

Filtering low-information words — standard lists, domain-specific words, and frequency-based detection.

Standard Stopword ListsDomain-Specific StopwordsStopword Removal Trade-offsFrequency-Based Stopword Detection

Cleaning and standardizing

Pattern-based extraction and format normalization

Regex for Text

Pattern matching and substitution — extracting emails, URLs, and cleaning HTML artifacts.

Character Classes and QuantifiersPattern Extraction with GroupsText Cleaning with SubstitutionCommon NLP Regex Patterns

Text Normalization

Case folding, unicode normalization, contraction expansion — standardizing text before modeling.

Case NormalizationUnicode NormalizationContraction ExpansionAccent and Diacritic Removal

Encoding the final text

Text Encoding

ASCII, UTF-8, and multilingual byte sequences — ensuring every character survives the pipeline.

ASCII and UTF-8Encoding Detection and ConversionHandling Special CharactersMultilingual Text and Encoding Best Practices

Tokenization is the process of splitting text into smaller, meaningful units called tokens. It is the very first step in virtually every NLP pipeline. The choice of tokenization strategy has profound effects on model vocabulary size, sequence length, and ability to handle unseen words.

At the word level, tokenization seems trivial---split on whitespace. But real text is full of edge cases: contractions ("don't"), hyphenated words ("state-of-the-art"), URLs, email addresses, and languages without whitespace (Chinese, Japanese). Modern NLP has largely moved to subword tokenization methods like Byte Pair Encoding (BPE) and WordPiece, which balance vocabulary size against token granularity.

The tokenization choice directly impacts downstream performance. Character-level tokenization produces long sequences that are expensive to process ( $O(n^2)$ for transformers), while word-level tokenization creates enormous vocabularies and cannot handle out-of-vocabulary words. Subword methods sit in the sweet spot: manageable vocabulary sizes (32K--128K tokens) with graceful handling of rare and unseen words.

In this topic

1Word Tokenization

2Subword Tokenization (BPE & WordPiece)

3Sentence Tokenization

4Character-Level Tokenization

1 of 4

Word Tokenization

The simplest approach: split text on whitespace and punctuation boundaries. Python's str.split() handles basic whitespace splitting, while NLTK's word_tokenize() uses the Penn Treebank rules to handle punctuation, contractions, and special cases. SpaCy's tokenizer is rule-based with exception lists for abbreviations and special tokens.

Mathematical Intuition

Word tokenization splits on whitespace and punctuation boundaries. For a string of length $L$ characters, a regex-based tokenizer runs in $O(L)$ time. The resulting vocabulary $|V|$ for a corpus follows Heaps' law: $|V| \approx k \cdot N^\beta$ where $N$ is total tokens, $k \approx 30$ , and $\beta \approx 0.5$ . A 1M-word corpus typically has $|V| \approx 30{,}000$ unique words.

Example:

Tokenize the sentence: "Dr. Smith's car costs $3,500.00!" using NLTK's word_tokenize.

2 of 4

Subword Tokenization (BPE & WordPiece)

$V_{\\text{subword}} \\ll V_{\\text{word}}, \\quad \\text{where } V = \\text{vocabulary size}$

Byte Pair Encoding (BPE) starts with individual characters and iteratively merges the most frequent adjacent pairs until a target vocabulary size is reached. WordPiece uses a similar approach but merges pairs that maximize the training data likelihood. Both produce a vocabulary of common words and frequent subword units, so rare words decompose into known pieces rather than becoming unknown tokens.

Mathematical Intuition

BPE starts with $|V_0| = 256$ byte-level tokens and performs $M$ merges, giving $|V| = 256 + M$ . Each merge greedily picks the most frequent adjacent pair $(a, b)$ and replaces all occurrences with $ab$ . The entropy of the token distribution decreases with each merge: $H(V_{i+1}) < H(V_i)$ , meaning tokens become more uniformly distributed. GPT-4 uses $\sim 100{,}000$ merges for a vocabulary of $\sim 100{,}256$ tokens.

Example:

Explain why subword tokenization handles the unseen word "unhappiest" gracefully.

3 of 4

Sentence Tokenization

Splitting text into individual sentences is essential for tasks like summarization, translation, and document processing. The challenge is distinguishing sentence-ending periods from abbreviations ("Dr.", "U.S.A."), decimal numbers ("3.14"), and ellipses ("..."). NLTK's sent_tokenize() uses the Punkt algorithm, which is trained on large corpora to learn abbreviation patterns.

Mathematical Intuition

The Punkt sentence tokenizer models abbreviation detection as a classification problem. It estimates $P(\text{abbrev} \mid w)$ using features like word length, final period, and capitalization of the next word. The decision boundary is: if $\log \text{likelihood-ratio} > \theta$ , classify as abbreviation (not sentence boundary). This handles 'Dr.', 'U.S.', and 'etc.' without a hand-curated list.

Example:

Split into sentences: "Dr. Smith arrived at 3 p.m. He said, 'Hello!' Ms. Jones replied."

4 of 4

Character-Level Tokenization

$\\text{Sequence length ratio} = \\frac{L_{\\text{char}}}{L_{\\text{word}}} \\approx 5\\text{--}6\\times$

Each character becomes a token. The vocabulary is tiny (around 256 for byte-level, ~100 for ASCII letters + digits + punctuation) and can represent any text without unknown tokens. The downside is dramatically longer sequences: a 10-word sentence becomes 50+ tokens, and transformers with $O(n^2)$ attention cost become expensive quickly.

Mathematical Intuition

Character-level tokenization has vocabulary $|V| \leq 256$ (byte-level) but sequence length $L_{\text{char}} \approx 5 L_{\text{word}}$ . For transformers with $O(n^2)$ attention, this means $\sim 25\times$ the compute cost. A 512-word document becomes $\sim 2{,}560$ characters — attention needs $2560^2 \approx 6.5M$ operations vs $512^2 \approx 262K$ for word-level, a $25\times$ increase.

Example:

Compare token counts for "Machine Learning" at word vs character level. What is the cost implication for a transformer?

Theory Exercise

Problem:

A search engine processes queries in 47 languages. You must choose a tokenization strategy. Word-level tokenization fails for Chinese (no spaces) and agglutinative languages like Turkish (one word = one sentence). Compare word, subword (BPE), and character tokenization for this use case. Which do you recommend and why?

Hints:

Consider languages without whitespace word boundaries

Papers

Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2015)SentencePiece: A Language-Independent Subword Tokenizer (Kudo & Richardson, 2018)

Blogs

Tokenizers: How Machines Read - Hugging Face Byte Pair Encoding Explained - Jay Alammar

Coding Exercise

Problem:

Write a Python function that tokenizes text using three methods (whitespace, NLTK word_tokenize, and character-level) and compares the token counts.

Hints:

Use str.split() for whitespace tokenization

Introduction to NLP

Text Representation

Chapter 2: Text Preprocessing

Chapter Overview

Chapter Roadmap

Tokenization

Stemming & Lemmatization

Stopword Removal

Regex for Text

Text Normalization

Text Encoding

Tokenization

In this topic

Word Tokenization

Subword Tokenization (BPE & WordPiece)

Sentence Tokenization

Character-Level Tokenization

Theory Exercise

Problem:

Hints:

Coding Exercise

Problem:

Hints:

Stemming & Lemmatization

Stopword Removal

Regular Expressions for Text

Text Normalization

Text Encoding