Chapter 2: Text Preprocessing

Martin Porter's 1980 algorithm strips English suffixes in five ordered steps. Step 1 handles plurals and -ed or -ing, step 2 maps double suffixes such as -ational to -ate, and later steps remove suffixes such as -ness and -ate and tidy a final e. Each rule fires only if the remaining stem is long enough, measured by $m$, the number of vowel-consonant sequences in it, which stops short words from being destroyed. The algorithm is fast and needs no dictionary. Its errors are over-stemming, conflating unrelated words ("organization", "organic", and "organ" all become "organ"), and under-stemming, missing irregular forms ("ran" stays "ran", "mice" stays "mice").

Snowball is a small language for writing stemmers, created by Martin Porter, and its English stemmer, often called Porter2, is a revised version of the original algorithm. It changes several rules: it defines the stem regions R1 and R2 more carefully, adds exceptions for words such as "generous", handles a final -ly better ("fairly" becomes "fair" rather than "fairli"), and treats a word-initial y as a consonant. Snowball also provides stemmers for many other languages; NLTK's SnowballStemmer covers 15, including French, German, Spanish, and Russian. It remains a suffix stripper, so it still misses irregular forms and still conflates some unrelated words.

NLTK's WordNetLemmatizer looks words up in WordNet, a lexical database with about 155,000 English words grouped into roughly 117,000 synonym sets. It applies inflection rules and an exception list, so irregular forms work: "mice" becomes "mouse" and "geese" becomes "goose". The output is always a real dictionary word. Its weakness is that it needs the part of speech, given as n, v, a, or r, and defaults to noun. Without the tag, "running" stays "running" and "better" stays "better"; with verb and adjective tags they become "run" and "good". In practice you run a POS tagger first and map its tags to WordNet's four categories.

Choose by what the base form is used for. Stemming is fast, needs no tagger or dictionary, and raises recall in search, because more surface forms match the same index key; the cost is lower precision when unrelated words collapse together. Lemmatization is slower and needs POS tags, but it returns real words and handles irregular forms, which matters when people read the output, for instance in keyword extraction or display text. Both throw away inflections that can carry meaning, such as tense. Pretrained models with subword tokenizers usually skip both steps, because they learn from all inflected forms and benefit from that information.

Libraries ship ready-made stopword lists, and they disagree. NLTK's current English list has 198 entries, and NLTK provides lists for 33 languages. spaCy's English list has 326 words and scikit-learn's has 318. The lists cover articles, prepositions, conjunctions, pronouns, auxiliaries, and negations such as "not" and "no". They were built for different purposes, and Nothman et al. (2018) found inconsistencies and surprising entries in several popular lists, such as scikit-learn including "system", "fire", and "bill". Whichever list you use, inspect it before applying it: removing negations, question words, or numbers can delete the very signal your task depends on.

A general list cannot know which words are uninformative in your corpus. In clinical notes, "patient", "history", and "denies" appear in almost every document; in legal filings, "court" and "section"; in machine-learning papers, "model" and "results". Such words carry little information for telling documents apart within that collection, so they crowd topic models and nearest-neighbor search. Adding them to the stopword list, or capping document frequency, often sharpens results. The risk is removing a word that matters in a subset: "patient" is noise in topic modeling but signal when separating patient-reported symptoms from clinician observations. Always check a candidate list against the downstream task.

Removing stopwords shrinks bag-of-words vectors, speeds up indexing, and stops function words from inflating similarity between unrelated documents. It hurts whenever function words carry meaning. Negations such as "not", "no", and "nor" are in NLTK's list, so sentiment can flip. Question words such as "who" and "where" are in the list, so questions lose their type. Phrases lose structure, as in "to be or not to be", which disappears entirely. Order-sensitive models such as Transformers use stopwords for grammar, so removal usually hurts them. A good default is to remove stopwords for topic models and lexical search and keep them for sentiment, question answering, translation, and pretrained models.

Instead of trusting a fixed list, you can let the corpus choose. Compute each term's document frequency $\text{DF}(t)$, the number of documents containing it, out of $N$ documents. Terms above a threshold, often around 80 to 90 percent of documents, act as stopwords for that corpus. The formula connects this to IDF: when $\text{DF}(t)$ approaches $N$, $\log(N / \text{DF}(t))$ approaches 0, so TF-IDF already shrinks such terms toward zero. scikit-learn exposes the same idea as max_df in its vectorizers. The method adapts to any domain, but on a small corpus it can flag topic words that just happen to be common there.

A character class matches one character from a set. Square brackets list the set, as in [aeiou], and ranges such as [A-Z] cover sequences. Shorthand classes cover common sets: \d is a digit, \w a word character (letter, digit, or underscore), and \s whitespace, with capital versions negating them. \b matches a word boundary without consuming a character. Quantifiers say how many times the preceding item repeats: * means zero or more, + one or more, ? zero or one, and {n,m} between n and m. Quantifiers are greedy by default and match as much as possible; adding ? makes them lazy. Python's \w matches Unicode letters by default.

Parentheses do two jobs in a pattern: they group items so a quantifier or alternation applies to all of them, and they capture the matched text for later use. When a pattern contains capture groups, re.findall() returns the group contents rather than the whole match, which makes extraction a one-liner. Named groups, written (?P<name>...), label each part, so code reads m.group('domain') instead of a group number. Non-capturing groups, written (?:...), group without capturing, which keeps findall output clean when grouping is only needed for alternation. A common bug is adding a group for alternation and suddenly getting only part of each match back.

re.sub(pattern, replacement, text) replaces every match with the replacement, which can be a string, a string with group references such as \1, or a function that computes each replacement. It is the main tool for cleaning: stripping tags, deleting URLs, collapsing runs of whitespace, and masking numbers or emails with placeholder tokens. Order matters, because each step changes the text the next step sees; for example, removing tags before URLs avoids leaving fragments of markup. Regex substitution suits flat patterns but cannot parse nested structures such as HTML, so for real web pages use an HTML parser to extract text and keep regex for the leftovers.

A few patterns cover most preprocessing needs: emails, [\w.+-]+@[\w-]+\.[\w.-]+; URLs, https?://\S+; hashtags, #\w+; mentions, @\w+; numbers with thousands separators and decimals, \d{1,3}(?:,\d{3})*(?:\.\d+)?; and simple dates. Each is a trade-off between precision and recall: no short regex matches every valid email address, and the simple URL pattern grabs trailing punctuation. Compile patterns once with re.compile, test them against real samples including edge cases, and prefer masking matches with placeholder tokens over deleting them when the model may need to know that a URL or number was present.

Lowercasing merges "The", "THE", and "the" into one token, which shrinks the vocabulary and pools training evidence for each word. It is standard for search and for uncased models such as bert-base-uncased. The cost is that case often carries meaning: "Apple" the company versus "apple" the fruit, "US" the country versus "us" the pronoun, and capitals that mark sentence starts or names, a key clue for named entity recognition. Truecasing restores the most likely casing from context, and cased models keep the information at the price of a larger vocabulary. For full Unicode text, str.casefold() is more aggressive than lower(), mapping German ß to "ss".

Unicode can encode the same visible text in different ways. "é" can be one code point, U+00E9, or two: "e" followed by the combining acute accent U+0301. The two strings look identical but compare unequal, so lookups, deduplication, and counting fail silently. Unicode defines four normalization forms: NFC composes characters where possible, NFD decomposes them, and NFKC and NFKD additionally fold compatibility characters, mapping the ligature "fi" to "f" and "i" or full-width letters to ASCII. Python's unicodedata.normalize('NFC', text) applies a form. NFC is the usual default for storage; NFKC suits search, but it is lossy, changing superscripts and other formatting distinctions.

Contractions fuse two words, as in "don't", "I'm", and "they've". Expanding them to "do not", "I am", and "they have" gives every word its own token, which helps whitespace tokenizers and makes negation explicit for bag-of-words models. A lookup table handles standard forms, but several are ambiguous: "he's" can be "he is" or "he has", and "I'd" can be "I would" or "I had", so resolving them correctly needs context such as the following verb. Penn Treebank tokenizers sidestep the problem by splitting "don't" into "do" and "n't", and subword models learn contractions directly, so expansion is mostly a concern for classical pipelines.

Accent removal maps "café" to "cafe" and "naïve" to "naive", so users who type without accents still find matches. The standard recipe is to decompose with NFD, which splits accented letters into a base letter plus combining marks, then delete every character in Unicode category Mn (nonspacing mark), then optionally recompose with NFC. It is deliberately lossy. In many languages accents distinguish words: Spanish "papá" (dad) and "papa" (potato), or French "où" (where) and "ou" (or). Letters that are not built from combining marks, such as "ø" or "ß", pass through unchanged. Use accent folding for matching and search keys, not for text a model needs to understand.

ASCII, standardized in 1963, uses 7 bits for 128 characters: English letters, digits, punctuation, and control codes. Unicode assigns a code point to every character in the world's scripts, more than 150,000 so far, and UTF-8 encodes each code point in 1 to 4 bytes. As the formula shows, code points up to U+007F take 1 byte, so every ASCII file is already valid UTF-8; Latin accents, Greek, Cyrillic, Arabic, and Hebrew take 2; most Chinese, Japanese, Korean, and Indic characters take 3; and emoji take 4. The variable width means byte length and character count differ, which breaks code that slices bytes or assumes one byte per character.

Bytes carry no label saying how to decode them, so files, scraped pages, and database dumps can arrive in UTF-8, Latin-1, Windows-1252, Shift-JIS, or something else. Detection libraries such as chardet and charset-normalizer guess from byte patterns and report a confidence, but short or mostly ASCII inputs are ambiguous. UTF-8 helps because its multibyte sequences follow strict rules, so random Latin-1 text rarely decodes as valid UTF-8. The classic failure is mojibake: UTF-8 bytes decoded as Latin-1 turn "é" into "é". When the damage is a single wrong decode, it is reversible by encoding back to Latin-1 and decoding as UTF-8.

Beyond letters, text carries characters that cause silent bugs. Zero-width characters such as U+200B (zero-width space) and the byte order mark U+FEFF are invisible but break string equality and tokenization. Non-breaking spaces (U+00A0) and curly quotes differ from their ASCII lookalikes, and homoglyphs such as Cyrillic "а" can spoof Latin letters. The replacement character U+FFFD signals that an earlier decode failed. Emoji are multi-byte and may combine several code points with joiners. For each category, choose to keep, remove, or map it to a token. Zero-width characters should almost always go, while emoji often carry sentiment and are worth keeping or converting to text.

Multilingual pipelines fail at the boundaries where bytes become text. A few habits prevent most bugs. Store and exchange text as UTF-8, and pass encoding='utf-8' to open() explicitly, because the platform default differs, notably on older Windows setups. Normalize to NFC right after decoding so equal strings compare equal. Remember that len() counts code points, not bytes or visible characters: a flag emoji is two code points, and accented letters may be one or two depending on normalization. Strip a leading byte order mark, which Windows tools often add to UTF-8 files, by reading with utf-8-sig. Finally, log and inspect decode errors instead of silently replacing them.