Chapter 1: Introduction to NLP

Most NLP tasks reduce to estimating $P(y \mid x)$ where $x$ is text and $y$ is the desired output. Bayes' theorem decomposes this as $P(y \mid x) = P(x \mid y) P(y) / P(x)$. A spam classifier with uniform prior needs only $P(x \mid \text{spam})$ vs $P(x \mid \text{ham})$ — whichever likelihood is larger wins. The ratio $P(x \mid \text{spam}) / P(x \mid \text{ham})$ is the Bayes factor: values above 1 favor spam, below 1 favor ham.

Analyzing language at $k$ levels means composing $k$ imperfect functions: $\text{semantics}(\text{syntax}(\text{morphology}(\text{phonology}(x))))$. If each level has accuracy $p$, the end-to-end accuracy is roughly $p^k$. With $p = 0.95$ and $k = 4$, you get $0.95^4 \approx 0.81$ — an 81% pipeline from 95% components. This explains why even small errors at the tokenization level propagate and compound through deeper analysis.

Rule-based systems have zero trainable parameters but $O(R)$ hand-written rules. Statistical models like Naive Bayes have $O(|V| \cdot |C|)$ parameters for vocabulary size $|V|$ and class count $|C|$. Neural models like BERT have $O(L \cdot d^2)$ parameters for $L$ layers of dimension $d$ — BERT-base has 110M. The parameter count roughly tracks with the amount of training data needed: more parameters require exponentially more data to avoid overfitting.

NLU maps text to a structured representation: $f_{\text{NLU}}: \text{string} \to \text{structure}$ (e.g., intent + entities). NLG maps structure back to text: $f_{\text{NLG}}: \text{structure} \to \text{string}$. Translation composes both: $f_{\text{NLG}} \circ f_{\text{NLU}}$. Modern seq2seq models learn this composition end-to-end, but the decomposition clarifies why translation is harder than classification — it requires both understanding and generation.