Chapter 12: Text Generation & Summarization

Text generation is the task of producing coherent, fluent natural language output conditioned on some input --- a prompt, a source document, a dialogue history, or simply a start token. It underlies virtually every modern NLP application: machine translation generates target-language text, summarization generates condensed versions of documents, dialogue systems generate responses, and large language models generate continuations of arbitrary prompts.

The core challenge in text generation is decoding: given a language model that assigns probabilities to the next token, how do you select tokens to form a complete output? Greedy decoding picks the highest-probability token at each step but often produces repetitive, dull text. Beam search maintains multiple hypotheses to find higher-probability sequences but can still lack diversity. Sampling methods --- top-k, nucleus (top-p), and temperature scaling --- introduce controlled randomness to produce more natural and varied outputs.

Summarization is one of the most important applications of text generation. Extractive summarization selects and concatenates the most informative sentences from the source document. Abstractive summarization generates a novel summary using a seq2seq model, potentially rephrasing, compressing, and synthesizing information. Both approaches require evaluation metrics that capture content overlap and quality --- ROUGE measures n-gram overlap, BERTScore uses contextual embeddings for semantic similarity, and human evaluation remains the gold standard for fluency and faithfulness.