Chapter 7: Prompt Engineering

Prompt specificity reduces output entropy: a vague prompt $x_v$ gives $H(Y \mid x_v) \gg H(Y \mid x_s)$ for a specific prompt $x_s$, where $H$ is the conditional entropy of the model's output distribution. Each constraint in the prompt eliminates a fraction of the output space: with $k$ independent constraints each reducing options by half, the remaining space is $2^{-k}$ of the original. In practice, constraints are correlated, so the reduction follows $|\mathcal{Y}_{\text{constrained}}| \approx |\mathcal{Y}| \cdot e^{-\alpha k}$ where $\alpha < \ln 2$ is the average constraint strength.

Role assignment shifts the model's conditional distribution from $P(y \mid x)$ to $P(y \mid \text{role}, x)$. By Bayes' rule, this is proportional to $P(\text{role} \mid y, x) \cdot P(y \mid x)$ — the model up-weights responses that are consistent with the assigned role. The effectiveness depends on how well the role is represented in pretraining data: 'expert cardiologist' activates patterns from medical texts (high-quality training signal), while 'alien from planet Zorg' has no grounding and mainly affects style. The expected quality improvement is proportional to $D_{\text{KL}}(P(y \mid \text{role}, x) \| P(y \mid x))$ — larger distributional shifts indicate the role is having more effect.

Decomposing a task into $k$ sequential steps reduces the effective complexity from $O(C^k)$ to $O(k \cdot C)$ where $C$ is the complexity per step. This is because the model processes one step at a time, each with bounded context. The error probability also changes: for independent steps with per-step error $\epsilon$, the overall success probability is $(1 - \epsilon)^k \approx 1 - k\epsilon$ for small $\epsilon$. With $k = 5$ steps and $\epsilon = 0.05$ per step, overall success is ~77%. Without decomposition, the single-step error for the full task might be 50%+ because the model must implicitly manage all subtasks simultaneously.

Negative instructions ('do not X') constrain the output distribution by zeroing out probability mass on undesired outputs: $P'(y) \propto P(y) \cdot \mathbb{1}[y \notin \mathcal{Y}_{\text{forbidden}}]$. The effectiveness depends on how much probability mass the model originally placed on forbidden outputs. If $P(y \in \mathcal{Y}_{\text{forbidden}}) = p$, then after the constraint, the remaining outputs are renormalized by $1/(1-p)$. A common failure mode: the constraint is too vague ("don't be verbose") so $\mathcal{Y}_{\text{forbidden}}$ is not well-defined, and the model cannot reliably exclude those outputs.

In-context learning is a form of implicit Bayesian inference: given examples $(x_1, y_1), \ldots, (x_k, y_k)$ in the prompt, the model computes $P(y \mid x, \{(x_i, y_i)\}) \approx \sum_f P(y \mid x, f) P(f \mid \{(x_i, y_i)\})$ — it infers a latent function $f$ from the examples and applies it to the new input. The transformer's attention mechanism implements this by attending from the query to the example outputs, computing a weighted combination. The effective 'training' from $k$ examples occurs entirely in the forward pass — no gradient updates, just context-dependent activation patterns.

Performance scales logarithmically with shot count: $\text{acc}(k) \approx a + b \cdot \log(k)$ for $k$ examples. This means the marginal value of each additional example decreases rapidly: going from 1 to 3 shots often improves accuracy by 10-15%, but going from 5 to 15 shots may only add 2-3%. The practical limit is the context window: $k$ examples each consuming $L$ tokens leaves $C - kL$ tokens for the actual query and response, where $C$ is the context length. The optimal $k^*$ balances example coverage against context space: $k^* = \arg\max_k [b \log k - \lambda k L / C]$ where $\lambda$ penalizes context consumption.

Choosing the most informative examples is a combinatorial optimization: from a pool of $N$ candidates, select $k$ to maximize task performance. Random selection gives a baseline; similarity-based selection (choosing examples closest to the query in embedding space: $\arg\min_{S \subset [N], |S|=k} \sum_{i \in S} \|e(x_i) - e(x_{\text{query}})\|$) typically improves by 5-10%. Diverse selection (maximizing coverage of the input space) can further improve by reducing redundancy: $\arg\max_S \sum_{i \in S} \text{sim}(x_i, x_{\text{query}}) - \lambda \sum_{i,j \in S} \text{sim}(x_i, x_j)$, balancing relevance and diversity.

Format consistency establishes a pattern that the model extrapolates. If examples use format $F = (\text{prefix}, \text{delimiter}, \text{suffix})$, the model assigns higher probability to outputs matching $F$: $P(y \in F \mid \text{examples in } F) \gg P(y \in F \mid \text{mixed formats})$. The attention mechanism learns to copy the formatting pattern: attention weights from the output position to delimiter tokens in examples become large, effectively 'locking in' the structural template. Inconsistent formatting reduces this pattern-matching signal by splitting attention across competing format hypotheses, reducing the model's confidence in any single output structure.

Chain-of-thought decomposes the conditional distribution $P(\text{answer} \mid \text{question})$ into $P(\text{answer} \mid \text{reasoning}, \text{question}) \cdot P(\text{reasoning} \mid \text{question})$. The key insight: $P(\text{correct answer} \mid \text{reasoning}, q)$ is much higher than $P(\text{correct answer} \mid q)$ directly because the intermediate tokens carry computation forward. A transformer with $L$ layers and $d$ dimensions has $O(L \cdot d^2)$ computation per token — by generating $T$ reasoning tokens, the effective computation increases to $O(T \cdot L \cdot d^2)$, allowing the model to solve problems that exceed a single forward pass's capacity.

Appending 'Let's think step by step' shifts the model's output distribution from $P(y \mid x)$ to $P(y \mid x, \text{'step by step'})$. This activates patterns from the pretraining data where step-by-step reasoning preceded correct answers (textbooks, tutorials, worked examples). The prompt acts as a retrieval cue: $P(\text{reasoning pattern} \mid \text{'step by step'})$ is high because this phrase co-occurred with structured reasoning in millions of training documents. Zero-shot CoT is less reliable than few-shot CoT because the model must infer both the reasoning format and the answer — two sources of uncertainty versus one.

Few-shot CoT provides explicit demonstrations of the reasoning process: $(q_1, r_1, a_1), \ldots, (q_k, r_k, a_k)$ where $r_i$ is the reasoning chain. The model learns to map questions to reasoning patterns via in-context learning: $P(r \mid q, \{(q_i, r_i)\})$. The quality of reasoning chains matters more than their correctness: even reasoning chains with wrong final answers improve performance if the intermediate steps are logically structured. This suggests CoT teaches the model a reasoning format (the 'how') more than factual knowledge (the 'what'). The improvement is most dramatic on multi-step problems where the number of reasoning steps $>$ 3.

Self-consistency samples $N$ independent reasoning chains and takes a majority vote: $a^* = \arg\max_a \sum_{i=1}^N \mathbb{1}[a_i = a]$. If each chain has accuracy $p > 0.5$, the majority vote accuracy is $P(\text{correct}) = \sum_{k=\lceil N/2 \rceil}^N \binom{N}{k} p^k (1-p)^{N-k}$, which converges to 1 as $N \to \infty$ by the law of large numbers. For $p = 0.6$ and $N = 10$: majority vote accuracy rises to ~83%. The key assumption is that errors are independent across chains — using temperature sampling ($T > 0$) ensures diversity in reasoning paths, approximating independence.

The system prompt occupies the first position in the context window: $[\text{SYS}] \oplus s \oplus [\text{/SYS}] \oplus [\text{USER}] \oplus u \oplus [\text{ASST}]$. Due to the causal attention mask, every generated token attends to the system prompt, making it a persistent bias on the entire output distribution. The influence of the system prompt on token $t$ is approximately $\alpha_t(s) = \text{softmax}(Q_t \cdot K_s^\top / \sqrt{d})$ — the attention weight from position $t$ to the system prompt tokens. In early layers, $\alpha_t(s)$ is high (system prompt provides the behavioral context); in later layers, recent tokens dominate.

Persona engineering conditions the model on a role description that activates a specific region of the learned output distribution. Formally, the persona $p$ defines a subspace of outputs $\mathcal{Y}_p \subset \mathcal{Y}$ where the model concentrates probability mass: $P(y \in \mathcal{Y}_p \mid p) \gg P(y \in \mathcal{Y}_p)$. The effectiveness scales with specificity: 'You are a helpful assistant' provides weak conditioning (broad $\mathcal{Y}_p$), while 'You are a board-certified oncologist specializing in lung cancer immunotherapy' provides strong conditioning (narrow $\mathcal{Y}_p$). The cross-entropy between persona-conditioned and unconditioned distributions measures persona strength: $H(P(\cdot \mid p), P(\cdot))$.

Safety guardrails define a rejection region $\mathcal{X}_{\text{reject}} \subset \mathcal{X}$ where the model should refuse to comply. The challenge is that $\mathcal{X}_{\text{reject}}$ is defined in natural language (e.g., 'harmful content') and has fuzzy boundaries. In the model's representation space, the decision boundary between compliance and refusal is a hyperplane $w^\top h(x) + b = 0$ where $h(x)$ is the hidden representation. Adversarial attacks find inputs near this boundary: $x^* = x + \delta$ where $\|\delta\|$ is small but $w^\top h(x^*) + b$ changes sign. The robustness of guardrails depends on the margin $\min_{x \in \mathcal{X}_{\text{reject}}} |w^\top h(x) + b| / \|w\|$.

System prompt optimization searches over the space of natural language instructions to maximize a quality metric $Q(s) = \mathbb{E}_{(x,y) \sim \mathcal{D}}[\text{score}(\pi(x; s), y)]$ where $s$ is the system prompt, $\pi(x; s)$ is the model's output given prompt $x$ and system prompt $s$, and score measures quality. This is a black-box optimization problem in a discrete (text) space. Methods: APE (Automatic Prompt Engineer) uses the LLM itself to propose prompt candidates, evaluates them on a validation set, and iterates. The search space is exponential in prompt length, so heuristic methods (beam search over prompt variations, evolutionary strategies) are necessary.

JSON mode constrains the model's output to valid JSON by modifying the sampling process at each step: $P'(t_i \mid t_{<i}) \propto P(t_i \mid t_{<i}) \cdot \mathbb{1}[t_{<i} \oplus t_i \text{ is a valid JSON prefix}]$. This is context-free grammar (CFG) guided decoding where the grammar is the JSON specification. At each token position, only tokens that maintain valid JSON are allowed — the valid set depends on the parser state (e.g., after '"key":', only '"', number, '{', '[', true, false, null are valid starts). The constraint never changes the relative probabilities of valid tokens — only invalid tokens are zeroed out.

JSON Schema defines a type system over the output space: each field has a type ($\text{string}$, $\text{number}$, $\text{array}$, etc.), constraints ($\text{minLength}$, $\text{enum}$, $\text{pattern}$), and optionality. The schema partitions the output space $\mathcal{Y}$ into valid $\mathcal{Y}_v$ and invalid $\mathcal{Y}_i$ sets. The model's effective output distribution becomes $P'(y) = P(y \mid y \in \mathcal{Y}_v) = P(y) / P(\mathcal{Y}_v)$ for $y \in \mathcal{Y}_v$. The renormalization factor $1/P(\mathcal{Y}_v)$ measures how much the schema constrains the output: if $P(\mathcal{Y}_v) = 0.01$, the model must concentrate 100$\times$ more probability mass on valid outputs.

Constrained decoding implements a finite-state machine (FSM) or pushdown automaton (PDA) that tracks the parser state during generation. At state $q$, the set of valid next tokens is $\Sigma(q) \subseteq \mathcal{V}$. The modified distribution is $P'(t \mid q) = P(t) \cdot \mathbb{1}[t \in \Sigma(q)] / Z(q)$ where $Z(q) = \sum_{t \in \Sigma(q)} P(t)$. For JSON, the PDA has states for: object-start, key, colon, value, comma, array, nested structures. The constraint eliminates ~99%+ of the vocabulary at each step (e.g., after '{', only '"' and '}' are valid), dramatically reducing effective perplexity.

Even with constrained decoding, semantic errors can occur: the JSON is syntactically valid but semantically wrong (e.g., age = -5, email = 'not-an-email'). Post-generation validation applies the full schema including semantic constraints: $\text{valid}(y) = \text{syntax}(y) \wedge \text{types}(y) \wedge \text{constraints}(y)$. Retry strategies on validation failure use the error message as feedback: $y' \sim \pi(\cdot \mid x, y, \text{error})$, giving the model a chance to self-correct. The expected number of retries before success is $1/p_v$ where $p_v$ is the per-attempt success probability — for well-prompted models, $p_v > 0.9$ and rarely more than 2 attempts are needed.