Chapter 7: Prompt Engineering

Prompt specificity reduces output entropy: a vague prompt $x_v$ gives $H(Y \mid x_v) \gg H(Y \mid x_s)$ for a specific prompt $x_s$, where $H$ is the conditional entropy of the model's output distribution. Each constraint in the prompt eliminates a fraction of the output space: with $k$ independent constraints each reducing options by half, the remaining space is $2^{-k}$ of the original. In practice, constraints are correlated, so the reduction follows $|\mathcal{Y}_{\text{constrained}}| \approx |\mathcal{Y}| \cdot e^{-\alpha k}$ where $\alpha < \ln 2$ is the average constraint strength.

Role assignment shifts the model's conditional distribution from $P(y \mid x)$ to $P(y \mid \text{role}, x)$. By Bayes' rule, this is proportional to $P(\text{role} \mid y, x) \cdot P(y \mid x)$ — the model up-weights responses that are consistent with the assigned role. The effectiveness depends on how well the role is represented in pretraining data: 'expert cardiologist' activates patterns from medical texts (high-quality training signal), while 'alien from planet Zorg' has no grounding and mainly affects style. The expected quality improvement is proportional to $D_{\text{KL}}(P(y \mid \text{role}, x) \| P(y \mid x))$ — larger distributional shifts indicate the role is having more effect.

Decomposing a task into $k$ sequential steps reduces the effective complexity from $O(C^k)$ to $O(k \cdot C)$ where $C$ is the complexity per step. This is because the model processes one step at a time, each with bounded context. The error probability also changes: for independent steps with per-step error $\epsilon$, the overall success probability is $(1 - \epsilon)^k \approx 1 - k\epsilon$ for small $\epsilon$. With $k = 5$ steps and $\epsilon = 0.05$ per step, overall success is ~77%. Without decomposition, the single-step error for the full task might be 50%+ because the model must implicitly manage all subtasks simultaneously.

Negative instructions ('do not X') constrain the output distribution by zeroing out probability mass on undesired outputs: $P'(y) \propto P(y) \cdot \mathbb{1}[y \notin \mathcal{Y}_{\text{forbidden}}]$. The effectiveness depends on how much probability mass the model originally placed on forbidden outputs. If $P(y \in \mathcal{Y}_{\text{forbidden}}) = p$, then after the constraint, the remaining outputs are renormalized by $1/(1-p)$. A common failure mode: the constraint is too vague ("don't be verbose") so $\mathcal{Y}_{\text{forbidden}}$ is not well-defined, and the model cannot reliably exclude those outputs.