Chapter 6: Recognition

A predicted box is rarely pixel-perfect, so we need a graded measure of how well it matches the ground truth rather than a yes/no test. IoU, also called the Jaccard index, divides the area where two boxes overlap by the area they jointly cover: $|A \cap B| / |A \cup B|$. It is 0 when the boxes are disjoint and 1 only when they coincide exactly, and it is scale-invariant, so a small box and a large box are judged by the same standard. Detection benchmarks turn this continuous score into a correctness threshold — PASCAL VOC accepts a detection at IoU $\geq 0.5$, COCO demands $\geq 0.75$. Variants like GIoU add penalty terms so the measure still gives useful gradients when boxes do not overlap at all.

A detector typically fires many boxes on a single object because neighbouring anchors all latch onto it, so raw output is cluttered with duplicates. NMS is the greedy cleanup: sort all boxes by confidence, take the highest-scoring one as a true detection, discard every remaining box whose IoU with it exceeds a threshold (commonly 0.5), and repeat on what survives. The IoU from the previous concept is exactly the overlap test used here. The threshold trades errors: too low and it deletes genuinely separate but nearby objects, too high and duplicates leak through. Soft-NMS softens the hard deletion by decaying scores instead, and class-specific NMS only suppresses boxes of the same label so a person standing in front of a car is not erased.

Objects appear at unpredictable positions, sizes, and aspect ratios, so a network cannot simply regress a fixed number of boxes. Anchor boxes solve this by tiling the image with a predefined menu of reference boxes $(a_x, a_y, a_w, a_h)$ at every spatial location; the network only predicts small offsets $(t_x, t_y, t_w, t_h)$ that nudge an anchor toward a real object. The center offset passes through a sigmoid $\sigma$ so it stays inside the cell, while width and height are scaled by $e^{t_w}$ and $e^{t_h}$, guaranteeing positive sizes. Anchor shapes are usually chosen by k-means clustering on the training boxes. The design's weakness is that most anchors match no object, creating heavy foreground-background imbalance, which is why anchor-free detectors like FCOS and CenterNet were later proposed.

A detector produces a ranked list of boxes at every confidence level, so no single accuracy number captures its quality; we need a metric that summarizes the whole precision-recall trade-off. Average precision (AP) for one class is the area under its precision-recall curve, obtained by sweeping the confidence threshold from high to low. Mean average precision averages AP across all classes: $\text{mAP} = \frac{1}{|C|}\sum_{c} \text{AP}_c$. Whether a detection counts as a true positive depends on the IoU threshold from the first concept — PASCAL VOC uses [email protected], while COCO averages over ten thresholds from 0.5 to 0.95 to reward tight localization. It is the number leaderboards rank on, and reporting AP separately for small, medium, and large objects exposes where a detector struggles.

Classification emits one label per image and detection one label per box, but segmentation asks the harder question: what class is each individual pixel? The output is a full $H \times W$ grid of labels, formally $\hat{y}_{i,j} = \arg\max_k P(y_{i,j}=k \mid \mathbf{x})$ where $(i,j)$ indexes a pixel and $k$ ranges over classes. Training minimizes cross-entropy — the same loss from the classification topic — but averaged over every pixel rather than once per image. The dominant failure mode is class imbalance: background pixels can outnumber a thin object like a pole by a hundred to one, so a network can score high accuracy while ignoring the rare class entirely. This is why mean IoU, not accuracy, is the reported metric.

There is a tension at the heart of segmentation: recognizing what an object is requires a large receptive field and abstract features, which pooling provides by shrinking the image, yet labeling pixels requires the full original resolution that pooling destroys. The encoder-decoder resolves this in two halves. The encoder $E$ is a classification-style CNN that downsamples, distilling what is present; the decoder $D$ then upsamples back to input resolution, reconstructing where each thing is. Skip connections, the U-Net innovation, forward high-resolution encoder features directly to the decoder so fine boundaries lost during downsampling are restored. Without them, masks come out blurry and boundaries smear. The encoder answers 'what', the decoder answers 'where', and skip connections stitch the two answers together.

The decoder needs to increase spatial resolution, and while simple interpolation works, we would rather the network learn how to upsample so it can synthesize sharp boundaries. Transposed convolution (sometimes called deconvolution) does this by conceptually inserting zeros between input elements and then convolving, with the output size given by $(\text{in}-1)\times s + k - 2p$ for stride $s$, kernel $k$, and padding $p$. A stride of 2 doubles each dimension. Because the upsampling weights are learned, the layer can produce crisper edges than fixed bilinear interpolation. Its notorious failure is the checkerboard artifact: when the kernel size is not divisible by the stride, output positions receive uneven numbers of contributions, leaving a grid pattern. Practitioners often avoid this with bilinear upsampling followed by an ordinary convolution.

Pixel-wise cross-entropy struggles when a class is tiny, because a network can drive the loss down simply by predicting 'background' everywhere and still be right 99% of the time. Dice loss attacks this directly by optimizing overlap rather than per-pixel correctness: $\mathcal{L}_{Dice} = 1 - \frac{2|P \cap G|}{|P| + |G|}$, where $P$ is the predicted mask (soft probabilities) and $G$ the ground-truth mask. Because it normalizes by the total foreground area of both masks, a small object counts as much as a large one — solving the imbalance that defeats accuracy and plain cross-entropy. It relates directly to IoU from the detection topic; both measure overlap, but Dice weights the intersection twice. In practice it is usually summed with cross-entropy, which supplies smoother early-training gradients.

A per-person classifier fails the moment a new identity appears, so face recognition instead learns an embedding: a network $f_\theta$ maps a face image $\mathbf{x}$ to a vector $\mathbf{e} \in \mathbb{R}^{128}$ (or 512) that is L2-normalized to unit length, placing every face on a hypersphere. On that sphere, similar faces cluster and different people separate, so identity becomes a question of distance rather than of a fixed label — exactly the flexibility a fixed-output softmax classifier from the first topic lacks. Normalization matters because it makes Euclidean distance and cosine similarity equivalent, so a threshold learned on one image pair transfers to all others. The embedding is only as good as its training loss, which the next two concepts supply.

To shape the embedding we do not tell the network absolute coordinates — only relative facts: this anchor is the same person as that positive and a different one from this negative. Triplet loss encodes that directly as $\mathcal{L} = \max(0, \|f(a)-f(p)\|^2 - \|f(a)-f(n)\|^2 + \alpha)$, using the embedding $f$ from the previous concept. It stays zero as long as the negative is farther than the positive by at least the margin $\alpha$ (typically 0.2), and turns on only when that ordering is violated or too tight. The catch is that most random triplets already satisfy the margin and produce no gradient, so training stalls unless hard negative mining is used to select the difficult, informative triplets that actually teach the network.

Triplet loss works but is finicky, needing careful triplet mining. ArcFace takes a different route: keep the softmax-style classification over training identities, but impose a geometric margin. It measures the angle $\theta_{y_i}$ between an embedding and its class-center weight vector, adds a fixed margin $m$ (about 0.5 radians) to that angle before the cosine, and scales by $s$ (30–64). The loss $-\log \frac{e^{s\cos(\theta_{y_i}+m)}}{e^{s\cos(\theta_{y_i}+m)} + \sum_{j\neq y_i} e^{s\cos\theta_j}}$ therefore demands the correct class beat the others by a constant angular gap. Because the margin lives in angle space, it corresponds to a fixed geodesic distance on the same hypersphere the embedding concept introduced, producing tighter within-person clusters and cleaner between-person gaps than triplet loss, with far simpler batching.

Once embeddings exist, they are queried in two distinct ways with different costs and error profiles. Verification (1:1) asks 'are these two faces the same person?' — compute one distance $d(e_1, e_2)$ and accept if it falls below a threshold $\tau$; this is unlocking a phone. Identification (1:N) asks 'who is this?' — compare the query against a gallery of $N$ enrolled identities and return the nearest, or 'unknown' if even the closest exceeds $\tau$; this is a watchlist search. Verification is $O(1)$ per query while brute-force identification is $O(N)$, so large galleries use approximate nearest-neighbour indexes. Both lean on the metric structure from the embedding concept, and the threshold $\tau$ trades false accepts against false rejects along a ROC curve.