Scale-Invariant Depth Loss

Depth networks for monocular images cannot know the absolute distance to objects, only their relative depths up to a global scale factor. Using a standard $L_2$ loss on raw depth values forces the model to match absolute scale, which is ill-posed and varies across datasets and cameras. A scale-invariant loss instead measures error in log-depth and explicitly removes any constant offset in log-space, so the network is trained to get shape/relative depth right while being free to choose an overall scale. The given loss does this by combining the mean squared log-error with a correction term based on the mean log-error across all valid pixels.

Mathematically, if $d_i = \log$\hat{Z}$_i - \log Z_i$, then adding a constant $c$ to all $\log$\hat{Z}$_i$ just adds $c$ to all $d_i$; the second term in the loss penalizes this global offset, controlled by $\lambda \in [0,1]$. When $\lambda = 1$, the loss is fully scale-invariant: any constant multiplicative scaling of $\hat{Z}$ leaves $L$ unchanged, which aligns with the intrinsic ambiguity of monocular depth. In code, implementing this is mainly a matter of carefully computing $d_i$, then the mean of $d_i^2$ and the square of the sum of $d_i$, over only the valid pixels.

---

1. Background Knowledge

• Monocular depth ambiguity From a single RGB image, many 3D scenes can project to the same image if you scale all depths by a constant factor; the camera sees the same 2D picture. So absolute scale is unobservable; only relative depths (who is in front of whom, relative distances) are reliably learnable.

• Why log-depth and log-error Depth values can span orders of magnitude; using $\log Z$:

• turns multiplicative errors in $Z$ into additive errors in $\log Z$ (good for scale),

• reduces sensitivity to large absolute depths,

• naturally interacts with scale: multiplying depth by $s$ corresponds to adding $\log s$ to log-depth.

• Scale-invariant loss form Let $d_i = \log$\hat{Z}$_i - \log Z_i$. The loss

$$L = \frac{1}{n}\sum_i d_i^2 - \frac{\lambda}{n^2}\left(\sum_i d_i\right)^2$$

is essentially:

• first term: mean squared error in log-depth,
• second term: subtracts a term depending on the mean error, removing sensitivity to a global offset in log-depth (global scale in depth).

---

2. Algorithm / Approach

At a high level, to compute this loss for a batch:

1. Select valid pixels: Filter out invalid depths (e.g., zeros, negatives, NaNs, masked regions).
2. Work in log-space: Compute $\log$\hat{Z}$_i$ and $\log Z_i$ on valid pixels.
3. Compute per-pixel log-error: $d_i = \log$\hat{Z}$_i - \log Z_i$.
4. Aggregate statistics:

• mean squared error term: $\frac{1}{n}$\sum$_i d_i^2$,
• mean error term: \left(\sum_i d_i\right)^2 / n^2.

5. Combine with $\lambda$: return

$$L = \text{mean}(d^2) - \lambda \cdot (\text{mean}(d))^2$$

which is equivalent to the given formula.

You typically implement this in a vectorized way over all pixels (and possibly over the batch).

---

3. Step-by-Step Strategy

Assume tensor library like PyTorch, with predicted depths pred and ground truth target of shape (B, H, W) or (B, 1, H, W):

1. Create a validity mask

• Valid if ground truth is positive and finite (and optionally pred is positive):

valid = (target > 0) & torch.isfinite(target)

• Optionally incorporate any existing mask from the dataset.

2. Extract valid predictions and targets

pred_valid = pred[valid]
target_valid = target[valid]

Option: if no valid pixels for a sample, decide to skip or return zero loss.

3. Compute log-depths with numerical safety

• Clamp to avoid log(0):

eps = 1e-6
log_pred = torch.log(pred_valid.clamp(min=eps))
log_target = torch.log(target_valid.clamp(min=eps))

4. Compute log-errors $d_i$

d = log_pred - log_target # shape: (n_valid,)
n = d.numel()

5. Compute the two terms

term1 = (d ** 2).mean() # (1/n) * sum d_i^2
mean_d = d.mean() # (1/n) * sum d_i
term2 = mean_d ** 2 # (1/n^2) * (sum d_i)^2

6. Combine with $\lambda$

loss = term1 - lambda_ * term2

Where lambda_ is in [0, 1]. For fully scale-invariant, set lambda_ = 1.0.

7. Batch handling (optional refinement)

• You can compute per-image losses then average across batch:
• reshape each image, mask per image, compute its own term1 and term2.
• Or treat all valid pixels in the batch as one big set; both are common.

---

4. Common Pitfalls

• Forgetting to mask invalid pixels Including zeros, missing depth, or invalid values in log will produce -inf or nan and break training. Always mask and/or clamp.

• Taking log of non-positive values Predictions can briefly go to zero or negative. Either:

• enforce positive outputs via activation (e.g., softplus, exp),

• clamp before log with a small eps as shown.

• Mis-implementing the second term The second term is square of the sum divided by $n^2$, i.e. $($\text{mean}$(d))^2$, not mean(d**2) (which is the first term). Many off-by-one mistakes come from mixing those.

• Ignoring per-image vs per-batch semantics If dataset scales vary across images, computing mean over the entire batch may blend different scales. Often better to compute loss per image (using its own n) then average across batch.

• Unstable gradients if $\lambda$ too high with noisy data While $\lambda = 1$ is fully scale-invariant, in some setups people choose $\lambda < 1$ to trade off absolute scale vs. pure relative accuracy. If your targets are already normalized to a consistent scale, a smaller $\lambda$ might train more stably.

---

5. Time & Space Complexity

Let $N$ be the total number of pixels (or valid pixels) processed in a batch.

• Time complexity

• All operations are elementwise or simple reductions (sum, mean).

• Computing logs, differences, squares, sums: $O(N)$ total.

• Space complexity

• You store:

• predicted depths and targets: $O(N)$,

• intermediate tensors (log_pred, log_target, d): also $O(N)$.

• Overall additional memory overhead beyond inputs: $O(N)$.

This loss is therefore linear in the number of pixels and inexpensive compared to the forward/backward pass of the depth network itself.