Chapter 5: Depth Estimation
Master the techniques for recovering depth from images and sensors: monocular depth estimation with deep networks, stereo and multi-view approaches, active sensing technologies like LiDAR and structured light, and the metrics used to evaluate depth quality. Understand the trade-offs between passive and active methods, and learn how to refine sparse depth into dense, accurate maps.
- Topics
- 7
- Demos
- 2
- Videos
- 3
- Min read
- 40
Chapter Overview
Depth estimation is the problem of assigning a distance value to every pixel or point in a scene. It is a cornerstone of 3D computer vision because depth bridges the gap between 2D images and 3D geometry---once you have depth, you can reconstruct surfaces, measure distances, and reason about spatial relationships.
What is this chapter about? We cover the full spectrum of depth estimation methods. We begin with monocular depth estimation, where a single RGB image is used to predict depth using learned cues. We then examine state-of-the-art architectures like MiDaS and DPT that use vision transformers for robust relative depth. Multi-view stereo extends depth estimation to dense reconstruction from many calibrated images. On the hardware side, we explore active sensors---LiDAR, time-of-flight cameras, and structured light systems---that directly measure depth. Finally, we address depth completion (filling in sparse measurements) and the evaluation metrics used to benchmark all these methods.
Why does this matter? Autonomous vehicles need dense, accurate depth maps for obstacle avoidance. AR/VR applications require real-time depth for scene understanding and occlusion handling. Robotics depends on depth for manipulation and navigation. Every downstream 3D task---reconstruction, detection, segmentation---benefits from better depth estimation.
How the topics connect: We start with monocular depth as the most accessible entry point (just one image), then explore MiDaS/DPT as the leading architectures. Multi-view stereo provides denser and more accurate depth when multiple images are available. Active sensors (LiDAR, ToF, structured light) offer direct physical measurements. Depth completion bridges sparse sensor data with dense predictions. We close with evaluation metrics that let you quantitatively compare all these approaches.
Chapter Roadmap
Click any topic to jump in
Monocular Depth
Predicting dense depth from a single RGB image using learned monocular cues and encoder-decoder networks.
Learned architectures and geometric approaches
MiDaS & DPT
State-of-the-art architectures that combine mixed-dataset training with Vision Transformers for zero-shot depth.
Multi-View Stereo
Dense depth from multiple calibrated images using photometric consistency and plane-sweep volumes.
Active sensing — direct measurement without learned inference
LiDAR & ToF
Active sensors that directly measure depth using time-of-flight or spinning laser principles.
Structured Light
Projecting coded patterns onto scenes and recovering depth from pattern deformation via triangulation.
Sparse-to-dense refinement and benchmarking
Depth Completion
Filling sparse sensor depth into dense maps using guided upsampling and self-supervised refinement.
Evaluation Metrics
Quantitative metrics for depth quality — absolute error, threshold accuracy, and scale-invariant measures.
Topics
7 topics in this chapter
Chapter 4 recovered depth by triangulating the same point seen from several cameras. A phone camera, a dashcam or an old photograph gives you exactly one view, and one view throws away the third dimension: every pixel is the image of an entire ray, and a toy car close up and a real car far away can produce the same picture. Geometry alone cannot undo that. Yet people judge depth from one eye reasonably well, because scenes carry statistical cues: objects resting lower in the frame are usually nearer, texture gets denser with distance, nearer things occlude farther ones, and familiar objects have familiar sizes. Monocular depth estimation trains a network to learn those cues from data and output a depth value for every pixel. This topic sets up the problem: why the answer is only defined up to scale, how encoder-decoder networks produce dense maps, and how self-supervised training on stereo pairs or video replaces expensive ground-truth depth with a photometric reprojection loss. The next topic scales the idea up with MiDaS and DPT, which train on many datasets at once.
Definition
Monocular depth estimation predicts a dense depth map from a single RGB image using learned priors, since one view does not determine depth geometrically. Without a metric reference, the output is defined only up to an unknown scale (and often shift), so models predict either relative depth or metric depth calibrated on known camera and scene statistics.
In this topic
- 1The Monocular Depth Problem
- 2Encoder-Decoder Architecture for Depth
- 3Scale Ambiguity and Self-Supervised Training
The Monocular Depth Problem
Given a single image, monocular depth estimation assigns a depth value to every pixel. The network must learn monocular cues: nearby objects appear larger and lower in the image, texture density increases with distance, edges and occlusion boundaries indicate depth discontinuities, and atmospheric haze reduces contrast at range. The output is either a relative depth map (ordinal relationships only) or a metric depth map (absolute distances in meters). Scale ambiguity is the fundamental challenge---without knowing the physical size of objects in the scene, the absolute scale of depth is unrecoverable from a single image alone.
Monocular depth is fundamentally ill-posed: the projection loses one dimension, so infinitely many depth maps satisfy for a given image . The network learns a prior over plausible depth maps. Scale ambiguity means the likelihood is invariant under global scaling — the loss landscape has a degenerate direction that only metric supervision or known reference objects can break.
A monocular depth model predicts relative depth values ranging from 0.1 to 0.9 for an image. If you know a reference object in the scene is 3 meters away and its predicted depth value is 0.4, how would you convert the entire map to metric depth?
Encoder-Decoder Architecture for Depth
Most monocular depth networks follow an encoder-decoder design. The encoder (often a pretrained ResNet, EfficientNet, or Vision Transformer) extracts multi-scale feature maps from the input image, progressively reducing spatial resolution while increasing channel depth. The decoder upsamples these features back to the original resolution, combining high-level semantic information from deep layers with fine-grained spatial details from skip connections. The final output is a single-channel depth map where each pixel holds a predicted depth value. Loss functions typically combine L1 or L2 pixel-wise depth error with scale-invariant loss and gradient-matching loss to preserve sharp edges.
The encoder-decoder computes a function via successive downsampling (each stage halves resolution: ) and upsampling with skip connections. The receptive field at the bottleneck covers the entire image, giving global context, while skip connections preserve gradients at image edges — without them, the decoder must reconstruct spatial detail from a resolution bottleneck, causing blurry depth boundaries.
An encoder produces feature maps at resolutions 1/4, 1/8, 1/16, and 1/32 of the input. The decoder upsamples by 2x at each stage. If the input is 640x480, what is the resolution of the depth map before the final upsampling to full resolution?
Scale Ambiguity and Self-Supervised Training
Scale ambiguity means that a single image cannot determine whether a scene is 1 meter or 100 meters deep---both produce identical images. Supervised methods sidestep this by training on datasets with ground-truth depth from LiDAR or structured light. Self-supervised methods (like Monodepth2) train without ground-truth depth by using stereo image pairs or video sequences. The key insight is photometric consistency: if the predicted depth and camera motion are correct, warping one image to another viewpoint should reproduce the target image. The reprojection error between the warped and actual image serves as the training signal.
The photometric reprojection loss warps source pixels to the target view: . This involves differentiable bilinear sampling, making the loss a function of and pose . The scale ambiguity manifests algebraically: replacing and produces identical reprojection coordinates, so the network can only recover depth up to an unknown scale factor .
A self-supervised depth model predicts depth and camera motion between two video frames. The photometric reprojection loss is . If the predicted depth is twice the true depth and is also scaled by 2x, does the loss change?
Related Problems on PixelBank
The previous topic trained a depth network on one dataset and hit a wall: a model trained on KITTI driving scenes falls apart on indoor photos, and no single dataset covers the world. Bigger datasets exist, but they disagree on units. Stereo movies give disparity with an unknown baseline, structure from motion gives depth up to scale, laser scanners give metres, and each has its own range and missing regions, so their losses cannot simply be added. MiDaS solved this by making the loss blind to scale and shift: before comparing, the prediction is aligned to each ground truth by a least-squares fit, so any dataset with correct relative depth can join the training mix. DPT then swapped the convolutional encoder for a Vision Transformer, whose self-attention relates distant image regions from the first layer, giving globally consistent depth. This topic covers the mixing loss, the DPT architecture, and the difference between relative and metric depth, including why a relative model needs a reference before it can measure anything. Depth Anything later scaled the same recipe with tens of millions of pseudo-labelled images.
Definition
MiDaS is a monocular depth model trained on a mixture of datasets with a scale- and shift-invariant loss in inverse-depth space, producing relative depth that transfers zero-shot to new domains. DPT (Dense Prediction Transformer) is its transformer backbone: ViT tokens from several layers are reassembled into image-like feature maps and fused by a convolutional decoder into a dense prediction.
In this topic
- 1MiDaS: Mixing Datasets for Robust Depth
- 2DPT: Dense Prediction with Vision Transformers
- 3Relative vs. Metric Depth and Zero-Shot Transfer
MiDaS: Mixing Datasets for Robust Depth
MiDaS achieves generalization by training on multiple depth datasets simultaneously, each with different depth formats (stereo disparity, LiDAR, SfM). The key innovation is a scale- and shift-invariant loss: instead of comparing absolute depth values, MiDaS aligns predictions to ground truth using a least-squares fit of scale and shift before computing the error. This allows mixing datasets with incompatible depth scales. The result is a model that produces high-quality relative depth on arbitrary images without fine-tuning, because it has seen diverse indoor, outdoor, synthetic, and real scenes during training.
MiDaS trains on a mixture of datasets with different depth scales by using a scale-and-shift-invariant loss: where with chosen per sample to minimize the loss. This linear alignment removes dataset-specific scale and offset, forcing the network to learn only relative depth structure — the invariant geometric relationships between scene elements.
MiDaS predicted depths [0.2, 0.5, 0.8] for three pixels. The ground truth depths are [2.0, 5.0, 8.0]. Find the optimal scale and shift to align predictions: .
DPT: Dense Prediction with Vision Transformers
DPT replaces the convolutional encoder with a Vision Transformer (ViT). The image is split into 16x16 patches, each embedded as a token. Self-attention across all tokens gives every patch a global receptive field from the very first layer, unlike CNNs that build receptive fields gradually through stacking. DPT extracts feature maps at multiple stages of the transformer and feeds them into a convolutional decoder (Reassemble + Fusion blocks) that reconstructs spatial resolution. This global context awareness allows DPT to produce depth maps with better long-range consistency---a wall 10 meters away is correctly flat even if it spans the entire image.
DPT replaces the CNN encoder with a Vision Transformer, where self-attention computes over all patch tokens simultaneously. Every patch attends to every other patch with complexity, giving global receptive field at every layer — a CNN needs the full depth of the network to achieve this. For depth estimation, this means a nearby object can immediately influence the predicted depth of a far-away region through long-range spatial dependencies.
A DPT model uses ViT-Base with patch size 16 on a 384x384 input. How many tokens are produced? If self-attention has complexity , how does this compare to a 640x480 input?
Relative vs. Metric Depth and Zero-Shot Transfer
Relative (affine-invariant) depth preserves ordering and ratios but not absolute scale---pixel A is 2x farther than pixel B, but neither has a metric value. Metric depth provides absolute distances in meters or feet. MiDaS and early DPT produce relative depth, which transfers well across domains but cannot be used directly for measurement. Recent models like ZoeDepth and Metric3D add metric heads trained on specific domains, predicting both relative structure and absolute scale. Zero-shot transfer refers to applying a model to images from domains it was never trained on (e.g., underwater, medical, aerial) and still getting reasonable results, which MiDaS achieves through its multi-dataset training strategy.
Zero-shot transfer works because MiDaS learns relative depth invariants that are scale-independent. Given training datasets with different depth distributions , the model must find features that predict ordinal relationships consistently across all distributions. This forces the representation to encode scene geometry rather than dataset-specific statistics. At test time on unseen domains, the ordinal predictions remain valid because the underlying geometric cues (occlusion, perspective, size) are universal.
A drone hovers 50 m above flat ground and looks straight down. MiDaS outputs relative inverse depth (larger means closer): 0.2 on the ground and 0.5 on a building roof. Can you recover the building's height from the known 50 m altitude alone?
Related Problems on PixelBank
Monocular networks guess depth from learned priors, and the previous topic showed that their output is relative at best. When several calibrated images of a scene exist, guessing is unnecessary: the structure-from-motion pipeline of Chapter 4 already gives every camera's pose, but only a sparse cloud of a few thousand matched features. Multi-view stereo turns those poses into dense geometry. For each pixel of a reference image it asks which depth makes the pixel look the same when projected into every neighbouring view, the photo-consistency test. Doing that by brute force means checking every depth hypothesis at every pixel, which is expensive in memory and time, and it fails on textureless walls, reflections and occlusions where appearance does not identify the point. This topic covers the three main pieces: plane-sweep cost volumes that test a fixed set of depths, PatchMatch propagation that searches cleverly instead of exhaustively (and its learned version, PatchmatchNet), and depth-map fusion that keeps only depths several views agree on. These per-view depth maps are what Chapter 6 turns into meshes.
Definition
Multi-view stereo (MVS) estimates dense depth or geometry from several images with known intrinsics and poses. For each reference pixel it selects the depth whose projection into neighbouring views is most photo-consistent, using plane sweeping or PatchMatch search, then fuses per-view depth maps into a point cloud by keeping depths that are geometrically consistent across views.
In this topic
- 1Plane-Sweep Volumes
- 2PatchMatchNet and Learned MVS
- 3Depth Fusion from Multiple Views
Plane-Sweep Volumes
The plane-sweep algorithm evaluates depth hypotheses by sweeping a set of fronto-parallel planes through the scene at regular depth intervals. For each depth hypothesis, all source images are warped to the reference view using the known camera parameters and the assumed depth. The warped images are then compared to the reference image using a cost metric (e.g., absolute difference, normalized cross-correlation). The depth with the lowest cost at each pixel is selected as the final depth estimate. This creates a 3D cost volume of dimensions , where is the number of depth hypotheses.
Plane-sweep volumes evaluate depth hypotheses by warping source images to the reference view at each depth: where is a photometric similarity metric. The cost volume encodes matching confidence at each pixel and depth. The optimal depth minimizes the cost: , which can be solved independently per pixel or regularized spatially.
An MVS system uses 64 depth hypotheses from 0.5m to 10m with 3 source views. The images are 640x480. How large is the cost volume in memory (float32)?
PatchMatchNet and Learned MVS
PatchMatch-based MVS replaces exhaustive plane-sweep search with an iterative scheme: each pixel starts with a random depth hypothesis, then propagates good hypotheses to neighbors and refines them. This is orders of magnitude faster than evaluating all depth hypotheses at every pixel. PatchMatchNet combines this idea with deep learning: a feature extraction network produces discriminative features for matching, and a differentiable PatchMatch module iteratively updates depth hypotheses using learned cost functions. The result is both fast (sparse evaluation) and accurate (learned features handle textureless regions and reflections).
PatchMatch rests on two facts. Random initialization is cheap but useful: with depth bins, a large planar region of pixels almost surely contains a pixel whose random guess is close, because the chance that none does is for a tolerance of bins. Propagation then spreads that guess, since neighbouring pixels on one surface share nearly the same depth or plane. Each iteration therefore costs a constant number of evaluations per pixel instead of . The weakness is the same assumption: across depth discontinuities and on textureless surfaces, neighbours are poor predictors and the cost cannot tell hypotheses apart.
A PatchMatch MVS method uses 3 iterations with 8 depth samples per iteration per pixel, on a 1920x1080 image. Compare the total evaluations to exhaustive search with 256 depth hypotheses.
Depth Fusion from Multiple Views
Each reference image in an MVS pipeline produces its own depth map. These per-view depth maps must be fused into a single consistent 3D representation. Depth fusion involves reprojecting each depth map into 3D, checking geometric consistency (a depth estimate is kept only if it agrees with depth maps from other views within a threshold), and merging consistent points. Common fusion approaches include volumetric TSDF fusion (accumulating signed distances on a voxel grid) and point-based fusion (filtering and merging point clouds). The consistency check rejects outliers from occluded regions, textureless areas, and matching failures.
Depth fusion from views uses geometric consistency: a depth at pixel in view is consistent with view if the reprojected depth agrees within tolerance . Points passing consistency checks in views are retained. This voting scheme suppresses outliers because incorrect depths are unlikely to be geometrically consistent across multiple independent viewpoints — the probability of a random outlier passing checks drops exponentially as .
A point at world position (3.0, 1.5, 7.0) is estimated from 5 different views with depths [6.98, 7.02, 7.01, 12.50, 6.99]. Using a consistency threshold of 0.1m and requiring 3 agreeing views, is this point kept?
Related Problems on PixelBank
Every method so far in this chapter is passive: it waits for ambient light and infers depth from appearance, so it fails in the dark, on blank walls and at long range, exactly where the previous topic's photo-consistency has nothing to match. Active sensors remove that dependence by sending out their own light and timing its return. Distance follows from the speed of light, about 30 cm per nanosecond of round trip, so the engineering problem becomes measuring time with picosecond-level precision. Two families dominate. Time-of-flight cameras flood the scene with modulated infrared light and measure, at every pixel at once, the phase shift of the returning signal, which gives dense depth at video rate but over a short range that wraps around. LiDAR fires narrow laser pulses and times each echo directly, reaching hundreds of metres with centimetre accuracy, but only along a sparse set of beams. This topic covers both principles, the trade-offs between spinning and solid-state LiDAR, and the peculiar structure of LiDAR point clouds, rings and range-dependent density, that later perception models must handle.
Definition
Active depth sensors emit light and measure its round trip to compute distance. Continuous-wave time-of-flight cameras recover depth from the phase shift of modulated light, , unambiguous up to . LiDAR times individual laser pulses, , and scans beams over azimuth and elevation to build a sparse point cloud.
In this topic
- 1Time-of-Flight Cameras
- 2Spinning and Solid-State LiDAR
- 3LiDAR Point Cloud Characteristics
Time-of-Flight Cameras
ToF cameras emit modulated infrared light and measure the phase shift of the returned signal to compute distance for every pixel simultaneously. The phase shift is proportional to the round-trip travel time of light. ToF sensors produce dense depth maps at video frame rates (30-60 fps) with relatively low latency. However, they have limited range (typically 0.2-10m), suffer from multi-path interference (reflected light bouncing off multiple surfaces before returning), and have lower spatial resolution than RGB cameras. Consumer ToF sensors include Apple's LiDAR Scanner (iPhone/iPad Pro) and Microsoft's Azure Kinect.
Time-of-flight cameras measure depth from the phase shift between emitted and received modulated light: where is the speed of light and is the modulation frequency. The maximum unambiguous range is — at 20 MHz modulation this is 7.5m. Beyond this, the phase wraps around causing aliasing. Multi-frequency systems resolve this by combining measurements at different values.
A ToF camera modulates light at 20 MHz. What is the maximum unambiguous range? If the phase measurement has 0.1 degree precision, what is the depth resolution?
Spinning and Solid-State LiDAR
LiDAR sensors emit short laser pulses and measure the time for each pulse to reflect back from a surface. Spinning LiDAR (e.g., Velodyne VLP-16, Ouster OS1) uses a rotating assembly with multiple laser-detector pairs arranged vertically, producing a 360-degree point cloud as the assembly spins. Each rotation generates one scan with characteristic ring patterns. Solid-state LiDAR (e.g., Livox, Hesai QT) uses no moving parts, instead steering the beam electronically or using MEMS mirrors. Solid-state LiDAR is cheaper, more reliable (no mechanical wear), and easier to integrate into vehicles, but has a narrower field of view. Both types measure distance with centimeter-level accuracy at ranges up to 200m or more.
Spinning LiDAR samples the 3D world as a function of azimuth and elevation : each measurement yields in spherical coordinates, converted to Cartesian as , , . With vertical channels and azimuth steps per revolution, each scan produces points. A 64-beam LiDAR at 0.1 degree azimuth resolution: points per scan.
A Velodyne VLP-16 has 16 laser channels, spins at 10 Hz and fires each channel every 0.2 degrees of azimuth. How many points per second does it produce, and what raw data rate does that imply at 32 bytes per point?
LiDAR Point Cloud Characteristics
LiDAR point clouds have distinctive properties that differ from photogrammetry-derived clouds. The density is non-uniform: points are dense near the sensor and sparse at range because the beams diverge. For spinning LiDAR, points form concentric rings corresponding to each laser channel. Range accuracy is typically 1-3cm but degrades on dark, reflective, or wet surfaces. Each point carries intensity (reflectance strength), which depends on the surface material, angle of incidence, and range. Some sensors also provide multiple returns per pulse, detecting both foreground objects (tree leaves) and the surface behind them (ground), which is critical for forestry and terrain mapping.
A scanner samples fixed angles, in azimuth and between beams, so at range neighbouring points are about and apart, and the number of points on an object of fixed area falls as . With a VLP-16 (0.2 by 2 degrees), the side of a car, 4 m by 1.5 m, gets roughly 490 points at 10 m but only about 23 at 50 m, a single ring. Beam divergence adds a separate effect, widening the footprint so returns blur edges at range. Range noise, by contrast, stays near a few centimetres regardless of distance.
A LiDAR at height 1.8m scans a flat ground plane. For a laser channel at -15 degrees from horizontal, what is the range to the ground and the spacing between consecutive ground points at 0.2 degree horizontal angular resolution?
Related Problems on PixelBank
Time-of-flight sensors, the previous topic, time light directly and need picosecond electronics. Structured light gets depth from a cheap projector and an ordinary camera by turning the problem back into stereo. Stereo matching, from Chapter 3, fails on blank surfaces because nothing identifies which pixel matches which. A projector fixes that by painting the scene with a known pattern, so every surface point carries a code: the camera reads the code, looks up which projector column emitted it, and triangulates exactly as with two cameras, the projector playing the second view. The design choice is the pattern. Sequences of stripes such as Gray codes give an exact code at every pixel but need many frames, so the scene must hold still; a single random-dot or speckle pattern works in one shot but only identifies small neighbourhoods. This topic covers coding, the triangulation and its error growth with distance, and three products that shipped the idea to millions of people: Kinect v1, Intel RealSense active stereo and Face ID. The next topic deals with depth maps these sensors leave full of holes.
Definition
Structured light projects a known pattern onto a scene and observes it with a camera displaced by a baseline . Decoding the pattern gives, for each camera pixel, the projector coordinate that lit it, and depth follows by triangulation, for rectified geometry with disparity . Temporal codes use several frames; spatial codes use one.
In this topic
- 1Coded Light and Pattern Projection
- 2Triangulation from Structured Light
- 3Practical Systems: Kinect, RealSense, Face ID
Coded Light and Pattern Projection
Structured light systems work by projecting patterns that uniquely encode each projector pixel or column. The simplest approach is temporal coding: project a sequence of binary stripe patterns (Gray codes), where each projector column has a unique binary sequence. The camera captures one image per pattern and decodes each pixel's corresponding projector column from the observed sequence. Single-shot methods project a single complex pattern (speckle, color-coded stripes, or De Bruijn sequences) where local neighborhoods are unique, enabling depth from a single frame at the cost of lower resolution. The Intel RealSense D400 series projects a static infrared dot pattern, while the Kinect v1 used a fixed speckle pattern.
Structured light encodes spatial position in projected patterns: a Gray code sequence of binary patterns divides the projector's columns into unique codes, so each projector column is identified by its -bit binary label. With patterns, this yields unique column identities. The decoding step reads the binary code at each camera pixel to establish projector-camera correspondences, which are then triangulated to recover depth.
A structured light system needs to encode 1024 projector columns. How many Gray code patterns are needed? If the camera runs at 60 fps, what is the maximum depth frame rate?
Triangulation from Structured Light
Once the camera pixel is matched to a projector column , depth is computed by triangulation. The projector and camera form a stereo pair with known baseline . The projector column corresponds to a known angle from the projector's optical axis, and the camera pixel corresponds to angle . Depth is computed from the triangle formed by the baseline and these two angles. For a parallel projector-camera setup, depth simplifies to , identical to stereo vision with the projector replacing one camera. The precision of the depth measurement depends on the baseline length, focal length, and the accuracy of the pattern decoding.
Triangulation from structured light uses the baseline between projector and camera: where is the focal length, is the camera pixel coordinate, and is the corresponding projector column. This is geometrically identical to stereo disparity, but with one camera replaced by a projector. The depth precision degrades quadratically with distance, explaining why structured light systems work best at close range (m).
A structured light system has a 75mm baseline, focal length 600 pixels, and projects 1024 columns across a 60-degree field of view. A point is decoded at projector column 600 and observed at camera pixel 580. What is the depth?
Practical Systems: Kinect, RealSense, Face ID
Microsoft Kinect v1 (2010) used structured light with a fixed infrared speckle pattern and an IR camera to produce 640x480 depth maps at 30fps, ranging from 0.8m to 3.5m. Kinect v2 (2014) switched to time-of-flight. Intel RealSense D400 series combines active IR projection (dot pattern) with stereo matching between two IR cameras, achieving 0.1-10m range at up to 90fps. Apple's Face ID uses a dot projector that casts 30,000 IR points onto the face, which are captured by an IR camera and matched against a stored 3D face model. Each system makes different trade-offs between range, resolution, accuracy, frame rate, power consumption, and cost.
Kinect v1 is a triangulation sensor with pixels and cm, so depth error grows as . A whole pixel of disparity is 23 mm at 1 m and 368 mm at 4 m; Kinect quantizes disparity to 1/8 pixel, giving steps of about 3 mm and 46 mm. Khoshelham and Elberink measured random error rising from a few millimetres to about 4 cm at the 5 m limit. Face ID projects more than 30,000 dots at 25 to 50 cm, where the same quadratic law makes millimetre resolution possible with a short baseline.
Face ID projects 30,000 dots onto a face 30cm away. If the face area is roughly 20cm x 25cm, what is the average dot spacing? How does this compare to Kinect v1's resolution at 2m range?
Related Problems on PixelBank
The previous two topics built sensors that measure depth directly, and both leave gaps. A 64-beam LiDAR projected into a KITTI camera image lands on only about 4 to 5 percent of the pixels, in rings that leave whole object parts unmeasured. Time-of-flight and structured-light cameras give dense maps but drop out on black, shiny or transparent surfaces and at depth edges, and they usually have lower resolution than the colour camera beside them. Applications want the opposite: a full-resolution depth value at every pixel, aligned with the colour image. Depth completion fills that gap by using the colour image as a guide. Depth tends to be smooth within a surface and to jump where the image shows an edge, so the measurements can be spread along surfaces but stopped at boundaries. This topic covers sparse-to-dense networks that take LiDAR plus RGB, guided upsampling with joint bilateral filters, and self-supervised refinement with the photometric loss from the first topic. The last topic then defines the metrics used to score all of these outputs.
Definition
Depth completion predicts a dense depth map from sparse or incomplete depth measurements, usually guided by an aligned RGB image. The network or filter propagates measured values along surfaces and stops at image edges, and is trained against denser ground truth or with self-supervised photometric losses. Guided upsampling is the special case of a dense but low-resolution input.
In this topic
- 1Sparse-to-Dense Depth Completion
- 2Guided Depth Upsampling
- 3Self-Supervised Depth Refinement
Sparse-to-Dense Depth Completion
Sparse-to-dense completion takes a few thousand depth measurements, typically LiDAR points projected into the image as a mostly-zero depth channel, together with the RGB image, and fills in every pixel. Ordinary convolutions treat the zeros as real values, so Uhrig et al. proposed sparsity-invariant convolutions that normalize by the number of valid inputs; later networks learn image-guided propagation instead. The colour image tells the network where surfaces end, so depth spreads within an object and stops at its silhouette. Failure modes are thin structures missed by every beam, and edges where the image shows no contrast. On KITTI the best methods reach about 0.68 m RMSE.
Sparse-to-dense completion solves an interpolation problem: given sparse measurements at known pixels, predict depth everywhere. The problem is ill-posed without a prior — with 5% input density, % of pixels have no direct measurement. The network learns a spatial prior that fills gaps by propagating depth along surfaces detected in the RGB image, using image edges as boundaries that prevent depth bleeding across object boundaries.
About 20,000 LiDAR points from one scan project into a 1242 by 375 KITTI image. What fraction of pixels have a measurement, and what errors do the best completion methods achieve on this benchmark?
Guided Depth Upsampling
Guided upsampling increases the resolution of a low-resolution depth map using a high-resolution RGB image as a guide. This is needed when the depth sensor has lower resolution than the color camera (common with ToF cameras) or when depth is computed at reduced resolution for efficiency. Joint bilateral upsampling applies a bilateral filter where the spatial kernel comes from the target resolution and the range kernel is driven by the RGB image. This ensures depth edges align with color edges. Learned approaches use encoder-decoder networks that fuse depth and RGB features, producing high-resolution depth that respects the fine-grained structure visible in the color image.
Guided depth upsampling uses the RGB image as an edge-preserving prior: the joint bilateral filter computes where is a spatial Gaussian and weights by color similarity. Pixels with similar color get similar depth — this transfers the sharp edges from the high-res RGB to the low-res depth. Learned variants replace the hand-crafted kernels with convolutional weights conditioned on image features.
A ToF sensor produces 320x240 depth at 30fps. The RGB camera is 1280x960. What upsampling factor is needed? If we use a guided bilateral filter with 5x5 spatial kernel, how many multiply-adds per output pixel?
Self-Supervised Depth Refinement
Self-supervised methods refine depth maps without ground-truth supervision by exploiting multi-view consistency. Given an initial depth estimate and camera poses, the method warps one image to another viewpoint and compares with the actual image at that viewpoint. The photometric error, structural similarity (SSIM), and depth smoothness terms form the loss function. Iterative refinement networks take a coarse depth map and progressively improve it by correcting errors in a residual fashion. These methods are particularly useful when ground-truth dense depth is unavailable, such as outdoor scenes where LiDAR is sparse or indoor scenes without structured light calibration.
Self-supervised depth refinement uses photometric consistency as a training signal: the refined depth should produce better view synthesis than the initial depth . The loss penalizes depth errors that cause visible artifacts in the warped image. This is powerful because it does not require ground-truth dense depth — only temporally adjacent frames or stereo pairs. The gradient pushes depth values toward geometrically consistent predictions.
A self-supervised refinement model reduces photometric loss from 0.15 to 0.08 over 5 iterations. If each iteration costs 12ms of GPU time, and the baseline depth estimation takes 35ms, what is the total latency? Is this real-time at 30fps?
Related Problems on PixelBank
Every topic in this chapter produces a depth map, from monocular networks to completed LiDAR, and each paper claims to beat the last. Comparing them needs a shared yardstick, and depth error is not one number. A 10 cm error matters at 1 m and hardly at all at 50 m; a few catastrophic pixels can dominate a squared error while barely moving a median; and a relative model like MiDaS cannot be scored in metres at all until its scale is aligned. The field therefore reports a standard bundle, introduced by Eigen et al. in 2014 and used by KITTI and NYU-Depth-v2 ever since. This topic defines it in three groups. Error metrics, AbsRel, SqRel, RMSE and RMSE in log space, measure how far predictions are off. Threshold accuracies, the delta-1, delta-2 and delta-3 scores, count the fraction of pixels within a ratio of the truth. Scale-invariant and ordinal metrics ignore global scale and score only structure or ordering, which is how relative depth models are compared. Knowing what each metric hides is how you read a results table critically.
Definition
Depth evaluation metrics compare a predicted map with ground truth over valid pixels. AbsRel averages , RMSE is the root mean squared error in metres, and is the fraction of pixels with . Scale-invariant log error and ordinal metrics remove global scale before scoring.
In this topic
- 1Absolute and Relative Error Metrics
- 2Delta Threshold Accuracy (d1, d2, d3)
- 3Ordinal and Scale-Invariant Metrics
Absolute and Relative Error Metrics
AbsRel (Absolute Relative error) divides the absolute depth error by the ground-truth depth: AbsRel = mean(|d_pred - d_gt| / d_gt). This normalizes error by range, so a 10cm error at 1m depth counts the same as a 1m error at 10m depth. SqRel (Squared Relative error) is mean((d_pred - d_gt)^2 / d_gt), which penalizes large errors more heavily. RMSE (Root Mean Squared Error) = sqrt(mean((d_pred - d_gt)^2)) gives the absolute error in the same units as depth (meters). RMSE_log uses log-transformed depth: sqrt(mean((log(d_pred) - log(d_gt))^2)), which is more sensitive to errors in nearby regions. AbsRel is the single most reported metric because it is intuitive, scale-aware, and robust.
AbsRel, , divides by the true depth, so a 0.5 m error at 2 m (25 percent) counts 25 times more than the same error at 50 m (1 percent). SqRel, , is not a square root: it squares the error and divides by depth once, so it grows with absolute error and punishes large misses at range. RMSE keeps metres and is dominated by the farthest, largest errors, while RMSE in log space measures ratios and treats a 10 percent miss the same at any distance.
A depth model predicts [1.2, 4.5, 9.0] meters for pixels with ground truth [1.0, 5.0, 10.0]. Compute AbsRel, SqRel, and RMSE.
Delta Threshold Accuracy (d1, d2, d3)
The delta accuracy metric measures the percentage of pixels where the ratio between predicted and ground-truth depth is within a threshold. Specifically, delta_k = percentage of pixels where max(d_pred/d_gt, d_gt/d_pred) < 1.25^k. The standard thresholds are d1 (k=1, threshold 1.25), d2 (k=2, threshold 1.5625), and d3 (k=3, threshold 1.953125). Higher is better: d1=0.95 means 95% of pixels have predicted depth within 25% of ground truth. This metric is robust to outliers (a single catastrophic error does not dominate) and gives an intuitive sense of how many pixels are 'correct' at different tolerance levels.
The threshold metrics count the fraction of pixels where for and . At , this requires predictions within % of ground truth; at , within % (). The ratio formulation is scale-aware: measures overestimation and measures underestimation, with the max ensuring both directions are penalized equally. High () indicates that most predictions are tightly accurate.
For predictions [1.2, 4.5, 2.0] and ground truth [1.0, 5.0, 10.0], compute d1, d2, d3.
Ordinal and Scale-Invariant Metrics
Ordinal metrics evaluate whether the model correctly predicts depth ordering (which pixel is closer) regardless of absolute values. Ordinal error counts the percentage of pixel pairs where the predicted relative order disagrees with ground truth. This is useful for evaluating relative depth models like MiDaS that do not predict metric depth. Scale-invariant RMSE (siRMSE) aligns prediction and ground truth with an optimal scale factor before computing error, removing global scale ambiguity. The Spearman rank correlation measures how well the predicted depth ranking matches the true ranking. These metrics are critical when ground truth comes from different sources with incompatible scales.
Scale-invariant log error subtracts the squared mean of log-differences, removing the global scale component. The parameter controls how much scale invariance to apply: gives standard log MSE, gives fully scale-invariant loss. This metric is essential for evaluating monocular depth models that can only predict relative depth — it compares the structure of the depth map rather than absolute values.
Predicted depths: [3, 1, 5, 2, 4]. Ground truth: [6, 2, 10, 4, 8]. Compute the ordinal error (fraction of incorrectly ordered pairs) and Spearman rank correlation.