Chapter 9: 3D Scene Understanding

3D scene understanding encompasses a family of tasks that aim to parse an entire 3D environment into meaningful components. While object detection identifies individual objects with bounding boxes, scene understanding seeks a denser, more complete interpretation: labeling every point with a semantic class, separating individual object instances, modeling spatial relationships between entities, and reconstructing continuous surfaces.

These capabilities are essential for autonomous systems that must navigate complex environments. A self-driving car needs to know not just where cars are, but also the layout of roads, sidewalks, and buildings. A household robot must understand room structure, furniture arrangements, and object relationships ("the mug is on the table, which is next to the chair").

The chapter progresses from fundamental tasks (semantic segmentation, instance segmentation) through their combination (panoptic segmentation), to higher-level representations (scene graphs) and implicit continuous representations (occupancy networks). Each builds on the previous, creating increasingly rich models of 3D environments.

Key areas covered:

• 3D Semantic Segmentation: Per-point classification using efficient point cloud architectures
• 3D Instance Segmentation: Grouping points into individual object instances
• Panoptic Segmentation: Unified segmentation of both "stuff" and "things"
• Scene Graphs: Representing spatial relationships between objects as a graph
• Occupancy Networks: Continuous implicit 3D representations learned from data
• Indoor Scene Reconstruction: Structured understanding of indoor environments