Chapter 1: Introduction to 3D Computer Vision

The world coordinate system is a fixed reference frame that every object, sensor and camera in a scene can be expressed in, so measurements from different devices can be compared directly. Its origin and axes are a choice, not a physical fact: a robot might put the origin at its charging dock with Z up, a reconstruction tool might use the first camera's position, and a graphics engine might use Y up. What matters is that the choice is fixed and documented. Mixing conventions, for example Z-up robotics data with Y-up graphics assets, silently rotates everything by 90 degrees, which is one of the most common integration bugs.

The camera coordinate system has its origin at the optical centre, with Z along the viewing direction into the scene, X to the right and Y down in the OpenCV convention. Graphics libraries such as OpenGL instead look down negative Z with Y up, so always check which one a tool uses. A world point is moved into this frame by undoing the camera's pose: subtract the camera position $C$ and rotate by the camera's orientation $R$, giving $P_{cam} = R(P_{world} - C)$. Only points with positive $Z_{cam}$ are in front of the camera, which makes a negative depth an immediate sign of a convention error.

The image coordinate system is measured in pixels on the sensor, with the origin at the top-left corner, $u$ increasing to the right and $v$ increasing downward. A camera-frame point reaches it in two steps. First, perspective division by the depth gives normalized coordinates $x = X/Z$ and $y = Y/Z$, which is where size shrinks with distance. Second, an affine map scales by the focal length in pixels and shifts by the principal point $(c_x, c_y)$, where the optical axis meets the sensor. The division is the only nonlinear step, and it is also why depth cannot be recovered from one image.

A rotation matrix $R$ is a 3-by-3 matrix that turns vectors about the origin without changing their lengths or the angles between them. Two conditions define it: orthogonality, $R^T R = I$, which preserves lengths and makes the inverse just the transpose, and $\det R = +1$, which rules out mirror images. Together they cut nine numbers down to three degrees of freedom. The columns of $R$ are the rotated frame's axes written in the original frame. All such matrices form the group SO(3). Because rotations compose by multiplication, numerical drift can creep in, so long chains are periodically re-orthonormalized.

A translation vector $t = (t_x, t_y, t_z)$ shifts every point by the same displacement, so it preserves distances, angles and orientation, and unlike a rotation it leaves no point fixed. Translation is not a linear map in ordinary 3D coordinates, because the origin moves, which is why it cannot be written as a 3-by-3 matrix and must be added separately. A rigid transformation applies the rotation first and then the translation, $P' = RP + t$. The order is part of the definition: translating first and then rotating also moves the translation itself, producing a different result whenever the rotation is not the identity.

Rigid transformations chain: if $T_1$ maps frame A to frame B and $T_2$ maps B to C, the composite $T_2 \circ T_1$ maps A straight to C. Substituting gives $R_2 R_1 p + (R_2 t_1 + t_2)$, so the rotations multiply and the first translation is rotated by the second rotation. Composition is associative but not commutative: rotations about different axes do not commute, and a rotation does not commute with a translation. Inverting a transform gives $R^T(p - t)$. Getting the order or the direction of a chain wrong is the most common pose bug, which is why careful code names transforms by their source and target frames.

A 3D point $(x, y, z)$ becomes $(x, y, z, 1)$ by appending a fourth component, the weight $w$. Any nonzero multiple, such as $(2x, 2y, 2z, 2)$, names the same point, and converting back means dividing the first three entries by $w$. This redundancy is the whole trick. Because the point carries a constant 1, a matrix can add a constant offset, so translation becomes linear. Because only ratios matter, the divide-by-depth of perspective projection becomes the final divide-by-$w$. Code must divide before comparing or measuring distances, since two vectors that look different may be the same point.

A rigid transformation becomes the 4-by-4 matrix $T = \begin{bmatrix} R & t \\ 0 & 1 \end{bmatrix}$, with the rotation in the top-left block and the translation in the last column. Multiplying $T$ by $(x, y, z, 1)$ returns $(Rp + t, 1)$, so rotation and translation happen in one product, and a chain of transforms is a chain of matrix multiplications. The inverse keeps the same shape, with $R^T$ in the corner and $-R^T t$ as the translation, which avoids a general 4-by-4 inversion. Robotics, graphics and SLAM libraries all store poses this way.

A homogeneous vector with $w = 0$, such as $(1, 0, 0, 0)$, cannot be divided back to a finite point. It represents a direction, a point at infinity. Projective geometry treats these as ordinary points, so two parallel lines meet at the point at infinity in their shared direction. A camera maps that point to a finite pixel, the vanishing point, which is why railway tracks converge in a photo. Rigid transformations treat directions correctly for free: the translation column is multiplied by $w = 0$, so a direction is rotated but never shifted.

Euler angles write a rotation as three successive rotations about coordinate axes, for example yaw about Z, then pitch about Y, then roll about X. Three numbers with physical meaning make them popular for user interfaces and for vehicles. Their flaw is gimbal lock: when the middle angle reaches 90 degrees, the first and third axes line up, so two of the angles control the same motion and one degree of freedom disappears. Near that configuration small rotations need huge angle changes, which breaks interpolation and optimization. There are also 12 axis orders, plus intrinsic or extrinsic readings, so mismatched conventions are a classic source of bugs.

A unit quaternion $q = (w, x, y, z)$ with norm 1 encodes a rotation by angle $\theta$ about unit axis $n$ as $w = \cos(\theta/2)$ and $(x, y, z) = \sin(\theta/2)\, n$. Four numbers on a smooth sphere give no gimbal lock, composition is a quaternion product, and renormalizing is cheap compared with re-orthogonalizing a matrix. Spherical linear interpolation, SLERP, moves along the great circle between two quaternions at constant angular speed, which is why game engines, robotics middleware and SLAM systems store orientation this way. The catch is the double cover: $q$ and $-q$ are the same rotation, so interpolation and averaging must first flip one onto the same hemisphere.

Euler's rotation theorem says every rotation is a single turn by angle $\theta$ about some unit axis $n$. The rotation vector packs both into one 3-vector, $r = \theta n$, whose direction is the axis and whose length is the angle in radians. Rodrigues' formula converts it to a matrix, and it equals the matrix exponential of the skew-symmetric matrix built from $r$. With three numbers and no constraints, it suits optimization: bundle adjusters and OpenCV's solvePnP use it for small pose updates. Its weak spot is the wrap at $\pi$: rotation vectors of length near $\pi$ in opposite directions describe nearly the same rotation, and $\theta = 0$ leaves the axis undefined.

PLY, the Stanford polygon format, starts with a short text header that declares each element, such as vertex and face, with its count and a typed list of properties, followed by the data in ASCII or in little- or big-endian binary. Because the header describes the layout, a file can add any per-vertex or per-face property, such as colour, normals, confidence or curvature, and readers skip what they do not understand. That flexibility made it the standard output of 3D scanners and the default in Open3D, MeshLab and CloudCompare. For large data, always write binary: ASCII spends many characters per number and must be parsed.

OBJ is a plain-text format from graphics with one record per line: v for a vertex position, vt for a texture coordinate, vn for a normal, and f for a face. A face corner is written as position/texture/normal indices, counted from 1, so positions, texture coordinates and normals live in separate lists and can be shared independently. That is how a cube can share 8 corner positions while each face still gets its own flat normal. Materials go in a companion MTL file. OBJ is universally supported and easy to inspect, but it has no binary form, no per-vertex colour in the specification, and large meshes load slowly.

PCD is the native format of the Point Cloud Library; Open3D also reads and writes it. Its header lists field names, sizes, types and counts, then data in ASCII, binary or compressed binary. It records width and height, so an organized cloud from a depth camera keeps its pixel grid, which allows fast neighbour lookups. Colour is usually packed into one 4-byte field. Other formats serve other jobs. STL stores bare triangles for 3D printing. glTF is a compact, GPU-ready format for the web and AR. LAS and E57 archive survey LiDAR with return and scanner metadata. Choose by the downstream tool.

Self-driving vehicles combine LiDAR, cameras and radar to build a 3D picture of the road many times per second. 3D object detection outputs oriented boxes for cars, cyclists and pedestrians, with position, size and heading. Simultaneous localization and mapping, SLAM, tracks the vehicle's pose against a map that it also updates. Robots use the same tools to estimate an object's 6D pose before grasping it. All of this rests on calibration: every sensor pair needs an accurate rigid transform between frames, because a small angular error becomes a large position error at range, and a detection projected to the wrong pixels can be fused with the wrong object.

AR and VR headsets track the head's pose in six degrees of freedom, three rotations and three translations, using visual-inertial odometry that fuses camera images with the accelerometer and gyroscope of an inertial measurement unit. The IMU gives fast, smooth motion, and the cameras correct its drift. The headset also builds a mesh of the room, so virtual objects can sit on real tables and be hidden behind real furniture, and it estimates the 3D pose of the hands for gesture input. Because rendering must keep up with the display, every tracking step must fit in a few milliseconds.

Medical imaging reconstructs 3D volumes: CT combines X-ray projections from many angles into a stack of slices, and MRI measures a volume directly. Analysis then works in 3D, segmenting organs, tumours and vessels, measuring their volumes, and planning surgery. Navigation systems register the preoperative scan to the patient on the table with a rigid transform, so instruments can be shown inside the anatomy. Industrial scanners use structured light or lasers to triangulate millions of surface points and compare a manufactured part to its CAD model, flagging deviations of a fraction of a millimetre. Both domains are limited by data volume and by registration accuracy.