Rolling Shutter Correction

Implement a rolling shutter correction algorithm to rectify distorted video frames using gyroscope data. This task involves compensating for the skew caused by camera rotation during the rolling shutter capture process, where each row is captured at a slightly different time.

The rolling shutter effect is a consequence of the way images are captured in most cameras, where the sensor reads out the pixel values row by row, rather than all at once. When the camera rotates during this process, it results in a skewed image, which can be modeled using affine transformations. The gyroscope data provides the per-row rotation, which can be used to apply an inverse transformation to correct the distortion.

To achieve this, the following steps are involved:

1. Understand the rotation matrix and how it relates to the gyroscope data.
2. Apply the inverse rotation to each row of the image.
3. Resample: each output pixel copies a pixel from the same row of the input, at the (generally non-integer) source column rounded down.

$$\begin{pmatrix} x' \\ y' \end{pmatrix} = \begin{pmatrix} \cos(\theta) & -\sin(\theta) \\ \sin(\theta) & \cos(\theta) \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix}$$

Exact mapping

For a frame of height $H$ and width $W$, let $c_x = W/2$, $c_y = H/2$ and $\theta_y = -r_y$, the inverse of row $y$'s rotation $r_y$. Output pixel $(x, y)$ takes its value from input row $y$, column $\lfloor x_s \rfloor$, where

$$x_s = c_x + (x - c_x)\cos\theta_y - (y - c_y)\sin\theta_y$$

(the first row of the rotation above, applied about the image centre). If $x_s < 0$ or $x_s \ge W$, the output pixel is $0$. For RGB frames all three channels are copied together. Return an 8-bit unsigned integer array with the same shape as the input.

This technique is widely used in video stabilization applications.

Frame and rotation per row

Corrected frame