Bilinear Flow Warping

hardPROFrame InterpolationInterpolationCV: Motion Estimation

Implement frame warping using optical flow with bilinear interpolation to create an intermediate frame. This task involves understanding how to apply optical flow to warp a source frame.

The concept of optical flow is crucial here, as it represents the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and the scene. Given the optical flow $(u, v)$ at each pixel, the goal is to warp the source frame to create an intermediate frame. The warping process involves shifting each pixel in the source frame by the corresponding optical flow values.

Here are the steps to achieve this:

For each output pixel $(x, y)$ , calculate the source location by adding the optical flow values $(u, v)$ .
Check if the calculated source location falls between pixels.
If it does, apply bilinear interpolation to determine the intensity value of the output pixel.

I_{out}(x, y) = I_{src}(x + u(x,y), y + v(x,y))

Details

The flow is read at the output pixel: row $y$ , column $x$ of flow_u and flow_v give $u(x,y)$ and $v(x,y)$ in pixels. There is no separate time parameter; to build the frame at time $t$ , pass the flow already scaled by $t$ .
The source location $(x_s, y_s) = (x + u, y + v)$ is valid only when $0 \le x_s < W-1$ and $0 \le y_s < H-1$ , so that all four neighbouring pixels exist. Otherwise the output pixel is $0$ . (A location exactly on the last row or column counts as outside, so with zero flow the last row and column of the output are $0$ .)
Return an $H \times W$ array of 8-bit unsigned integers; interpolated values keep only their integer part (truncated, not rounded).

This technique is widely used in video processing applications, such as frame interpolation and video slow-motion.

Example

Input

src_frame = image at t=0
flow = motion from t=0 to t=1
warp_factor = 0.5

Output

Intermediate frame at t=0.5

Reasoning

src_frame is an $H \times W$ intensity grid; flow_u, flow_v are same-shape grids of horizontal and vertical shift in pixels. The sample's warp_factor = 0.5 means the flow passed in is pre-halved.
Take source rows $[0, 100, 200]$ , $[40, 140, 240]$ , $[80, 180, 255]$ and $u = 0.5$ , $v = 0.25$ everywhere. Output pixel $(0, 0)$ reads source point $(0.5, 0.25)$ , between $0, 100, 40, 140$ , with $d_x = 0.5$ , $d_y = 0.25$ .
Weights: $(1-d_x)(1-d_y) = 0.375$ on $0$ , $d_x(1-d_y) = 0.375$ on $100$ , $(1-d_x)d_y = 0.125$ on $40$ , $d_x d_y = 0.125$ on $140$ , giving $0 + 37.5 + 5 + 17.5 = 60$ .
Pixel $(1, 1)$ reads $(1.5, 1.25)$ : $0.375 \cdot 140 + 0.375 \cdot 240 + 0.125 \cdot 180 + 0.125 \cdot 255 = 196.875$ , kept as $196$ in the 8-bit output.
Column $x = 2$ ( $x_s = 2.5 \ge W - 1$ ) and row $y = 2$ ( $y_s = 2.25$ ) have no right or lower neighbour, so $0$ . Output rows: $[60, 160, 0]$ , $[100, 196, 0]$ , $[0, 0, 0]$ .
Notice zero flow would still blank the last row and column: those points sit exactly on the edge.

Constraints

src_frame: Source frame (H, W)
flow_u, flow_v: Optical flow fields (H, W)
Return: Warped frame

Learn this in

Motion EstimationCV Study Plan

More from CV: Motion Estimation

Hungarian Algorithm for MOThard Rolling Shutter Correctionhard Horn-Schunck Global Optical Flowmedium Intersection over Union (IoU) for Trackingmedium Trajectory Smoothing Filtermedium

Concept primerPREVIEW

Background Knowledge

Frame interpolation is about synthesizing new frames between existing video frames to create smoother motion, e.g., for slow-motion or frame rate up-conversion. A common classical approach uses optical flow, which estimates per-pixel motion vectors between frames. Each vector $(u(x,y), v(x,y))$ indicates where a pixel at $(x,y)$ moves to in the next (or previous) frame.

To generate an intermediate frame, you typically warp one or more source frames: for each pixel in the output frame, you sample from a source frame at a displaced (non-integer) coordinate $(x+u, y+v)$ . Since images are defined on an integer grid, you need interpolation to get an intensity value at non-integer positions. Bilinear interpolation is the standard choice: it interpolates over the 2×2 neighborhood around the floating-point coordinate and linearly blends the four pixels.

Here, the problem is the core low-level operation: given a source image and a dense flow field $(u,v)$ , compute the warped image

I_{\text{out}}(x,y) = I_{\text{src}}(x + u(x,y),\, y + v(x,y))

using bilinear interpolation whenever $(x+u, y+v)$ is not on integer coordinates. This is often called backward warping (or inverse warping), because you iterate over target pixels and pull values from the source image using the flow.

Algorithm / Approach

General pattern (backward warping with bilinear interpolation):

For each integer pixel $(x,y)$ in the output image:
Use the flow to compute the corresponding continuous coordinate in the source: $(x_s, y_s) = (x + u(x,y),\, y + v(x,y))$ .
If $(x_s, y_s)$ is outside the source image, handle via a chosen border policy (e.g., skip, clamp, or fill with 0).
Otherwise:
Find the integer neighbors around $(x_s, y_s)$ .
Compute bilinear interpolation weights.
Linearly combine the 4 neighbor pixels with those weights to get $I_{$ \text{out} $}(x,y)$ .

This pattern is the same used in many libraries’ “grid_sample” / “remap” / “warp” operations.

Step-by-Step Strategy

Assume:

Source image size: height H, width W, possibly channels C.
Flow fields u[y][x], v[y][x] defined for each pixel in the output grid.

Step 1: Loop over output pixels

for y in range(H):
 for x in range(W):
 ux = u[y, x]
 vy = v[y, x]
 xs = x + ux
 ys = y + vy
 #... interpolate from src at (xs, ys)

Step 2: Boundary check

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.