OpenCV warping image based on calcOpticalFlowFarneback - c++

Question

I'm trying to perform a complex warp of an image using Dense Optical Flow. I am trying to warp the second image into roughly the same shape as the first image.
cv::Mat flow;
cv::calcOpticalFlowFarneback( mGrayFrame1, mGrayFrame2, flow, 0.5, 3, 15, 3, 5, 1.2, 0 );
cv::Mat newFrame = cv::Mat::zeros( frame.rows, frame.cols, frame.type() );
cv:remap( frame, newFrame, flow, cv::Mat(), CV_INTER_LINEAR );
I calculate the flow from two grayscale frames. I am now trying to remap my original (i.e. non-grayscale) image using this flow information using the cv::remap function. However, I get a very badly distorted image from it. I simply end up with an orange and black image that bears a small resemblance to my original image.
How do I use cv::remap with the calculated flow?

Answer 1

The remap function cannot work with flow directly. One must use a separate map that is computed by taking the backwards flow (from frame2 to frame1) and then offsetting each flow vector by its (x, y) location on the pixel grid. See details below.
Recall the backwards optical flow formula:
frame1(x, y) = frame2(x + flowx(x, y), y + flowy(x, y))
The remap function transforms the source image using a specified map:
dst(x, y) = src(mapx(x, y), mapy(x, y))
Comparing the two equations above, we may determine the map that remap requires:
mapx(x, y) = x + flowx(x, y)
mapy(x, y) = y + flowy(x, y)
Example:
Mat flow; // backward flow
calcOpticalFlowFarneback(nextFrame, prevFrame, flow);
Mat map(flow.size(), CV_32FC2);
for (int y = 0; y < map.rows; ++y)
{
for (int x = 0; x < map.cols; ++x)
{
Point2f f = flow.at<Point2f>(y, x);
map.at<Point2f>(y, x) = Point2f(x + f.x, y + f.y);
}
}
Mat newFrame;
remap(prevFrame, newFrame, map);

Answer 2

Here is a Python solution to warp image from optical flow:
import cv2
import numpy as np
def warp_flow(flow, img1=None, img2=None, interpolation=cv2.INTER_LINEAR):
"""Use remap to warp flow, generating a new image.
If img1 is input, the output will be img2_warped, but there will be multiple pixels corresponding to a single pixel, resulting in sparse holes.
If img2 is input, the output will be img1_warped, and there will be no sparse holes. The latter approach is preferred.
Args:
flow (np.ndarray): flow
img1 (np.ndarray, optional): previous frame
img2 (np.ndarray, optional): next frame
Returns:
warped image"
"""
h, w, _ = flow.shape
remap_flow = flow.transpose(2, 0, 1)
remap_xy = np.float32(np.mgrid[:h, :w][::-1])
if img1 is not None:
uv_new = (remap_xy + remap_flow).round().astype(np.int32)
mask = (uv_new[0] >= 0) & (uv_new[1] >= 0) & (uv_new[0] < w) & (uv_new[1] < h)
uv_new_ = uv_new[:, mask]
remap_xy[:, uv_new_[1], uv_new_[0]] = remap_xy[:, mask]
remap_x, remap_y = remap_xy
img2_warped = cv2.remap(img1, remap_x, remap_y, interpolation)
mask_remaped = np.zeros((h, w), np.bool8)
mask_remaped[uv_new_[1], uv_new_[0]] = True
img2_warped[~mask_remaped] = 0
return img2_warped
elif img2 is not None:
remap_x, remap_y = np.float32(remap_xy + remap_flow)
return cv2.remap(img2, remap_x, remap_y, interpolation)
img1 = cv2.imread("img1.jpg")
img2 = cv2.imread("img2.jpg")
flow = cv2.calcOpticalFlowFarneback(
img1.mean(-1), img2.mean(-1), None, 0.5, 3, 15, 3, 5, 1.2, 0
)
img2_warped = warp_flow(flow, img1=img1)
img1_warped = warp_flow(flow, img2=img2)
cv2.imwrite("warped.jpg", np.vstack([img1_warped, img2_warped]))
cv2.imwrite("target.jpg", np.vstack([img1, img2]))
The examples img1 img2 and flow visualization :
The results warped.jpg target.jpg: