Google Research Blog

The latest news from Research at Google

PhotoScan: Taking Glare-Free Pictures of Pictures

Thursday, April 20, 2017

Posted by Ce Liu, Michael Rubinstein, Mike Krainin and Bill Freeman, Research Scientistsupdate to PhotoScaniOSAndroid

Left: A regular digital picture of a physical print. Right: Glare-free digital output from PhotoScan

Left: The captured, input images (5 in total). Right: If we stabilize the images on the photo, we can see just the glare moving, covering different parts of the photo. Notice no single image is glare-free.

SIGGRAPH 2015obstruction-free photographyHow it Workssparse feature pointsORB featuresHarris cornershomographies

Left: Detected feature matches between the reference frame and each other frame (left), and the warped frames according to the estimated homographies (right).

optical flow

Comparison between the warped frames using homographies (left) and after the additional warp refinement using optical flow (right).

Comparison between the glare removal result with (right) and without (left) optical flow refinement. In the result using homographies only (left), notice artifacts around the eye, nose and teeth of the person, and duplicated stems and flower petals on the fabric.

bilinear combination

The grid setup for grid optical flow. A point p is represented as the bilinear interpolation of the four corner points of the cell that encapsulates it.

Left: Illustration of the computed flow field on one of the frames. Right: The flow color coding: orientation and magnitude represented by hue and saturation, respectively.

Szeliski and Coughlan (1997)

Flipping between the homography-registered frame and the flow-refined warped frame (using the above flow field), superimposed on the (clean) reference frame, shows how the computed flow field “snaps” image parts to their corresponding parts in the reference frame, improving the registration.