OpenPano: How to write a Panorama Stitcher

This is a summary of the algorithms I used to write OpenPano: an open source panorama stitcher. You can find the source code on github.

SIFT Feature

Lowe's SIFT [1] algorithm is implemented in feature/. The procedure of the algorithm and some results are briefly described in this section.

Scale Space & DOG Space

A Scale Space consisting of S×O grey-scale images is built at the beginning. The original image is resized in Odifferent sizes (AKA. octaves), and each is then Gaussian-blurred by S different σ. Since features are then detected on different resized version of the original image, features will then have scale-invariant properties.

The gaussian blur here is implemented by applying two 1-D convolutions, rather than a 2-D convolution. This speeds up the computation significantly.

In each octave, calculate the differences of every two adjacent blurred images, to build a Difference-of-Gaussian space. DOG Space consists of (

S−1)×O(S−1)×O grey images.

Extrema Detection

In DOG Space, detect all the minimum and maximum by comparing a pixel with its 26 neighbors in three directions: x,y,σ.

Then use parabolic interpolation to look for the accurate (x,y,σ) location of the extrema. Reject the points withlow contrast (by thresholding pixel value in DOG image) or on the edge (by thresholding principle curvature), to get more distinctive features. The results are like this:

Orientation Assignment

First, we calculate gradient and orientation for every point in the Scale space. For each keypoint detected by the previous procedure, the orientations of its neighbor points will be collected and used to build an orientation histogram, weighted by the magnitude of their gradients, together with a gaussian kernel centered at the keypoint. The peak in the histogram is chosen to be the major orientation of the keypoint, as shown by the arrows below:

Descriptor Representation

Lowe suggested [1] choosing 16 points around the keypoint, to build orientation histograms for each point and concatenate them as SIFT feature. Each histogram uses 8 different bins ranging from 0 to 360 degree. Therefore the result feature is a 128-dimensional floating point vector. Since the major orientation of the keypoint is known, by using relative orientation to the major one, this feature is rotation-invariant.

Also, I followed the suggestions from [2] to use RootSIFT, which is a simple modification on SIFT, and is considered more robust.

Feature Matching

Euclidean distance of the 128-dimensional descriptor is the distance measure for feature matching between two images. A match is considered not convincing and therefore rejected, if the distances from a point to its closest neighbor and second-closest neighbor are similar. A result of matching is shown:

Feature matching turned out to be the most time-consuming step. So I use FLANN library to query 2-nearest-neighbor among feature vectors. To calculate Euclidean distance of two vectors, Intel SSE intrinsics are used to speed up.

Transformation Estimation

Estimate from Match

It's well known [3] that for any two images taken from a camera at some fixed point, the homogeneous coordinates of matching points can be related by a homography matrix H, such that for a pair of corresponding point p=(x,y,1),q=(u,v,1), we have

p∼Hq=K1R1RT2K−12qp∼Hq=K1R1R2TK2−1q The homography matrix is a 3x3 matrix up to a scale ambiguity. It has the following two possible formulation:

Homography I: H=⎡⎣⎢a11a21a31a12a22a32a13a231⎤⎦⎥

Homography II: H=⎡⎣⎢a11a21a31a12a22a32a13a23a33⎤⎦⎥ with constraint ∥H∥=1

When two images are taken with only camera translation and rotation aligned with the image plane (no perspective skew), then they can be related by an affine matrix of the form: A=⎡⎣⎢a11a210a12a220a13a231⎤⎦⎥

Given a set of matches, a matrix of any of the above formulations can be estimated via Direct Linear Transform. These estimation methods are implemented in lib/imgproc.cc. See [3] for details.

However these methods are not robust to noise. In practice, due to the existence of false matches, RANSAC(Random Sample Consensus) algorithm [4] is used to estimate a transformation matrix from noisy data. In every RANSAC iteration, several matched pairs of points are randomly chosen to produce a best-fit transformation matrix, and the pairs which agree with the matrix are taken as inliers. After a number of iterations, the transform that has most number of inliers are taken as the final result. It's implemented in stitch/transform_estimate.cc.

Match Validation

After each matrix estimation, I performed a health check to the matrix, to avoid malformed transformation estimated from false matches. This includes two checks: (see stitch/homography.hh)

The skew factor in homography (H3,1,H3,2) cannot be too large. Their absolute values are usually less than 0.002.
Flipping cannot happen in image stitching. A homography is rejected if it flips either x or y coordinate.

In unconstrained stitching, it's necessary to decide whether two images match or not. The number of inliers estimated above could be one criteria. However, for high-resolution images, it's common that you'll actually find false matches with a considerable number of inliers, as long as enough RANSAC iterations are performed. To solve this, note that false matches are usually more randomly distributed spatially, geometry constraints of matching can also help filter out bad matches. Therefore, after RANSAC finished and returned a set of inliers, some overlapping test can be used to further validate the matching, as suggested by [5].

Specifically, with a candidate transformation matrix, I could find out the region within one image covered by another image, by calculating the convex hull of the transformed corners. Two filters can then be applied after the overlapping region is found:

The overlapping area within two images shouldn't differ too much.
Within the region, a reasonably large portion of matches should be inliers. I also used the inlier ratio as the confidence score of this match.

Since false matches are likely to be random and irregular, this technique help filter out false matches with irregular geometry.

Cylindrical Panorama

Necessity of Projection

If a single-direction rotational input is given, as most panoramas are built, using planar homography leads tovertical distortion, as following:

This is because a panorama is essentially an image taken with a cylindrical or spherical lens, not a planar lens any more. Under this setting, the white circle on the grass around the camera (as in the above picture) is expected to become a straight line. A good demo revealing the reason of this projection can be seen at this page.

The way to handle this problem is to warp images by a cylindrical or spherical projection, before or after transform estimation. These are the two modes used in this system. In this section we will introduce the pipeline based on pre-warping, which is used in cylinder mode. This pipeline is generally good, although it has more assumptions on the data and requires some tricks to work well. It is implemented in stitch/cylstitcher.cc.

Warp

We can project the image to a cylinder surface at the beginning of stitching, by the following formula:

OpenPano: How to write a Panorama Stitcher

OpenPano: How to write a Panorama Stitcher

SIFT Feature

Scale Space & DOG Space

Extrema Detection

Orientation Assignment

Descriptor Representation

Feature Matching

Transformation Estimation

Estimate from Match

Match Validation

Cylindrical Panorama

Necessity of Projection

Warp

OpenPano: How to write a Panorama Stitcher

How to write a robust system level service - some key learning - 如何寫好一個健壯的系統級服務

How to write a comparison and contrast essay?

How to write a cell address encoder in ruby.

How To Write A Stand-Out Technical Resume

How to Write a Bug Report

How to write a super fast link shortener with Elixir, Phoenix, and Mnesia

How to write good content as a mediocre UI/UX Designer

How to Remove A Service Entry From Win10 Service List

WPF:How to display a Bitmap on Image control

【轉】How to initialize a two-dimensional array in Python?

How to Have a Healthy Relationship --shanbei 為單身節寫

How to force a log switch-強制切換日誌

HOW TO BECOMING A FREELANCE ENGINEERING CONSULTANT.

How to write threats to validity?

jquery ----> How to Create a Basic Plugin (翻譯)

How to Create a Perl Based Custom Monitor on NetScaler

How To Read A Paper

How to setup a slave for replication in 6 simple steps with Percona XtraBackup

How to Write Go Code

OpenPano: How to write a Panorama Stitcher

OpenPano: How to write a Panorama Stitcher

SIFT Feature

Scale Space & DOG Space

Extrema Detection

Orientation Assignment

Descriptor Representation

Feature Matching

Transformation Estimation

Estimate from Match

Match Validation

Cylindrical Panorama

Necessity of Projection

Warp

相關推薦