EmguCV Image Process: Estimating Projective Relations in Images
參考OpenCV 2 Computer Vision Application Programming Cookbook第九章
介紹內容如下：
Calibrating a camera
Computing the fundamental matrix of an image pair
Matching images using random sample consensus
Computing a homography between two images
前一篇介紹了如何透過兩張不同的角度影像取出fundamental matrix
這篇繼續延伸這個主題
透過SURF feature detector能夠個別取出兩張影像的feature points
這時希望找到對應的match feature points能夠落在前一篇所提的epipolar lines上
那要計算出epipolar lines就必須計算出fundamental matrix
但要計算出fundamental matrix又必須透過好的match feature points
這似乎像是雞生蛋、蛋生雞的議題
這便是本篇要討論的方向
這裡建立一個class來實作這樣的功能
class RobustMatcher
{
//pointer to the feature point detector object
private Feature2DBase<float> _Detector;
//max ratio between 1st and 2nd NN
private float _Ratio;
//if true will refine the F matrix
private bool _RefineF;
//min distance to epipolar
private double _Distance;
//confidence level (probabillity)
private double _Confidence;
public RobustMatcher()
{
this._Ratio = 0.55f;
this._RefineF = true;
this._Distance = 3.0;
this._Confidence = 0.99;
this._Detector = new SURFDetector(100, false);
}
...
}
建立一個RobustMatcher在建構式時將欄位的值帶入
這裡提供的欄位皆是可以更換的
可以再透過一層公開的屬性將欄位的值作替換
再來這裡提供一個公開的方法match來做操作
/// <summary>
/// Match faeture points using symmetry test and RANSAC
/// </summary>
/// <param name="image1">input image1</param>
/// <param name="image2">input image2</param>
/// <param name="keypoints1">output keypoint1</param>
/// <param name="keypoints2">output keypoint2</param>
/// <returns>return fundemental matrix</returns>
public Image<Gray, Byte> Match(Image<Gray, Byte> image1, Image<Gray, Byte> image2,out VectorOfKeyPoint keypoints1,out VectorOfKeyPoint keypoints2)
{
//1a. Detection of the SURF features
keypoints1 = this._Detector.DetectKeyPointsRaw(image1, null);
keypoints2 = this._Detector.DetectKeyPointsRaw(image2, null);
//1b. Extraction of the SURF descriptors
Matrix<float> descriptors1 = this._Detector.ComputeDescriptorsRaw(image1, null, keypoints1);
Matrix<float> descriptors2 = this._Detector.ComputeDescriptorsRaw(image2, null, keypoints2);
//2. Match the two image descriptors
//Construction of the match
BruteForceMatcher<float> matcher = new BruteForceMatcher<float>(DistanceType.L2);
//from image 1 to image 2
//based on k nearest neighbours (with k=2)
matcher.Add(descriptors1);
//Number of nearest neighbors to search for
int k=2;
int n = descriptors2.Rows;
//The resulting n*k matrix of descriptor index from the training descriptors
Matrix<int> trainIdx1 = new Matrix<int>(n, k);
//The resulting n*k matrix of distance value from the training descriptors
Matrix<float> distance1 = new Matrix<float>(n, k);
matcher.KnnMatch(descriptors2, trainIdx1, distance1, k, null);
matcher.Dispose();
//from image 1 to image 2
matcher = new BruteForceMatcher<float>(DistanceType.L2);
matcher.Add(descriptors2);
n = descriptors1.Rows;
//The resulting n*k matrix of descriptor index from the training descriptors
Matrix<int> trainIdx2 = new Matrix<int>(n, k);
//The resulting n*k matrix of distance value from the training descriptors
Matrix<float> distance2 = new Matrix<float>(n, k);
matcher.KnnMatch(descriptors1, trainIdx2, distance2, k, null);
//3. Remove matches for which NN ratio is > than threshold
int removed = RatioTest(ref trainIdx1, ref distance1);
removed = RatioTest(ref trainIdx2, ref distance2);
//4. Create symmetrical matches
Matrix<float> symMatches;
int symNumber = SymmetryTest(trainIdx1, distance1, trainIdx2, distance2, out symMatches);
Matrix<float> fundementalMatrix = RANSACTest(symMatches, keypoints1, keypoints2, symNumber, image2);
return null;
}
這裡Match的方法帶入兩張要比較的影像，以及回傳取得的key points (也就是featue points)個別取出key points之後
再透過BruteForceMatcher找出對應於兩張影像的key points
由於要找出互相對應的點
所以第一次做KnnMatch是以第一張影像為基準，求得trainIdx1、distance1
第二次便是以第二張影像為基準，求得trainIdx2、distance2
然後透過等等會介紹的RatioTest方法
過濾一些key points
/// <summary>
/// Clear matches for which NN ratio is > than threshold
/// </summary>
/// <param name="trainIdx">match descriptor index</param>
/// <param name="distance">match distance value</param>
/// <returns>return the number of removed points</returns>
int RatioTest(ref Matrix<int> trainIdx, ref Matrix<float> distance)
{
int removed = 0;
for (int i = 0; i < distance.Rows; i++)
{
if (distance[i, 0] / distance[i, 1] > this._Ratio)
{
trainIdx[i, 0] = -1;
trainIdx[i, 1] = -1;
removed++;
}
}
return removed;
}
RatioTest很簡單就是把透過 2 NN的match方法取出的key points
計算兩個distance的比值，以this._Ratio為threshold的方式來過濾
這裡this._Ratio的值設的越小，過濾的效果就越大
範例中的SURFDetector採用100作為threshold
若是沒有透過RatioTest的過濾
其找到的對應key points如下：
透過RatioTest的過濾，左邊濾掉了1155個key points，右邊的濾掉了1339個
對應的結果如下：
這時再透過所謂的symmetrical matching scheme來處理
也就是要在match key points的結果trainIdx1與trainIdx2中
找到有互相匹配的結果
/// <summary>
/// Create symMatches vector
/// </summary>
/// <param name="trainIdx1">match descriptor index 1</param>
/// <param name="distance1">match distance value 1</param>
/// <param name="trainIdx2">match descriptor index 2</param>
/// <param name="distance2">match distance value 2</param>
/// <param name="symMatches">return symMatches vector</param>
/// <returns>return the number of symmetrical matches</returns>
int SymmetryTest(Matrix<int> trainIdx1, Matrix<float> distance1, Matrix<int> trainIdx2, Matrix<float> distance2, out Matrix<float> symMatches)
{
symMatches = new Matrix<float>(trainIdx1.Rows, 4);
int count = 0;
//for all matches image1 -> image2
for (int i = 0; i < trainIdx1.Rows; i++)
{
//ignore deleted matches
if (trainIdx1[i, 0] == -1 && trainIdx1[i, 1] == -1)
{
continue;
}
//for all matches image2 -> image1
for (int j = 0; j < trainIdx2.Rows; j++)
{
//ignore deleted matches
if (trainIdx2[j, 0] == -1 && trainIdx2[j, 1] == -1)
{
continue;
}
//Match symmetry test
//if (trainIdx1[i, 0] == trainIdx2[j, 1] &&
// trainIdx1[i, 1] == trainIdx2[j, 0])
if (trainIdx1[i, 0] == j && trainIdx2[j, 0] == i)
{
symMatches[i, 0] = j;
symMatches[i, 1] = i;
symMatches[i, 2] = distance1[i, 0];
symMatches[i, 3] = distance1[i, 1];
count++;
break;
}
}
}
return count;
}
簡單說就是在以左邊影像為基準時，然後去找出位於右邊影像的對應點中將對應的配對結果放在trainIdx1
反之，以右邊影像為基準，找出左邊影像的配對結果放在trainIdx2
這裡要找出兩個結果中，有互相匹配的key points
假設在trainIdx1中有一組配對是左邊影像的第105的key point對應到右邊影像的第333的key point
以此為表示：105->333
而在trainIdx2中，如果有找到一組匹配是右邊影像的第333的key point對應到左邊影像的第105的key point
也就是：333->105
就把這組配對留著，其餘的就過濾
再透過這層的過濾的結果如下：
最後將symmetrical matching scheme的結果帶入RANSACTest
/// <summary> /// Identify good matches using RANSAC /// </summary> /// symmetrical matches /// keypoint1 /// keypoint2 /// the number of symmetrical matches MatrixRANSACTest(Matrix matches, VectorOfKeyPoint keyPoints1, VectorOfKeyPoint keyPoints2, int matchesNumber) { Matrix selPoints1 = new Matrix (matchesNumber, 2); Matrix selPoints2 = new Matrix (matchesNumber, 2); int selPointsIndex = 0; for (int i = 0; i < matches.Rows; i++) { if (matches[i, 0] == 0 && matches[i, 1] == 0) { continue; } //Get the position of left keypoints float x = keyPoints1[(int)matches[i, 0]].Point.X; float y = keyPoints1[(int)matches[i, 0]].Point.Y; selPoints1[selPointsIndex, 0] = x; selPoints1[selPointsIndex, 1] = y; //Get the position of right keypoints x = keyPoints2[(int)matches[i, 1]].Point.X; y = keyPoints2[(int)matches[i, 1]].Point.Y; selPoints2[selPointsIndex, 0] = x; selPoints2[selPointsIndex, 1] = y; selPointsIndex++; } Matrix fundamentalMatrix = new Matrix (3, 3); //IntPtr status = CvInvoke.cvCreateMat(1, matchesNumber, MAT_DEPTH.CV_8U); Matrix status = new Matrix (1, matchesNumber); //Compute F matrix from RANSAC matches CvInvoke.cvFindFundamentalMat( selPoints1, //points in first image selPoints2, //points in second image fundamentalMatrix, //fundamental matrix CV_FM.CV_FM_RANSAC, //RANSAC method this._Distance, //Use 3.0 for default. The parameter is used for RANSAC method only. this._Confidence, //Use 0.99 for default. The parameter is used for RANSAC or LMedS methods only. status);//The array is computed only in RANSAC and LMedS methods. if (this._RefineF) { matchesNumber = 0; for (int i = 0; i < status.Cols; i++) { if (status[0, i] == 1) { matchesNumber++; } } selPoints1 = new Matrix (matchesNumber, 2); selPoints2 = new Matrix (matchesNumber, 2); int statusIndex = -1; selPointsIndex = 0; for (int i = 0; i < matches.Rows; i++) { if (matches[i, 0] == 0 && matches[i, 1] == 0) { continue; } statusIndex++; if (status[0, statusIndex] == 1) { //Get the position of left keypoints float x = keyPoints1[(int)matches[i, 0]].Point.X; float y = keyPoints1[(int)matches[i, 0]].Point.Y; selPoints1[selPointsIndex, 0] = x; selPoints1[selPointsIndex, 1] = y; //Get the position of right keypoints x = keyPoints2[(int)matches[i, 1]].Point.X; y = keyPoints2[(int)matches[i, 1]].Point.Y; selPoints2[selPointsIndex, 0] = x; selPoints2[selPointsIndex, 1] = y; selPointsIndex++; } } status = new Matrix (1, matchesNumber); //Compute F matrix from RANSAC matches CvInvoke.cvFindFundamentalMat( selPoints1, //points in first image selPoints2, //points in second image fundamentalMatrix, //fundamental matrix CV_FM.CV_FM_RANSAC, //RANSAC method this._Distance, //Use 3.0 for default. The parameter is used for RANSAC method only. this._Confidence, //Use 0.99 for default. The parameter is used for RANSAC or LMedS methods only. status);//The array is computed only in RANSAC and LMedS methods. } return fundamentalMatrix; }
這裡一樣透過CvInvoke.cvFindFundamentalMat的方法來取出fundamental matrix
只是與之前不同的是
這是採用的是CV_FM.CV_FM_RANSAC的方法
第一次取出fundamentalMatrix後
根據status的結果
會再建立一次selPoints1、selPoints1
然後以此新的點資訊再計算一次fundamental matrix
這時候我們可以利用前幾篇的方法把key points和epipolar lines畫出來
是不是都是位在相同的物件區域上
而這些key points也都落在epipolar lines上面
而最後的結果
也從原先的SURF feature detector擷取出的一千多個key points
收斂至122個key points
而這樣的結果看起來是相當不錯的呢!!
下一篇將接續討論如何利用key points計算兩張影像的homography
沒有留言:
張貼留言