Robust Feature Matching and Fast GMS Solution Singapore University - - PowerPoint PPT Presentation

▶
robust feature matching and
SMART_READER_LITE
LIVE PREVIEW

Robust Feature Matching and Fast GMS Solution Singapore University - - PowerPoint PPT Presentation

Mar 09, 2023 155 likes •731 views

Robust Feature Matching and Fast GMS Solution Singapore University of Technology and Design (SUTD) Advanced Digital Sciences Center (ADSC) Speaker JiaWang Bian( ) http://jwbian.net/ June 14,2017 6/14/2017 Robust Feature Matching

slide-1
SLIDE 1

Robust Feature Matching and Fast GMS Solution 6/14/2017 1/57

Robust Feature Matching and Fast GMS Solution

Singapore University of Technology and Design (SUTD) Advanced Digital Sciences Center (ADSC)

Speaker：JiaWang Bian(边佳旺) http://jwbian.net/

June 14,2017

slide-2
SLIDE 2

Robust Feature Matching and Fast GMS Solution 6/14/2017 2/57

  • Feature Matching Introduction
  • Feature Matching
  • Feature Detector & Descriptor
  • Matching
  • RANSAC-based Geometry Estimation (or Verification)
  • Recent Robust Matchers
  • CODE (PAMI,2016)
  • RepMatch (ECCV,2016)
  • Fast and Robust GMS Solution(CVPR,2017)
  • Video Demo
  • Methodology
  • Algorithm
  • Share (Material Links)

Content

slide-3
SLIDE 3

Robust Feature Matching and Fast GMS Solution 6/14/2017 3/57

Feature Matching Introduction

slide-4
SLIDE 4

Robust Feature Matching and Fast GMS Solution 6/14/2017 4/57

Feature Matching Introduction

  • Feature Matching
  • Pipeline

Detection Description Matching Geometry

slide-5
SLIDE 5

Robust Feature Matching and Fast GMS Solution 6/14/2017 5/57

Feature Matching Introduction

  • Applications

Correct Correspondences Geometry between 2 views Similarity(Number of matches) Image retrieval Object Recognition Loop Closing (SLAM) Re-localization (SLAM) … Estimate Camera Pose Localization (SFM) Tracking (SLAM) …

slide-6
SLIDE 6

Robust Feature Matching and Fast GMS Solution 6/14/2017 6/57

Sparse Feature Matching

  • Feature detector & descriptor

SIFT SURF, ORB, AKAZE, … PCA-SIFT, ASIFT, LIFT, … Faster Better

slide-7
SLIDE 7

Robust Feature Matching and Fast GMS Solution 6/14/2017 7/57

Feature Matching Introduction

  • Matching

Matching Nearest-Neighbor Optimization Graph Matching Others Brute-Force Approximate(FLANN) Matching Algorithms

CODE, RepMatch, GMS…

slide-8
SLIDE 8

Robust Feature Matching and Fast GMS Solution 6/14/2017 8/57

Feature Matching Introduction

  • RANSAC-based Geometry Estimation (or Verification)
  • An example for RANSAC framework (fitting a line)
slide-9
SLIDE 9

Robust Feature Matching and Fast GMS Solution 6/14/2017 9/57

Feature Matching Introduction

  • RANSAC-based Geometry Estimation (or Verification)
  • Fundamental Matrix (for 3D scenes)
  • Point to Line (weak, general)
  • Homography (for 2D scenes)
  • Point to Point (strong, narrow range)
slide-10
SLIDE 10

Robust Feature Matching and Fast GMS Solution 6/14/2017 10/57

Recent Robust Matchers

slide-11
SLIDE 11

Robust Feature Matching and Fast GMS Solution 6/14/2017 11/57

Recent Robust Matchers

  • CODE[1]
  • For wide-baseline matching.
  • RepMatch[2]
  • Based on CODE[1].
  • Solve the repeated structure problem.

[1] CODE: Coherence Based Decision Boundaries for Feature Correspondence, IEEE TPAMI,2016, Lin et. al. [2] RepMatch: Robust Feature Matching and Pose for Reconstructing Modern Cities, ECCV, 2016,, Lin et. al.

slide-12
SLIDE 12

Robust Feature Matching and Fast GMS Solution 6/14/2017 12/57

Recent Robust Matchers (CODE)

  • Wide-baseline matching
slide-13
SLIDE 13

Robust Feature Matching and Fast GMS Solution 6/14/2017 13/57

Recent Robust Matchers (CODE)

  • Idea
slide-14
SLIDE 14

Robust Feature Matching and Fast GMS Solution 6/14/2017 14/57

Recent Robust Matchers (CODE)

  • Regression models
  • Likelihood Regression
  • Affine motion regression -> x
  • Affine motion regression -> y
slide-15
SLIDE 15

Robust Feature Matching and Fast GMS Solution 6/14/2017 15/57

Recent Robust Matchers (CODE)

  • Likelihood Regression
  • Train Data
  • Selected distinctive correspondences(after ratio-test).
  • Test Data
  • All feature correspondences.
  • Features of a correspondence
  • 𝑌𝑗 = [𝑦, 𝑧, 𝑒𝑦, 𝑒𝑧, 𝑈1, 𝑈2, 𝑈3, 𝑈4].
  • T is a transformation matrix of [s1, r1] to [s2, r2].
  • s means scale, r represents rotation.
  • Labels
  • 1 for all correspondences
  • Cost function
  • Huber function
  • Non-linear Optimization
  • Construct Gaussian Similar Matrix
  • X(Matrix with n x n elements), Y(Matrix with nx1 elements(1) )
  • n is the number of train data
slide-16
SLIDE 16

Robust Feature Matching and Fast GMS Solution 6/14/2017 16/57

Recent Robust Matchers (CODE)

  • Affine motion regression
  • Train Data
  • The inliers of train data in the likelihood model
  • Test Data
  • Correspondences filtered by the likelihood model
  • Feature Space
  • Same as the likelihood model
  • Label
  • X2, and y2.(x,y represents pixel position, 2 means the second image)
  • Cost function
  • Huber function
  • Non-linear Optimization
  • Same as before(Gaussian Similar Matrix).
slide-17
SLIDE 17

Robust Feature Matching and Fast GMS Solution 6/14/2017 17/57

Recent Robust Matchers (CODE)

  • Insight (likelihood model)
slide-18
SLIDE 18

Robust Feature Matching and Fast GMS Solution 6/14/2017 18/57

Recent Robust Matchers (CODE)

  • Matching samples
slide-19
SLIDE 19

Robust Feature Matching and Fast GMS Solution 6/14/2017 19/57

Recent Robust Matchers (CODE)

  • Structure from Motion
  • C. Wu, “VisualSfM: A visual structure from motion system,” 2011[Online]. Available: http://ccwu.me/vsfm/
slide-20
SLIDE 20

Robust Feature Matching and Fast GMS Solution 6/14/2017 20/57

Recent Robust Matchers (CODE)

  • Run time comparison
slide-21
SLIDE 21

Robust Feature Matching and Fast GMS Solution 6/14/2017 21/57

Recent Robust Matchers (RepMatch)

  • RepMatch
slide-22
SLIDE 22

Robust Feature Matching and Fast GMS Solution 6/14/2017 22/57

Recent Robust Matchers (RepMatch)

  • Repetitive Structure
slide-23
SLIDE 23

Robust Feature Matching and Fast GMS Solution 6/14/2017 23/57

Recent Robust Matchers (RepMatch)

  • Idea
slide-24
SLIDE 24

Robust Feature Matching and Fast GMS Solution 6/14/2017 24/57

Recent Robust Matchers (RepMatch)

  • Structure from Motion
slide-25
SLIDE 25

Robust Feature Matching and Fast GMS Solution 6/14/2017 25/57

Recent Robust Matchers (RepMatch)

  • Structure from Motion
slide-26
SLIDE 26

Robust Feature Matching and Fast GMS Solution 6/14/2017 26/57

Recent Robust Matchers (RepMatch)

  • Structure from Motion
slide-27
SLIDE 27

Robust Feature Matching and Fast GMS Solution 6/14/2017 27/57

Recent Robust Matchers (RepMatch)

  • Structure from Motion
slide-28
SLIDE 28

Robust Feature Matching and Fast GMS Solution 6/14/2017 28/57

Fast and Robust GMS Solution

slide-29
SLIDE 29

Robust Feature Matching and Fast GMS Solution 6/14/2017 29/57

Video Demo

  • ORB with GMS vs SIFT with Ratio
slide-30
SLIDE 30

Robust Feature Matching and Fast GMS Solution 6/14/2017 30/57

Motivation: Trade-off of quality and speed

  • Trade-off

Matching Nearest-Neighbor Optimization Graph Matching Ratio test Current Methods GMS

Popular, Fast, Non-Robust Slow, Robust Fast, Robust

slide-31
SLIDE 31

Robust Feature Matching and Fast GMS Solution 6/14/2017 31/57

Methodology: Motion Smoothness

  • Observation
  • True matches(green) are visually smooth while false

matches(cyan) are not.

slide-32
SLIDE 32

Robust Feature Matching and Fast GMS Solution 6/14/2017 32/57

Methodology: Key idea

  • Inference
  • According to the Bayesian rule, as true matches are smooth in

motion space, consistent matches are thus more likely to be true.

  • Key idea
  • Find smooth matches from noisy data as our proposals.
  • Method

Motion Statistics Grid Framework Motion Kernels

slide-33
SLIDE 33

Robust Feature Matching and Fast GMS Solution 6/14/2017 33/57

Methodology: Motion Statistics

  • Motion Statistics Model
slide-34
SLIDE 34

Robust Feature Matching and Fast GMS Solution 6/14/2017 34/57

Methodology: Motion Statistics

  • Distribution
  • Let 𝑔𝑏 be one of the n supporting features in region 𝑏
  • Let 𝑞𝑢 , 𝑞𝑔 be the probability that, feature fa’s nearest

neighbor is in region 𝑐, given {𝑏, 𝑐} view the same and different location, respectively,

slide-35
SLIDE 35

Robust Feature Matching and Fast GMS Solution 6/14/2017 35/57

Methodology: Motion Statistics

  • Event
  • Assumption

Here, 𝑛 is the number of features in region 𝑐 and 𝑁 is the number

  • f features in second image. 𝛾 is a factor added to accommodate

violations of assumption caused by repeated patterns.

slide-36
SLIDE 36

Robust Feature Matching and Fast GMS Solution 6/14/2017 36/57

Methodology: Motion Statistics

  • Probability

Explanation: If {𝑏 𝑐} view the different location, event

𝑔

𝑏 𝑐 occurs only when 𝑔𝑏 matches wrongly and coincidentally

lands in region 𝑐. Explanation: If {𝑏 𝑐} view the same location, event 𝑔

𝑏 𝑐 occurs

when 𝑔𝑏 matches correctly or it matches wrongly but coincidentally lands in region 𝑐.

slide-37
SLIDE 37

Robust Feature Matching and Fast GMS Solution 6/14/2017 37/57

Methodology: Motion Statistics

  • Multi-region Generalization
slide-38
SLIDE 38

Robust Feature Matching and Fast GMS Solution 6/14/2017 38/57

Methodology: Motion Statistics

  • Distribution
  • Mean & Variance
slide-39
SLIDE 39

Robust Feature Matching and Fast GMS Solution 6/14/2017 39/57

Methodology: Motion Statistics

  • Analysis
  • Partionability
  • Quantity-Quality equivalence:
  • Relationship to Descriptors:
slide-40
SLIDE 40

Robust Feature Matching and Fast GMS Solution 6/14/2017 40/57

Methodology: Motion Statistics

  • Experiments on real data:

The model is evaluated on Oxford Affine Dataset. Here, we run SIFT matching and label all matches as inlier or outlier according to the ground truth. we count the supporting score for each match in a small region.

slide-41
SLIDE 41

Robust Feature Matching and Fast GMS Solution 6/14/2017 41/57

Algorithm: Grid Framework

  • Grid Framework
  • Both images are segmented by a pre-defined grid.
  • Calculating the Motion Statistics for cell-pairs instead of each

feature correspondence. O(N)  O(1)!

slide-42
SLIDE 42

Robust Feature Matching and Fast GMS Solution 6/14/2017 42/57

Algorithm: Motion Kernels

  • Basic Motion Kernel
slide-43
SLIDE 43

Robust Feature Matching and Fast GMS Solution 6/14/2017 43/57

Algorithm: Motion Kernels

  • Generalized Motion Kernels (Extension*)
  • Rotation
  • Scale
  • Varying the cell size of the second image by a scale factor.
slide-44
SLIDE 44

Robust Feature Matching and Fast GMS Solution 6/14/2017 44/57

Algorithm: Empirical parameters

  • How many grid-cells should be used?
  • Too fine: weak statistics and low efficiency.
  • Too coarse: low accuracy
  • The empirical results show 20 x 20 is a good choice.
  • How to set the threshold?
slide-45
SLIDE 45

Robust Feature Matching and Fast GMS Solution 6/14/2017 45/57

Algorithm: GMS

  • Grid Motion Statistics Algorithm
slide-46
SLIDE 46

Robust Feature Matching and Fast GMS Solution 6/14/2017 46/57

Algorithm: Full Feature Matching

  • Full feature matching pipeline
slide-47
SLIDE 47

Robust Feature Matching and Fast GMS Solution 6/14/2017 47/57

Algorithm: Run time

  • Run time on Image pairs
  • ORB feature extraction(about 35ms on cpu)
  • Nearest Neighbor Matching(106ms on cpu, 25ms on gpu)
  • GMS(1ms on cpu)
  • Overall : 1000 / (2 * 35 + 25 + 1) = 10.42fps
  • Real time on Video data
  • ORB and NN can run parallelly on video sequence.
  • Overall : 1000 / 35 = 28.57fps
slide-48
SLIDE 48

Robust Feature Matching and Fast GMS Solution 6/14/2017 48/57

Evaluation

  • Dataset
  • Capture of TUM dataset
slide-49
SLIDE 49

Robust Feature Matching and Fast GMS Solution 6/14/2017 49/57

Evaluation

  • Capture of Strecha dataset
  • Capture of VGG dataset
slide-50
SLIDE 50

Robust Feature Matching and Fast GMS Solution 6/14/2017 50/57

Evaluation

  • Matching ability
slide-51
SLIDE 51

Robust Feature Matching and Fast GMS Solution 6/14/2017 51/57

Evaluation

  • Pose Estimation
slide-52
SLIDE 52

Robust Feature Matching and Fast GMS Solution 6/14/2017 52/57

Evaluation

  • Wide-baseline matching

In both graphs raphs, the first row shows initial results and the second row illustrates illustrates the the matches matches after after RANSAC. RANSAC.

slide-53
SLIDE 53

Robust Feature Matching and Fast GMS Solution 6/14/2017 53/57

Evaluation

  • GMS on Images with Repetitive Structures

Images are selected by [1], where many state-of

  • f-art matchers fail and SIFT

fails all.

[1] Epipolar Geometry Estimation for Urban Scenes with Repetitive Structures, IEEE TPAMI, 2014, Kushnir et. al.

slide-54
SLIDE 54

Robust Feature Matching and Fast GMS Solution 6/14/2017 54/57

Evaluation

  • Non-rigid object
slide-55
SLIDE 55

Robust Feature Matching and Fast GMS Solution 6/14/2017 55/57

Evaluation

  • Video Demo(screen shot)
slide-56
SLIDE 56

Robust Feature Matching and Fast GMS Solution 6/14/2017 56/57

Share

  • JiaWang’s Home Page
  • http://jwbian.net/
  • Project Page
  • http://jwbian.net/gms/
  • Code on GitHub:
  • https://github.com/JiawangBian/GMS-Feature-Matcher
  • Videos on YouTube:
  • https://youtu.be/3SlBqspLbxI
  • Links to CODE and RepMatch
  • http://www.kind-of-works.com/
slide-57
SLIDE 57

Robust Feature Matching and Fast GMS Solution 6/14/2017 57/57

Q&A

Q&A