PIP - Perspective of Image Processing Introduction SIFT - - PowerPoint PPT Presentation

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

PIP - Perspective of Image Processing

Rishidhar Reddy Bommu, Bryan Humphreys, Aerin Kim, Sombo Koo Team Leader: Dhatri Chennavajula Project Director: Roger Dodd CAMCOS Director: Slobodan Simi´ c

San Jos´ e State University

May 14th, 2015

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Outline

Introduction to Image Processing

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Outline

Introduction to Image Processing The Scale - Invariant Feature Transform (SIFT)

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Outline

Introduction to Image Processing The Scale - Invariant Feature Transform (SIFT) Applications of Feature Description

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Outline

Introduction to Image Processing The Scale - Invariant Feature Transform (SIFT) Applications of Feature Description Future Work

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Introduction to Image Processing

Q: What is an Image?

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Introduction to Image Processing

Q: What is an Image? − →       11 11 12 12 23 12 14 23 22 21 22 20 23 24 25 33 34 36 33 33 32 31 10 10 15       A: It’s an array of numbers

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Image Properties

The smallest single component of a digital image is called a Pixel (Pixel = Picture Element) .

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Image Properties

The smallest single component of a digital image is called a Pixel (Pixel = Picture Element) . The resolution of an image is the number of pixels.

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Image Properties

The smallest single component of a digital image is called a Pixel (Pixel = Picture Element) . The resolution of an image is the number of pixels. e.g. A 12MP camera has a nominal 12,000,000 pixels.

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Image Properties

The smallest single component of a digital image is called a Pixel (Pixel = Picture Element) . The resolution of an image is the number of pixels. e.g. A 12MP camera has a nominal 12,000,000 pixels. Image processing becomes more expensive as the resolution of the image increases.

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

What about images?

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

What about images?

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Differentiation of Images

Given the following function, we can differentiate this function in three different ways, D←f (x) = f (x) − f (x − ∆x) ∆x

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Differentiation of Images

Given the following function, we can differentiate this function in three different ways, D←f (x) = f (x) − f (x − ∆x) ∆x D→f (x) = f (x + ∆x) − f (x) ∆x

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Differentiation of Images

Given the following function, we can differentiate this function in three different ways, D←f (x) = f (x) − f (x − ∆x) ∆x D→f (x) = f (x + ∆x) − f (x) ∆x DCf (x) = f (x + ∆x) − f (x − ∆x) ∆x

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Example of a Mask

1 2 3 4 5 6 1 2 3 4 5

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Example of a Mask

1 2 3 4 5 6 1 2 3 4 5

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Example of a Mask

1 2 3 4 5 6 1 2 3 4 5 Let ∆x = 1, compute Df←(3), Df→(3), DfC(3)

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Sobel Edge Detection

Given an Image I(x, y), we want to extract where the changes

• ccur on the image. So what we do is convolve G(x, y) with

an operator, ∇G(x, y), Gx(x, y) =   −1 1 −2 2 −1 1   Gy(x, y) =   −1 −2 −1 1 2 1   |∇G(x, y)| =

• (Gx(x, y))2 + (Gy(x, y))2

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

contd...

      1 1 2 2 2 1 1 2 2 2 1 1 2 2 2 1 1 2 2 2 1 1 2 2 2      

Gx(x,y)

− − − − →       4 4 4 4 4 4             1 1 2 2 2 1 1 2 2 2 1 1 2 2 2 1 1 2 2 2 1 1 2 2 2      

Gy(x,y)

− − − − →            

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Feature Detection

A feature is defined to be an “interesting” part of an image (edges, intensity changes, curves, lines, corners). There are many different interesting feature detection methods:

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Feature Detection

A feature is defined to be an “interesting” part of an image (edges, intensity changes, curves, lines, corners). There are many different interesting feature detection methods: Edge Detection: Sobel, Canny, Canny-Deriche, Differential, Prewitt, Roberts cross

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Feature Detection

A feature is defined to be an “interesting” part of an image (edges, intensity changes, curves, lines, corners). There are many different interesting feature detection methods: Edge Detection: Sobel, Canny, Canny-Deriche, Differential, Prewitt, Roberts cross Blob Detection: Difference of Gaussian, Laplacian of Gaussian, Determinant of Hessian

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Feature Detection

A feature is defined to be an “interesting” part of an image (edges, intensity changes, curves, lines, corners). There are many different interesting feature detection methods: Edge Detection: Sobel, Canny, Canny-Deriche, Differential, Prewitt, Roberts cross Blob Detection: Difference of Gaussian, Laplacian of Gaussian, Determinant of Hessian Line Detection: Hough Transform

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Feature Detection

A feature is defined to be an “interesting” part of an image (edges, intensity changes, curves, lines, corners). There are many different interesting feature detection methods: Edge Detection: Sobel, Canny, Canny-Deriche, Differential, Prewitt, Roberts cross Blob Detection: Difference of Gaussian, Laplacian of Gaussian, Determinant of Hessian Line Detection: Hough Transform Corner Detection: Harris

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Formulation of the Problem

Scaling − →

Introduction

Outline Image Processing Edges Feature Detection

SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Formulation of the Problem

Scaling − → Rotation − →

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

What is SIFT?

Scale Invariant Feature Transform is an algorithm used to describe local features of an image.

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

What is SIFT?

Scale Invariant Feature Transform is an algorithm used to describe local features of an image. Published by David Lowe

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

What is SIFT?

Scale Invariant Feature Transform is an algorithm used to describe local features of an image. Published by David Lowe U.S. Patent and owned by University of British Colombia.

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

What is SIFT?

Scale Invariant Feature Transform is an algorithm used to describe local features of an image. Published by David Lowe U.S. Patent and owned by University of British Colombia. A keypoint detector and descriptor algorithm that is invariant to rotation and scale.

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Detectors, Descriptors, Matching

Feature Detectors are algorithms that find the interesting points of an image: edges, corners, blobs.

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Detectors, Descriptors, Matching

Feature Detectors are algorithms that find the interesting points of an image: edges, corners, blobs. Feature Descriptors are algorithms that take an image, and describes each feature as a vector.

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Detectors, Descriptors, Matching

Feature Detectors are algorithms that find the interesting points of an image: edges, corners, blobs. Feature Descriptors are algorithms that take an image, and describes each feature as a vector. SIFT is an example of a feature descriptor.

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Detectors, Descriptors, Matching

Feature Detectors are algorithms that find the interesting points of an image: edges, corners, blobs. Feature Descriptors are algorithms that take an image, and describes each feature as a vector. SIFT is an example of a feature descriptor. Honorable Mention: Speeded-Up Robust Feature (SURF)

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Scale Space

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Gaussian Blur

Gaussian Kernel: g(x, y, σ) = 1 2πσ2 e−(x2+y2)/2σ L(x, y, σ) is the “improved” image. We improve the image by convolving the original image with a Gaussian kernel. In other words, L(x, y, σ) = g(x, y, σ) ∗ I(x, y)

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Difference of Gaussian (DoG)

Laplacian of Gaussian (LoG): ∆g(x, y, σ) = ∂2 ∂x2 g(x, y, σ) + ∂2 ∂y2 g(x, y, σ) We approximate the LoG with finite differencing. This is called Difference of Gaussian (DoG). This is given via ∆g(x, y, σ) = g(x, y, kσ) − g(x, y, σ) σ(k − 1)

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Keypoint Example

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Keypoint Descriptor

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Description

A 16 × 16 array around our keypoint is how we calculate

• ur orientation histograms around the feature (edge,

corner, blob). We calculate 8 gradient orientation, in 16 total windows = 128 features. Each gradient is subtracted from the overall orientation. Since this difference is invariant under any permutation of the sub arrays, as a result, feature description is invariant under rotation.

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Matching

We utilize the Euclidean distance to match feature descriptors.

Introduction SIFT

Scale Space Gaussian Blur Difference of Gaussian Matching

Implementation

• f Code

Neural Networks K-Means Color Cluster

Matching

We utilize the Euclidean distance to match feature descriptors. If the ratio of the closest feature desriptor is 0.8, then we consider that a match.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Open Course Computer Vision

Open Course Computer Vision (OpenCV) Library

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Open Course Computer Vision

Open Course Computer Vision (OpenCV) Library Launched in 1999.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Open Course Computer Vision

Open Course Computer Vision (OpenCV) Library Launched in 1999. Developed by Intel Russia reasearch center in Nizhny Novgorod.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

OpenCV Facts

OpenCV is FREE and Open Source, BSD license.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

OpenCV Facts

OpenCV is FREE and Open Source, BSD license. An open source library of over 500 API’s

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

OpenCV Facts

OpenCV is FREE and Open Source, BSD license. An open source library of over 500 API’s Its interface is written in C++, which makes it an easy development tool for implementing/experimenting computer vision.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

OpenCV Facts

OpenCV is FREE and Open Source, BSD license. An open source library of over 500 API’s Its interface is written in C++, which makes it an easy development tool for implementing/experimenting computer vision. 45k member usergroup.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

OpenCV Facts

OpenCV is FREE and Open Source, BSD license. An open source library of over 500 API’s Its interface is written in C++, which makes it an easy development tool for implementing/experimenting computer vision. 45k member usergroup. 3.5 million downloads

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

OpenCV Facts

OpenCV is FREE and Open Source, BSD license. An open source library of over 500 API’s Its interface is written in C++, which makes it an easy development tool for implementing/experimenting computer vision. 45k member usergroup. 3.5 million downloads Supported OS systems include Windows, Linux, Android, iOS platforms.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

OpenCV Facts

OpenCV is FREE and Open Source, BSD license. An open source library of over 500 API’s Its interface is written in C++, which makes it an easy development tool for implementing/experimenting computer vision. 45k member usergroup. 3.5 million downloads Supported OS systems include Windows, Linux, Android, iOS platforms. Supported by WillowGarage, Nvidia, and Google.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

What OpenCV can do?

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Feature Detection with SIFT

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Feature Detection with RANSAC

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Homography with RANSAC

A homography is a linear transformation of a pixel point in one image to a pixel point in another image.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Homography with RANSAC

A homography is a linear transformation of a pixel point in one image to a pixel point in another image. Random Sample Consensus (RANSAC) is an iterative method to estimate parameters of a mathematical model from a set of observed data which contains outliers.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

RANSAC

A fitted line with RANSAC; the outliers have no influence on the result.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

contd...

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Drawbacks of RANSAC

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Pedestrian detection using HOG

Histogram of Orient Gradients (HOG) is a feature descriptor.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Pedestrian detection using HOG

Histogram of Orient Gradients (HOG) is a feature descriptor. Unlike SIFT, HOG is a global descriptor

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Pedestrian detection using HOG

Histogram of Orient Gradients (HOG) is a feature descriptor. Unlike SIFT, HOG is a global descriptor HOG person detector was introduced by Dalal and Triggs at the CVR conference in 2005.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Pedestrian detection using HOG

Histogram of Orient Gradients (HOG) is a feature descriptor. Unlike SIFT, HOG is a global descriptor HOG person detector was introduced by Dalal and Triggs at the CVR conference in 2005. Dalal and Triggs trained a Support Vector Machine (SVM) to recognize HOG descriptors of people.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Pedestrian detection using HOG

Histogram of Orient Gradients (HOG) is a feature descriptor. Unlike SIFT, HOG is a global descriptor HOG person detector was introduced by Dalal and Triggs at the CVR conference in 2005. Dalal and Triggs trained a Support Vector Machine (SVM) to recognize HOG descriptors of people. SVMs are among the best “off-the-shelf” supervised learning algorithms.

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

contd..

Introduction SIFT Implementation

• f Code

OpenCV Feature Detection with SIFT RANSAC

Neural Networks K-Means Color Cluster

Pedestrian detection using HOG

Introduction SIFT Implementation

• f Code

Neural Networks

SVM

K-Means Color Cluster

Neural Networks

Introduction SIFT Implementation

• f Code

Neural Networks

SVM

K-Means Color Cluster

Support Vector Machine

D = {(xi, yi) : xi ∈ Rn, yi ∈ {−1, 1}}

Introduction SIFT Implementation

• f Code

Neural Networks

SVM

K-Means Color Cluster

Latent SVM

score(model, x) = score(root, x) +

• p∈Parts

max

p [score(model, y) − cost(p, x, y)]

Introduction SIFT Implementation

• f Code

Neural Networks

SVM

K-Means Color Cluster

Rishi’s Video

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

K-means Color Clustering

Partition n data points into K clusters.

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

K-means Color Clustering

Partition n data points into K clusters. Reduce the number of distinct colors in an image.

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

K-means Color Clustering

Partition n data points into K clusters. Reduce the number of distinct colors in an image. Each pixel will be considered a data point.

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

K-means Color Clustering

Partition n data points into K clusters. Reduce the number of distinct colors in an image. Each pixel will be considered a data point. The increased size in pixels is directly proportional to the number of clusters. As a result, this takes up computing time.

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

K-means Color Clustering

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

9 - Means Color Clustering

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

9 - Means Color Clustering

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Conclusion

Keypoint descriptors such as SIFT and HOG have different uses.

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Conclusion

Keypoint descriptors such as SIFT and HOG have different uses. Various Illumination Factors

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Conclusion

Keypoint descriptors such as SIFT and HOG have different uses. Various Illumination Factors Affine Transformations

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Conclusion

Keypoint descriptors such as SIFT and HOG have different uses. Various Illumination Factors Affine Transformations No theory of everything, yet.

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Acknowledgements

Professor Roger Dodd

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Acknowledgements

Professor Roger Dodd Professor Chang Choo and his team.

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Acknowledgements

Professor Roger Dodd Professor Chang Choo and his team. Professor Slobodan Simi´ c

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Acknowledgements

Professor Roger Dodd Professor Chang Choo and his team. Professor Slobodan Simi´ c Volkswagen - Robert Casey

Introduction SIFT Implementation

• f Code

Neural Networks K-Means Color Cluster

Acknowledgements

Professor Roger Dodd Professor Chang Choo and his team. Professor Slobodan Simi´ c Volkswagen - Robert Casey Everyone that came!