A Fast Approximation Algorithm for the Gaussian Filter Kentaro - - PowerPoint PPT Presentation

/32

A Fast Approximation Algorithm for the Gaussian Filter

Kentaro Imajo (Yamamoto Laboratory)

February 14, Master Thesis Defense

1

/32

I propose a fast approximation algorithm for the Gaussian filter, which computes

• ne Gaussian-filtered pixel

in O(1) in the area size n (∝ σ2). The naïve method requires O(n) time.

Key Algorithm

February 14, Master Thesis Defense

×

Area size is n Target Pixel

2 Introduction

/32

Application: Binary Classifier

Extending the key algorithm,

• The proposed method speeds up

prediction of SVMs with the Gaussian kernel,

• The proposed method speeds up training

with small loss of accuracy. Training (pre-computation): O(n2) → O(4dnlogdn) Prediction: O(n) → O(4dlogdn)

February 14, Master Thesis Defense

3 Introduction

/32

Outline

• 1. Key Algorithm:

A fast approximation algorithm for the Gaussian filter

Kentaro Imajo, "Fast Gaussian Filtering Algorithm Using Splines," The 21st International Conference on Pattern Recognition, 2012.

• 2. Application: Binary classification

Kentaro Imajo, Otaki Keisuke, Yamamoto Akihiro, "Binary Classification Using Fast Gaussian Filtering Algorithm,” Algorithms for Large-Scale Information Processing in Knowledge Discovery, 2012.

• 3. Conclusions

February 14, Master Thesis Defense

4 Introduction

/32

• 1. Key Algorithm: Gaussian Filtering
• 2. Application: Binary Classification
• 3. Conclusions

February 14, Master Thesis Defense

5 Key Algorithm: Gaussian Filtering

/32

Summary of Key Algorithm

It computes one Gaussian-filtered pixel in constant time in the area size n (∝ σ2) within 3% error. The Gaussian filter is convolution with the 2D Gaussian Function:

February 14, Master Thesis Defense

6

1 √ 2πσ exp

• −x2 + y2

2σ2

• =

where , the parameter σ is variance.

= ＊

convolution

Key Algorithm: Gaussian Filtering

/32

Problem

To compute one Gaussian-filtered pixel,

• The Naïve method takes linear time

in the area size of the Gaussian function.

• An FFT method cannot

compute pixels apart because FFT depends on all pixels.

• A down-sampling method has 52% error.

February 14, Master Thesis Defense

7 Key Algorithm: Gaussian Filtering

/32

Solution

We approximate the Gaussian function using a spline function:

• A spline function becomes discrete

if it is differentiated some times. – Convolution of an image and a discrete function is computed fast. A spline function is a piecewise polynomial function that is written as , where (･)+ is max(･, 0).

February 14, Master Thesis Defense

8

˜ ψ(x) =

• i

ai (x − bi)n

+

Key Algorithm: Gaussian Filtering

/32

˜ ψ(x) = 3(x + 11)2

+ 11(x + 3)2 +

+11(x 3)2

+ 3(x 11)2 +

• 2.657 102 exp
• x2

5.27202

• ,
• 15
• 12.5
• 10
• 7.5
• 5
• 2.5

2.5 5 7.5 10 12.5 15 80 160 240

Approximation Function

February 14, Master Thesis Defense

9

× × × ×

－

Approximation

－

Gaussian func.

×

Knots where (･)+ is max(･, 0).

Key Algorithm: Gaussian Filtering

/32

˜ ψ(x) = 3(x + 11)2

+ 11(x + 3)2 +

+11(x 3)2

+ 3(x 11)2 +

• 2.657 102 exp
• x2

5.27202

• ,
• 15
• 12.5
• 10
• 7.5
• 5
• 2.5

2.5 5 7.5 10 12.5 15 80 160 240

Approximation Function

February 14, Master Thesis Defense

10

× × × ×

－

Approximation

－

Gaussian func.

×

Knots where (･)+ is max(･, 0).

Key Algorithm: Gaussian Filtering

/32

Speed Up of Convolution

The convolution can be reduced by differentiating a spline function until it becomes discrete. Intuition:

February 14, Master Thesis Defense

11

(ψ ∗ I)(x) = d dxψ

• ∗
• I(x)dx
• (x)

Integrated Differentiated Becomes DISCRETE

˜ ψ

This can be computed fast.

Key Algorithm: Gaussian Filtering

/32

Nature of a Spline Function

Differentiating a spline function, we can get a discrete function: The Gaussian-like approximation function can also become a discrete function!

February 14, Master Thesis Defense

12

• 12
• 8
• 4

• 240
• 160
• 80

• 12
• 8
• 4

• 25

• 12
• 8
• 4

• 10
• 5

• 12
• 8
• 4

• 12
• 8
• 4

Differentiate 3-times

Key Algorithm: Gaussian Filtering

/32

Evaluation

Approximation error is about 2%. Higher order improves more: Approximation error of 4th order approximation is about 0.3%.

February 14, Master Thesis Defense

13

˜ ψ(x) = 70(x + 22)4

+ − 624(x + 11)4 + + 1331(x + 4)4 +

−1331(x − 4)4

+ + 624(x − 11)4 + − 70(x − 22)4 +.

(

• | ˜

ψ(x) − ψ(x)|dx) / (

• |ψ(x)|dx)

Key Algorithm: Gaussian Filtering

/32

Approximation of 2D

February 14, Master Thesis Defense

14

• 15

15 x

• 15

15 y

× × × × × × × × × × × × × × × ×

· · · Gaussian — Approximation × Control point

˜ ψ(x, y) = (3(x + 11)2

+ − 11(x + 3)2 + + 11(x − 3)2 + − 3(x − 11)2 +)

(3(y + 11)2

+ − 11(y + 3)2 + + 11(y − 3)2 + − 3(y − 11)2 +)

Contour lines

Key Algorithm: Gaussian Filtering

/32

The 2D Gaussian function is decomposed: The approximation can be defined as: where

( ˜ ψ ∗ I)(x, y) =

m

• i=1

ai

m

• j=1

ajJ(x − bi, y − bj)

2D Convolution

February 14, Master Thesis Defense

15

exp

• −x2 + y2

2σ2

• = exp
• − x2

2σ2

• exp
• − y2

2σ2

• O(m2) linear combination,

where m is # of knots for each dimension and m is equal to 4.

J(x, y) =

(∆x,∆y)∈Z2

+ ∆xn∆ynI(x − ∆x, y − ∆y)

Key Algorithm: Gaussian Filtering

/32

Experiments

February 14, Master Thesis Defense

16 100 101 102 103 104 # of application pixels of the filter 10−8 10−7 10−6 10−5 10−4 Computational time (sec.) — Proposed method - - - Na¨ ıve method

Figure Computational time for one pixel on average

For 70+ pixels

• ur algorithm is faster

Diameter of 9 has 69 pixels

Key Algorithm: Gaussian Filtering

/32

Conclusions of Part 1 (1/2)

We approximate the Gaussian function using a 2nd order spline function:

• It has only 2% approximation error,
• It has only 4 knots,
• It has unequal intervals of control points

while general approximation functions

• ften have equal intervals.

– It is sufficiently optimized as a spline.

February 14, Master Thesis Defense

17 Key Algorithm: Gaussian Filtering

/32

Conclusions of Part 1(2/2)

Using spline approximation and an integrated image of an input image, we can compute every pixel apart in constant time to the area size.

February 14, Master Thesis Defense

18

＊ = ＊ =

• …
• dx3dy3

Key Algorithm: Gaussian Filtering

/32

February 14, Master Thesis Defense

19

• 1. Key Algorithm: Gaussian Filtering
• 2. Application: Binary Classification
• 3. Conclusions

Application: Binary Classification

/32

Summary of Application

Extending the key algorithm,

• The proposed method

speeds up prediction of SVMs with the Gaussian kernel, – O(n) → O(4dlogdn)

• It speeds up training

with small loss of accuracy. – O(n2) → O(4dnlogdn)

February 14, Master Thesis Defense

Judges which is nearer to

20 Application: Binary Classification

/32

Problem

An SVM is a good binary classifier, but – prediction takes O(n) time, – training takes O(n2)～O(n3) time. We cannot use SVMs if # of data gets large.

Solution

• For prediction, we use the key algorithm.
• For training, we treat data equally

with small loss of accuracy.

February 14, Master Thesis Defense

21 Application: Binary Classification

/32

Every pixel has a value Few pixels have a value

Key Algorithm

It assumes that an image is given. Since we cannot integrate an image, we propose to use range trees.

February 14, Master Thesis Defense

22

Image Binary Classification

Application: Binary Classification

/32

Range Tree

A range tree [Bentley 1979] is a data structure that can compute the total sum in a box x1-x2 in O(logdn) time with O(nlogdn) space.

February 14, Master Thesis Defense

23 Application: Binary Classification

2k Balanced tree We can calculate the total sum with the sum

• f O(logdn) regions

x1 x2

/32

The Proposed Method

The proposed method can decide the class of any point in O(4dlogdn) time, and the pre-computation is O(4dnlogdn) time.

February 14, Master Thesis Defense

24

Range Trees

• 15

15

× × × × × × × × × × × × × × × ×

Key Algorithm

Application: Binary Classification

/32

Formulation

A range tree can give: Using 9 range trees, J can be computed with:

February 14, Master Thesis Defense

25

Ki,j(x, y) =

• (∆x,∆y)∈{(0,0),···,(x,y)}

∆xi∆yjI(∆x, ∆y).

J(x, y) =   y2 −2y 1  

T 

 K0,0(x, y) K1,0(x, y) K2,0(x, y) K0,1(x, y) K1,1(x, y) K2,1(x, y) K0,2(x, y) K1,2(x, y) K2,2(x, y)     x2 −2x 1   .

Application: Binary Classification

/32

Experiments on Time

Time consumption of prediction: The proposed method is faster when # of training data is over 150,000.

February 14, Master Thesis Defense

26

# of training data 1, 000 10, 000 100, 000 1, 000, 000 Proposed Prediction 2.43 ms 2.48 ms 2.55 ms 2.96 ms method (Precomputation) (0.13 s) (1.48 s) (17.8 s) (203 s) LIBSVM Prediction 0.02 ms 0.22 ms 2.29 ms 23.6 ms (Training) (0.05 s) (4.47 s) (442 s) (1000+ s)

Table Time of Training and Prediction for 2D Data

Application: Binary Classification

/32

Experiments on Accuracy

If we treat input data equally, accuracy of a new classifier and LIBSVM: The loss of accuracy is small.

February 14, Master Thesis Defense

27

Proposed C-SVM Method C = 0.1 C = 1 C = 10 C = 106 σ = 0.1 93% 58% 68% 69% 69% σ = 0.3 93% 83% 94% 93% 93% σ = 1 93% 93% 95% 93% 92% σ = 3 86% 90% 94% 95% 89% σ = 10 84% 85% 90% 93% 92%

Table Accuracy for Iris Flower Data Set

Application: Binary Classification

/32

Conclusions of Part 2

• It speeds up prediction of SVM:

O(n) → O(4dlogdn).

• With small loss of accuracy,

it speeds up training: O(n2) → O(4dlogdn). The proposed method consists of

• The key algorithm: A fast approximation

algorithm for the Gaussian filter,

• Range trees.

February 14, Master Thesis Defense

28 Application: Binary Classification

/32

February 14, Master Thesis Defense

29

• 1. Key Algorithm: Gaussian Filtering
• 2. Application: Binary Classification
• 3. Conclusions

Conclusions

/32

Conclusions

Key Algorithm: A fast approximation algorithm for the Gaussian filter:

• It uses a spline function for approximation.
• It speeds up Gaussian filtering,

which is often used in image processing. Application: Binary classification

• It extends the key algorithm with range trees.
• It speeds up prediction of SVMs.
• It speeds up training with small loss of accuracy.

February 14, Master Thesis Defense

30 Conclusions

/32

Related Achievements

International Conference (Reviewed):

• Kentaro Imajo, Otaki Keisuke, Yamamoto Akihiro,

"Binary Classification Using Fast Gaussian Filtering Algorithm,” ALSIP 2012.

• Kentaro Imajo,

"Fast Gaussian Filtering Algorithm Using Splines,” ICPR 2012.

February 14, Master Thesis Defense

31 Conclusions

/32

Other Achievements

International Conference (Reviewed):

• Takuya Akiba, Kentaro Imajo, Daisuke Okanohara,

"Engineering Parallel String Sorting Algorithms,” Algorithms for Large-Scale Information Processing in Knowledge Discovery, 2011.

Journal:

• Kentaro Imajo,

"Contest Environment Using Wireless Networks: A Case Study From Japan,” Olympiad in Informatics 5, 26―31, 2011.

• Mahito Sugiyama, Kentaro Imajo, Keisuke Otaki, Akihiro Yamamoto,

"Semi-Supervised Ligand Finding Using Formal Concept Analysis,” IPSJ Transactions on Mathematical Modeling and Its Applications 5(2), 39―48, 2012.

Programming Contests:

• ICFP Programming Contest, the 2nd place, September 2011.
• ACM/ICPC Asia Regional Contest in Fukuoka, Japan, the 5th place, November 2011.
• ACM/ICPC Asia Regional Contest in Hsinchu, Taiwan, the 3rd place, November 2011.
• ACM/ICPC World Finals 2012, an 18th place tie, May 2012.

February 14, Master Thesis Defense

32 Conclusions