Efficient Pattern Recognition Algorithm Including a Fast Retina - - PowerPoint PPT Presentation

▶
efficient pattern recognition algorithm including a
SMART_READER_LITE
LIVE PREVIEW

Efficient Pattern Recognition Algorithm Including a Fast Retina - - PowerPoint PPT Presentation

May 16, 2023 1.03k likes •1.35k views

Efficient Pattern Recognition Algorithm Including a Fast Retina Keypoint FPGA Implementation Lester Kalms Maximilian Hajduk Diana Ghringer Technische Universitt Technische Universitt Technische Universitt Dresden, Germany Dresden,

slide-1
SLIDE 1

Efficient Pattern Recognition Algorithm Including a Fast Retina Keypoint FPGA Implementation

Lester Kalms

Technische Universität Dresden, Germany

lester.kalms@tu-dresden.de

Maximilian Hajduk

Technische Universität Dresden, Germany

maximilian.hajduk@tu-dresden.de

Diana Göhringer

Technische Universität Dresden, Germany

diana.goehringer@tu-dresden.de

slide-2
SLIDE 2

09.09.2019 Lester Kalms

Content

  • 1. Motivation & Overview
  • 2. Pattern Recognition Algorithm
  • 3. HW Implementation of FREAK
  • 4. Evaluation
  • 5. Conclusion & Outlook

2

slide-3
SLIDE 3

Motivation

What is pattern recognition?

  • Feature detector: finds points of interest in images (e.g. corners, edges or blobs)
  • Feature descriptor: describes features to be comparable with others

Why pattern recognition?

  • Still algorithms make use of pattern recognition (e.g. object tracking or SLAM)
  • Most neural networks need a high amount of input data

09.09.2019 Lester Kalms 3

slide-4
SLIDE 4

Overview

First full FPGA implementation of the FREAK descriptor

  • Related work does not contain orientation computation
  • Contains optimized pattern generator (partially pre-computed)
  • Frequency optimized integral image with parallel prefix sum

Improved pattern recognition algorithm

  • Compared to different combinations of AKAZE, ORB, BRISK and FREAK
  • Improved and combined AKAZE detector & FREAK descriptor
  • Added a Retain Best function

09.09.2019 Lester Kalms 4

slide-5
SLIDE 5

09.09.2019 Lester Kalms 5

SW Implementation

AKAZE feature detection & FREAK feature description

slide-6
SLIDE 6

HW Implementation

Integral Image Function

SW: 𝐽𝑦,𝑧 = 𝑗𝑦,𝑧 + 𝐽x−1,y + 𝐽x,y−1 − 𝐽x−1,y−1 HW: 𝑡 = 𝑡 + 𝑗𝑦,𝑧 𝑏 = 𝑧 > 0 → 𝑡 + 𝑐𝑣𝑔

𝑦

˄ [ 𝑧 ≤ 0 → 𝑡] 𝐽𝑦,𝑧 = 𝑐𝑣𝑔

𝑦 = 𝑏

1 2 3 4 2 4 6 8 3 6 9 12 4 8 12 16 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Input Image Output Image

09.09.2019 Lester Kalms 6

slide-7
SLIDE 7

HW Implementation

Integral Image Function

SW: 𝐽𝑦,𝑧 = 𝑗𝑦,𝑧 + 𝐽x−1,y + 𝐽x,y−1 − 𝐽x−1,y−1 HW: 𝑡 = 𝑡 + 𝑗𝑦,𝑧 𝑏 = 𝑧 > 0 → 𝑡 + 𝑐𝑣𝑔

𝑦

˄ [ 𝑧 ≤ 0 → 𝑡] 𝐽𝑦,𝑧 = 𝑐𝑣𝑔

𝑦 = 𝑏

1 2 3 4 2 4 6 8 3 6 9 12 4 8 12 16 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Input Image Output Image

09.09.2019 Lester Kalms 7

slide-8
SLIDE 8

HW Implementation

Integral Image Function

SW: 𝐽𝑦,𝑧 = 𝑗𝑦,𝑧 + 𝐽x−1,y + 𝐽x,y−1 − 𝐽x−1,y−1 HW: 𝑡 = 𝑡 + 𝑗𝑦,𝑧 𝑏 = 𝑧 > 0 → 𝑡 + 𝑐𝑣𝑔

𝑦

˄ [ 𝑧 ≤ 0 → 𝑡] 𝐽𝑦,𝑧 = 𝑐𝑣𝑔

𝑦 = 𝑏

1 2 3 4 2 4 6 8 3 6 9 12 4 8 12 16 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Input Image Output Image

09.09.2019 Lester Kalms 8

slide-9
SLIDE 9

HW Implementation

Integral Image Function

  • 4 Pixels each clock cycle
  • High frequency

09.09.2019 Lester Kalms 9 1 2 3 4 5 6 7 8 2 4 6 8 10 12 14 16 3 6 9 12 15 18 21 24 4 8 12 16 20 24 28 32 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Input Image Output Image

slide-10
SLIDE 10

09.09.2019 Lester Kalms 10

HW Implementation

FREAK Descriptor

Implementation

  • VHDL
  • Fixed-Point
  • Pipelined
slide-11
SLIDE 11

09.09.2019 Lester Kalms 11

HW Implementation

FREAK Descriptor (Pattern) FREAK sampling pattern similar to the retinal ganglion cells [1]

slide-12
SLIDE 12

09.09.2019 Lester Kalms 12

HW Implementation

FREAK Descriptor

slide-13
SLIDE 13

09.09.2019 Lester Kalms 13

HW Implementation

FREAK Descriptor (intensity) FREAK sampling pattern similar to the retinal ganglion cells [1]

Intensity value calculation 𝑗 = 𝐽𝑧𝑐,𝑦𝑠 − 𝐽𝑧𝑐,𝑦𝑚 + 𝐽𝑧𝑢,𝑦𝑚 − 𝐽𝑧𝑢,𝑦𝑠 𝑦𝑠 − 𝑦𝑚 ⋅ 𝑧𝑐 − 𝑧𝑢

slide-14
SLIDE 14

09.09.2019 Lester Kalms 14

HW Implementation

FREAK Descriptor

slide-15
SLIDE 15

09.09.2019 Lester Kalms 15

HW Implementation

FREAK Descriptor

slide-16
SLIDE 16

09.09.2019 Lester Kalms 16

HW Implementation

FREAK Descriptor

slide-17
SLIDE 17

Evaluation

Software (Repeatability) Position Quality Time detect + describe match ratio avg. inlier ratio inlier ratio match ratio avg. inlier ratio inlier ratio time (ms) akaze + akaze

21.6 63.3 85.4 42.5

82.2

85.0 67.0

akaze + freak

23.1

57.2 84.7

46.8

83.2 86.2 70.3

akaze + brisk

19.4 59.4

87.8

46.6

85.3

88.0 77.0

akaze + orb

11.5 37.8 86.2 37.8 84.4

88.5

68.4

brisk + brisk

17.3

66.1

85.1 26.4 70.6 87.7 29.9

brisk + freak

21.7

58.7 81.4 31.6 64.0 82.5 29.5

brisk + orb

11.2 49.8 82.0 26.5 67.7 86.3 29.9

  • rb + orb

15.0 61.2 82.2 46.6 75.9 84.8

16.0

  • rb + freak

19.3 65.8 84.9

48.2

79.3 87.8

17.7

  • rb + brisk

16.9

66.8 85.8

44.9 81.3

89.5

25.3

proposed

23.1 70.2 87.9 60.9 87.2 92.8 24.9

09.09.2019 Lester Kalms 17

Setup

  • Core-i7 7700
  • Oxford Dataset:

800x640 - 1000x700

  • Quality: luminance,

blur & compression

  • Position: Scale,

rotation & viewpoint

slide-18
SLIDE 18

Evaluation

Hardware (Timing)

Setup

  • ZedBoard (Zynq 7020) & Vivado 2018.2
  • Vhdl, Fixed-point data types, 2048 key-points per frame

Frequency / FPS (detector)

  • 185 MHz (synthesis)
  • Simulation: for 150 MHz -> 850 fps (5.85 Gbit/s bandwidth)
  • Real system: for 167 MHz -> 73.4 fps (including DMAs)

Frequency (Integral)

  • 204 MHz (synthesis)
  • 3ns for parallel prefix sum

09.09.2019 Lester Kalms 18

slide-19
SLIDE 19

Evaluation

Hardware (Resources) Module FF LUT LUTRAM BRAM DSP

Parallel sum 596 448 Controller 209 51

Integral (%) (0.75) 800 (0.97) 515 (0.01) 2 (1.79) 2.5 (0.00) 0

Boundary 334 131 33 1 Pattern unrotated 311 12 2 1 Pattern rotated 1673 1570 104 4.5 3 Intensity 1183 1719 24 4.5 5 Orientation 1292 1292 1 6 Descriptor 2414 1837 Send keypoint 130 59 1.5 Send descriptor 187 67 1.5

Descriptor (%) (7.63) 8115 (17.17) 9133 (1.06) 184 (13.93) 19.5 (9.09) 20

09.09.2019 Lester Kalms 19

slide-20
SLIDE 20

Evaluation

HW/SW (Comparison)

Algorithm Inliers Ratio

  • Avg. Inliers Ratio

(arithmetic mean) Proposed SW 91.4 78.7 Proposed HW 90.9 77.0 ORB + FREAK 88.4 72.5 ORB + BRISK 88.0 74.0

09.09.2019 Lester Kalms 20

slide-21
SLIDE 21

Summary

  • Proposed implementation of AKAZE detector + FREAK descriptor
  • Improved algorithm, parameters and added Retain Best function
  • HW implementation of FREAK detector + Integral image
  • Achieved good computation time and repeatability in comparison to others
  • Achieved a low resource consumption

Outlook

  • Combine AKAZE detector with FREAK descriptor in HW
  • Integrate it into computer vision algorithm

Conclusion

09.09.2019 Lester Kalms 21

slide-22
SLIDE 22

[1] A. Alahi, R. Ortiz, and P. Vandergheynst. Freak: Fast retina keypoint. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 510–517, Jun 2012.

References

This work has been funded by the German Federal Ministry of Education and Research BMBF as part of the PARIS project under grant agreement number 16ES0657.

Acknowledgment

slide-23
SLIDE 23

Thank you for your attention! Any questions?

slide-24
SLIDE 24
slide-25
SLIDE 25