Efficient Pattern Recognition Algorithm Including a Fast Retina - PowerPoint PPT Presentation

Efficient Pattern Recognition Algorithm Including a Fast Retina Keypoint FPGA Implementation Lester Kalms Maximilian Hajduk Diana Göhringer Technische Universität Technische Universität Technische Universität Dresden, Germany Dresden, Germany Dresden, Germany lester.kalms@tu-dresden.de maximilian.hajduk@tu-dresden.de diana.goehringer@tu-dresden.de

Content 1. Motivation & Overview 2. Pattern Recognition Algorithm 3. HW Implementation of FREAK 4. Evaluation 5. Conclusion & Outlook 09.09.2019 Lester Kalms 2

Motivation What is pattern recognition? • Feature detector: finds points of interest in images (e.g. corners, edges or blobs) • Feature descriptor: describes features to be comparable with others Why pattern recognition? • Still algorithms make use of pattern recognition (e.g. object tracking or SLAM) • Most neural networks need a high amount of input data 09.09.2019 Lester Kalms 3

Overview First full FPGA implementation of the FREAK descriptor • Related work does not contain orientation computation • Contains optimized pattern generator (partially pre-computed) • Frequency optimized integral image with parallel prefix sum Improved pattern recognition algorithm • Compared to different combinations of AKAZE, ORB, BRISK and FREAK • Improved and combined AKAZE detector & FREAK descriptor • Added a Retain Best function 09.09.2019 Lester Kalms 4

SW Implementation AKAZE feature detection & FREAK feature description 09.09.2019 Lester Kalms 5

HW Implementation Integral Image Function 1 1 1 1 1 2 3 4 1 1 1 1 2 4 6 8 1 1 1 1 3 6 9 12 1 1 1 1 4 8 12 16 Input Image Output Image SW: 𝐽 𝑦,𝑧 = 𝑗 𝑦,𝑧 + 𝐽 x−1,y + 𝐽 x,y−1 − 𝐽 x−1,y−1 HW: 𝑡 = 𝑡 + 𝑗 𝑦,𝑧 𝑏 = 𝑧 > 0 → 𝑡 + 𝑐𝑣𝑔 ˄ [ 𝑧 ≤ 0 → 𝑡] 𝑦 𝐽 𝑦,𝑧 = 𝑐𝑣𝑔 𝑦 = 𝑏 09.09.2019 Lester Kalms 6

HW Implementation Integral Image Function 1 1 1 1 1 2 3 4 1 1 1 1 2 4 6 8 1 1 1 1 3 6 9 12 1 1 1 1 4 8 12 16 Input Image Output Image SW: 𝐽 𝑦,𝑧 = 𝑗 𝑦,𝑧 + 𝐽 x−1,y + 𝐽 x,y−1 − 𝐽 x−1,y−1 HW: 𝑡 = 𝑡 + 𝑗 𝑦,𝑧 𝑏 = 𝑧 > 0 → 𝑡 + 𝑐𝑣𝑔 ˄ [ 𝑧 ≤ 0 → 𝑡] 𝑦 𝐽 𝑦,𝑧 = 𝑐𝑣𝑔 𝑦 = 𝑏 09.09.2019 Lester Kalms 7

HW Implementation Integral Image Function 1 1 1 1 1 2 3 4 1 1 1 1 2 4 6 8 1 1 1 1 3 6 9 12 1 1 1 1 4 8 12 16 Input Image Output Image SW: 𝐽 𝑦,𝑧 = 𝑗 𝑦,𝑧 + 𝐽 x−1,y + 𝐽 x,y−1 − 𝐽 x−1,y−1 HW: 𝑡 = 𝑡 + 𝑗 𝑦,𝑧 𝑏 = 𝑧 > 0 → 𝑡 + 𝑐𝑣𝑔 ˄ [ 𝑧 ≤ 0 → 𝑡] 𝑦 𝐽 𝑦,𝑧 = 𝑐𝑣𝑔 𝑦 = 𝑏 09.09.2019 Lester Kalms 8

HW Implementation Integral Image Function 1 1 1 1 1 1 1 1 1 2 3 4 5 6 7 8 • 1 1 1 1 1 1 1 1 2 4 6 8 10 12 14 16 4 Pixels each clock cycle 1 1 1 1 1 1 1 1 3 6 9 12 15 18 21 24 • High frequency 1 1 1 1 1 1 1 1 4 8 12 16 20 24 28 32 Input Image Output Image 09.09.2019 Lester Kalms 9

HW Implementation FREAK Descriptor Implementation • VHDL • Fixed-Point • Pipelined 09.09.2019 Lester Kalms 10

HW Implementation FREAK Descriptor (Pattern) FREAK sampling pattern similar to the retinal ganglion cells [1] 09.09.2019 Lester Kalms 11

HW Implementation FREAK Descriptor 09.09.2019 Lester Kalms 12

HW Implementation FREAK Descriptor (intensity) Intensity value calculation 𝐽 𝑧 𝑐 ,𝑦 𝑠 − 𝐽 𝑧 𝑐 ,𝑦 𝑚 + 𝐽 𝑧 𝑢 ,𝑦 𝑚 − 𝐽 𝑧 𝑢 ,𝑦 𝑠 𝑗 = 𝑦 𝑠 − 𝑦 𝑚 ⋅ 𝑧 𝑐 − 𝑧 𝑢 FREAK sampling pattern similar to the retinal ganglion cells [1] 09.09.2019 Lester Kalms 13

HW Implementation FREAK Descriptor 09.09.2019 Lester Kalms 14

HW Implementation FREAK Descriptor 09.09.2019 Lester Kalms 15

HW Implementation FREAK Descriptor 09.09.2019 Lester Kalms 16

Evaluation Software (Repeatability) Position Quality Time Setup avg. avg. detect + match inlier match inlier time inlier inlier describe ratio ratio ratio ratio (ms) • Core-i7 7700 ratio ratio akaze + akaze 82.2 21.6 63.3 85.4 42.5 85.0 67.0 • Oxford Dataset: 23.1 46.8 akaze + freak 57.2 84.7 83.2 86.2 70.3 800x640 - 1000x700 87.8 85.3 akaze + brisk 19.4 59.4 46.6 88.0 77.0 • Quality: luminance, akaze + orb 88.5 11.5 37.8 86.2 37.8 84.4 68.4 blur & compression 66.1 brisk + brisk 17.3 85.1 26.4 70.6 87.7 29.9 brisk + freak 21.7 • 58.7 81.4 31.6 64.0 82.5 29.5 Position: Scale, brisk + orb 11.2 49.8 82.0 26.5 67.7 86.3 29.9 rotation & viewpoint 16.0 orb + orb 15.0 61.2 82.2 46.6 75.9 84.8 48.2 17.7 orb + freak 19.3 65.8 84.9 79.3 87.8 66.8 85.8 89.5 orb + brisk 16.9 44.9 81.3 25.3 23.1 70.2 87.9 60.9 87.2 92.8 24.9 proposed 09.09.2019 Lester Kalms 17

Evaluation Hardware (Timing) Setup • ZedBoard (Zynq 7020) & Vivado 2018.2 • Vhdl, Fixed-point data types, 2048 key-points per frame Frequency / FPS (detector) • 185 MHz (synthesis) • Simulation: for 150 MHz -> 850 fps (5.85 Gbit/s bandwidth) • Real system: for 167 MHz -> 73.4 fps (including DMAs) Frequency (Integral) • 204 MHz (synthesis) • 3ns for parallel prefix sum 09.09.2019 Lester Kalms 18

Evaluation Hardware (Resources) Module FF LUT LUTRAM BRAM DSP Parallel sum 596 448 0 0 0 Controller 209 51 0 0 0 Integral (%) (0.75) 800 (0.97) 515 (0.01) 2 (1.79) 2.5 (0.00) 0 Boundary 334 131 33 1 0 Pattern unrotated 311 12 0 2 1 Pattern rotated 1673 1570 104 4.5 3 Intensity 1183 1719 24 4.5 5 Orientation 1292 1292 1 0 6 Descriptor 2414 1837 0 0 0 Send keypoint 130 59 0 1.5 0 Send descriptor 187 67 0 1.5 0 Descriptor (%) (7.63) 8115 (17.17) 9133 (1.06) 184 (13.93) 19.5 (9.09) 20 09.09.2019 Lester Kalms 19

Evaluation HW/SW (Comparison) Avg. Inliers Ratio Algorithm Inliers Ratio (arithmetic mean) Proposed SW 91.4 78.7 Proposed HW 90.9 77.0 ORB + FREAK 88.4 72.5 ORB + BRISK 88.0 74.0 09.09.2019 Lester Kalms 20

Conclusion Summary • Proposed implementation of AKAZE detector + FREAK descriptor • Improved algorithm, parameters and added Retain Best function • HW implementation of FREAK detector + Integral image • Achieved good computation time and repeatability in comparison to others • Achieved a low resource consumption Outlook • Combine AKAZE detector with FREAK descriptor in HW • Integrate it into computer vision algorithm 09.09.2019 Lester Kalms 21

Acknowledgment This work has been funded by the German Federal Ministry of Education and Research BMBF as part of the PARIS project under grant agreement number 16ES0657. References [1] A. Alahi, R. Ortiz, and P. Vandergheynst. Freak: Fast retina keypoint . In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 510 – 517, Jun 2012.

Thank you for your attention! Any questions?