Hardware and Software architecture of a bio-inspired vision system - PowerPoint PPT Presentation

Hardware and Software architecture of a bio-inspired vision system for mobile robots   Thomas Lefebvre       ETIS / ENSEA - Université de Cergy-Pontoise - CNRS UMR 8051   95014 Cergy-Pontoise Cedex, France         5 avril 2012              

Plan 1 Context of the system 5 proposed architecture 6 a 3 example 2 Vision system 3 Software prototype 7 synthesis results 4 Hardware data flow chain 8 current prototypes HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 2 / 35

Context of the system Summary 1 Context of the system 5 proposed architecture 6 a 3 example 2 Vision system 3 Software prototype 7 synthesis results 4 Hardware data flow chain 8 current prototypes HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 3 / 35

Context of the system Bio-inspired robotics HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 4 / 35

Context of the system Bio-inspired robotics Autonomous navigation / facial expressions recognition / ... HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 4 / 35

Context of the system Bio-inspired robotics Autonomous navigation / facial expressions recognition / ... Bio-inspired neural-networks HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 4 / 35

Context of the system Bio-inspired robotics Autonomous navigation / facial expressions recognition / ... Bio-inspired neural-networks Camera ⇒ Vision system ⇒ neural network HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 4 / 35

Context of the system Bio-inspired robotics Autonomous navigation / facial expressions recognition / ... Bio-inspired neural-networks Camera ⇒ Vision system ⇒ neural network Current vision system : deported on workstations HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 4 / 35

Context of the system Bio-inspired robotics Autonomous navigation / facial expressions recognition / ... Bio-inspired neural-networks Camera ⇒ Vision system ⇒ neural network Current vision system : deported on workstations Goal : multi-resolution approach - embedded HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 4 / 35

Context of the system Goals and constraints Goals Constraints HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 5 / 35

Context of the system Goals and constraints Goals Constraints integration ⇒ Reduced power and volume available HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 5 / 35

Context of the system Goals and constraints Goals Constraints integration ⇒ Reduced power and volume available speed ⇒ 25 frames per second HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 5 / 35

Context of the system Goals and constraints Goals Constraints integration ⇒ Reduced power and volume available speed ⇒ 25 frames per second precision ⇒ 640 × 480 pixels camera or more HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 5 / 35

Context of the system Goals and constraints Goals Constraints integration ⇒ Reduced power and volume available speed ⇒ 25 frames per second precision ⇒ 640 × 480 pixels camera or more flexibility ⇒ Some parameters may vary at runtime HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 5 / 35

Context of the system Goals and constraints Goals Constraints integration ⇒ Reduced power and volume available speed ⇒ 25 frames per second precision ⇒ 640 × 480 pixels camera or more flexibility ⇒ Some parameters may vary at runtime Proposed solution : FPGA - embedded processor and hardware IPs HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 5 / 35

Context of the system Design flow SOFTWARE Original APPLICATION version HARDWARE/SOFTWARE PARTITIONING Software IP tuning optimization EMBEDDED IPs Implementation SOFTWARE VHDL FUNCTIONAL Verification EXECUTION SIMULATION Measures Synthesis FULL SYNTHESIS Validation HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 6 / 35

Vision system Summary 1 Context of the system 5 proposed architecture 6 a 3 example 2 Vision system 3 Software prototype 7 synthesis results 4 Hardware data flow chain 8 current prototypes HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 7 / 35

Vision system Gradient, Gaussian filtering, subsampling HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 8 / 35

Vision system Difference of Gaussians (DoG) HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 9 / 35

Vision system Keypoint search HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 10 / 35

Vision system Log-polar transform HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 11 / 35

Software prototype Summary 1 Context of the system 5 proposed architecture 6 a 3 example 2 Vision system 3 Software prototype 7 synthesis results 4 Hardware data flow chain 8 current prototypes HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 12 / 35

Software prototype Temporal behavior embedded - 1GHz Cortex A8, 256MB RAM 192 × 144 pixels frames HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 13 / 35

Software prototype Temporal behavior embedded - 1GHz Cortex A8, 256MB RAM 192 × 144 pixels frames Total execution Percentage Function time per frame of the total Gradient 11.1 ms 5.4% Gaussian filtering 145.7 ms 70.7% Subsampling 1.3 ms 6.3% DoG 9.1 ms 4.4% Keypoint search 27.3 ms 13.3% Neighborhoods 11.5 ms 5.6% Total 205.9 ms 100% HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 13 / 35

Software prototype Temporal behavior embedded - 1GHz Cortex A8, 256MB RAM 192 × 144 pixels frames Total execution Percentage Function time per frame of the total Gradient 11.1 ms 5.4% Gaussian filtering 145.7 ms 70.7% Subsampling 1.3 ms 6.3% DoG 9.1 ms 4.4% Keypoint search 27.3 ms 13.3% Neighborhoods 11.5 ms 5.6% Total 205.9 ms 100% 205.9 ms ⇒ 4.85 fps HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 13 / 35

Software prototype Temporal behavior embedded - 1GHz Cortex A8, 256MB RAM 192 × 144 pixels frames 205.9 ms ⇒ 4.85 fps laptop - 1.66GHz Intel Core Duo, 2GB RAM : 39.4 ms ⇒ 25.38 fps What about 640 × 480 pixels frames ? HD video ? HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 13 / 35

Hardware data flow chain Summary 1 Context of the system 5 proposed architecture 6 a 3 example 2 Vision system 3 Software prototype 7 synthesis results 4 Hardware data flow chain 8 current prototypes HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 14 / 35

Hardware data flow chain Pixel flow CONTINUOUS PIXEL FLOW CAMERA GAUSSIAN GRADIENT FILTER (1) KEYPOINT GAUSSIAN SEARCH FILTER (1) NEIGHBORHOOD SORT GAUSSIAN EXTRACTION FILTER ( 2 ) ... SUBSAMPLE LOG/POLAR TRANSFORM GAUSSIAN FILTER (1) NORMALIZATION GAUSSIAN TO THE ROBOT FILTER ( 2 ) SUBSAMPLE GAUSSIAN FILTER (1) GAUSSIAN FILTER ( 2 ) HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 15 / 35

Hardware data flow chain Pixel flow Modular, generic shell ⇒ greater flexibility clk Enable Enable_out x_in x_out y_in y_out pixel_in pixel_out HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 15 / 35

proposed architecture Summary 1 Context of the system 5 proposed architecture 6 a 3 example 2 Vision system 3 Software prototype 7 synthesis results 4 Hardware data flow chain 8 current prototypes HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 16 / 35

proposed architecture global system view EXTERNAL MEMORY FPGA CAMERA MEMORY GAUSSIAN CONTROLLER GRADIENT FILTER (1) KEYPOINT GAUSSIAN SORT SEARCH FILTER (1) EMBEDDED GAUSSIAN KEYPOINT SORT FILTER ( 2 ) SEARCH ETHERNET MODULE PROCESSOR SUBSAMPLE KEYPOINT GAUSSIAN SORT SEARCH FILTER (1) KEYPOINT GAUSSIAN SORT SEARCH FILTER ( 2 ) SUBSAMPLE KEYPOINT GAUSSIAN SORT SEARCH FILTER (1) KEYPOINT GAUSSIAN SORT SEARCH FILTER ( 2 ) HW/SW architecture of a bio-inspired robotic vision system | T. Lefebvre | 5 avril 2012 17 / 35

Hardware and Software architecture of a bio-inspired vision system - PowerPoint PPT Presentation

Hardware and Software architecture of a bio-inspired vision system for mobile robots Thomas Lefebvre ETIS / ENSEA - Universit de Cergy-Pontoise - CNRS UMR 8051 95014 Cergy-Pontoise Cedex,

Outline Background Venezia Hardware Architecture Venezia Software Architecture

A New Golden Age for 1. Software advances can inspire architecture Computer Architecture:

Biologically I nspired Hardware System What is Bio-Inspired System? Why do we need

software and hardware for the Internet of Things. Choose hardware Design hardware Design

Open Source Software/Hardware Decoupling Open Source Software (OpenStack, CORD)

SE2: Introduction to Software Architecture Mei Nagappan What is Architecture? Encyclopedia

Part I Security Challenges in Automotive Hardware/Software Architecture Design Martin

The Dover Architecture Hardware Enforcement of Software-Defined Security Policies Team: Greg

Some background... Started VJing in 2005 Kludgy, ad hoc software inspired by MilkDrop,

Spark architecture Spark architecture Hardware organization Hardware organization In local

Hardware Decompression for Compressed Sensing Applications Keith Dronson Frank Zovko Samuel

Whats new? Calibration Hardware IAEA Acquisition software Hardware and Software Upgrade for

Software Architecture Jay Urbain, Ph.D. urbain@msoe.edu 1 Software Architecture (SA) 1. SA: How

Basic Basic Structure Structure of of C Computer omputer Hardware and Software

SATIRE: A Software Introduction to SATIRE Architecture for Smart Software Architecture

Lecture 3 Hardware and Software 3. Hardware and Softw are 4. High Level Languages 5.

Spatial Filtering CS/BIOEN 4640: Image Processing Basics January 31, 2012 Limitations of Point

Images and Filters CSE 576 Ali Farhadi Many slides from

Gaussian Filter The Gaussian filter 1 2 1 A Gaussian kernel gives less 1 2 4 2 weight to

Coded Computation Against Straggling Channel Decoders In The Cloud For Gaussian Channels Jinwen

Fast Gaussian Filtering Algorithm Using Splines Kentaro Imajo (M2, Kyoto University) November

A Fast Approximation Algorithm for the Gaussian Filter Kentaro Imajo (Yamamoto Laboratory)

Using the Special Function Registers of the Digital I/O interface of STM32 Corrado Santoro

Hardware FUNDAMENTALS Getting started with Do It Yourself circuits and Raspberry Pi Hardware?