Hardware and Software architecture of a bio-inspired vision system - - PowerPoint PPT Presentation

▶
hardware and software architecture of a bio inspired
SMART_READER_LITE
LIVE PREVIEW

Hardware and Software architecture of a bio-inspired vision system - - PowerPoint PPT Presentation

Dec 23, 2023 378 likes •1.22k views

Hardware and Software architecture of a bio-inspired vision system for mobile robots Thomas Lefebvre ETIS / ENSEA - Universit de Cergy-Pontoise - CNRS UMR 8051 95014 Cergy-Pontoise Cedex,

slide-1
SLIDE 1

Hardware and Software architecture of a bio-inspired vision system for mobile robots

               

Thomas Lefebvre

ETIS / ENSEA - Université de Cergy-Pontoise - CNRS UMR 8051 95014 Cergy-Pontoise Cedex, France

5 avril 2012

               

slide-2
SLIDE 2

Plan

1 Context of the system 2 Vision system 3 Software prototype 4 Hardware data flow chain 5 proposed architecture 6 a3 example 7 synthesis results 8 current prototypes

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 2 / 35

slide-3
SLIDE 3

Context of the system

Summary

1 Context of the system 2 Vision system 3 Software prototype 4 Hardware data flow chain 5 proposed architecture 6 a3 example 7 synthesis results 8 current prototypes

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 3 / 35

slide-4
SLIDE 4

Context of the system

Bio-inspired robotics

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 4 / 35

slide-5
SLIDE 5

Context of the system

Bio-inspired robotics

Autonomous navigation / facial expressions recognition / ...

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 4 / 35

slide-6
SLIDE 6

Context of the system

Bio-inspired robotics

Autonomous navigation / facial expressions recognition / ... Bio-inspired neural-networks

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 4 / 35

slide-7
SLIDE 7

Context of the system

Bio-inspired robotics

Autonomous navigation / facial expressions recognition / ... Bio-inspired neural-networks Camera ⇒ Vision system ⇒ neural network

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 4 / 35

slide-8
SLIDE 8

Context of the system

Bio-inspired robotics

Autonomous navigation / facial expressions recognition / ... Bio-inspired neural-networks Camera ⇒ Vision system ⇒ neural network Current vision system : deported on workstations

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 4 / 35

slide-9
SLIDE 9

Context of the system

Bio-inspired robotics

Autonomous navigation / facial expressions recognition / ... Bio-inspired neural-networks Camera ⇒ Vision system ⇒ neural network Current vision system : deported on workstations Goal : multi-resolution approach - embedded

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 4 / 35

slide-10
SLIDE 10

Context of the system

Goals and constraints

Goals Constraints

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 5 / 35

slide-11
SLIDE 11

Context of the system

Goals and constraints

Goals integration Constraints ⇒ Reduced power and volume available

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 5 / 35

slide-12
SLIDE 12

Context of the system

Goals and constraints

Goals integration speed Constraints ⇒ Reduced power and volume available ⇒ 25 frames per second

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 5 / 35

slide-13
SLIDE 13

Context of the system

Goals and constraints

Goals integration speed precision Constraints ⇒ Reduced power and volume available ⇒ 25 frames per second ⇒ 640 × 480 pixels camera or more

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 5 / 35

slide-14
SLIDE 14

Context of the system

Goals and constraints

Goals integration speed precision flexibility Constraints ⇒ Reduced power and volume available ⇒ 25 frames per second ⇒ 640 × 480 pixels camera or more ⇒ Some parameters may vary at runtime

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 5 / 35

slide-15
SLIDE 15

Context of the system

Goals and constraints

Goals integration speed precision flexibility Constraints ⇒ Reduced power and volume available ⇒ 25 frames per second ⇒ 640 × 480 pixels camera or more ⇒ Some parameters may vary at runtime Proposed solution : FPGA - embedded processor and hardware IPs

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 5 / 35

slide-16
SLIDE 16

Context of the system

Design flow

Validation VHDL IPs Original version APPLICATION SOFTWARE HARDWARE/SOFTWARE PARTITIONING FUNCTIONAL SIMULATION SOFTWARE EMBEDDED EXECUTION FULL SYNTHESIS Measures Synthesis

  • ptimization

Software Implementation Verification IP tuning HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 6 / 35

slide-17
SLIDE 17

Vision system

Summary

1 Context of the system 2 Vision system 3 Software prototype 4 Hardware data flow chain 5 proposed architecture 6 a3 example 7 synthesis results 8 current prototypes

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 7 / 35

slide-18
SLIDE 18

Vision system

Gradient, Gaussian filtering, subsampling

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 8 / 35

slide-19
SLIDE 19

Vision system

Difference of Gaussians (DoG)

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 9 / 35

slide-20
SLIDE 20

Vision system

Keypoint search

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 10 / 35

slide-21
SLIDE 21

Vision system

Log-polar transform

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 11 / 35

slide-22
SLIDE 22

Vision system

Log-polar transform

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 11 / 35

slide-23
SLIDE 23

Vision system

Log-polar transform

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 11 / 35

slide-24
SLIDE 24

Software prototype

Summary

1 Context of the system 2 Vision system 3 Software prototype 4 Hardware data flow chain 5 proposed architecture 6 a3 example 7 synthesis results 8 current prototypes

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 12 / 35

slide-25
SLIDE 25

Software prototype

Temporal behavior

embedded - 1GHz Cortex A8, 256MB RAM 192 × 144 pixels frames

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 13 / 35

slide-26
SLIDE 26

Software prototype

Temporal behavior

embedded - 1GHz Cortex A8, 256MB RAM 192 × 144 pixels frames Total execution Percentage Function time per frame

  • f the total

Gradient 11.1 ms 5.4% Gaussian filtering 145.7 ms 70.7% Subsampling 1.3 ms 6.3% DoG 9.1 ms 4.4% Keypoint search 27.3 ms 13.3% Neighborhoods 11.5 ms 5.6% Total 205.9 ms 100%

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 13 / 35

slide-27
SLIDE 27

Software prototype

Temporal behavior

embedded - 1GHz Cortex A8, 256MB RAM 192 × 144 pixels frames Total execution Percentage Function time per frame

  • f the total

Gradient 11.1 ms 5.4% Gaussian filtering 145.7 ms 70.7% Subsampling 1.3 ms 6.3% DoG 9.1 ms 4.4% Keypoint search 27.3 ms 13.3% Neighborhoods 11.5 ms 5.6% Total 205.9 ms 100% 205.9 ms ⇒ 4.85 fps

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 13 / 35

slide-28
SLIDE 28

Software prototype

Temporal behavior

embedded - 1GHz Cortex A8, 256MB RAM 192 × 144 pixels frames 205.9 ms ⇒ 4.85 fps laptop - 1.66GHz Intel Core Duo, 2GB RAM : 39.4 ms ⇒ 25.38 fps What about 640 × 480 pixels frames ? HD video ?

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 13 / 35

slide-29
SLIDE 29

Hardware data flow chain

Summary

1 Context of the system 2 Vision system 3 Software prototype 4 Hardware data flow chain 5 proposed architecture 6 a3 example 7 synthesis results 8 current prototypes

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 14 / 35

slide-30
SLIDE 30

Hardware data flow chain

Pixel flow

SUBSAMPLE SUBSAMPLE ... CAMERA GRADIENT NEIGHBORHOOD EXTRACTION NORMALIZATION SORT KEYPOINT TO THE ROBOT LOG/POLAR TRANSFORM SEARCH GAUSSIAN FILTER (1) GAUSSIAN FILTER ( 2 ) GAUSSIAN FILTER ( 2 ) GAUSSIAN FILTER (1) GAUSSIAN FILTER ( 2 ) FILTER (1) GAUSSIAN GAUSSIAN FILTER (1) CONTINUOUS PIXEL FLOW

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 15 / 35

slide-31
SLIDE 31

Hardware data flow chain

Pixel flow

Modular, generic shell ⇒ greater flexibility Enable Enable_out y_in x_in x_out y_out pixel_in pixel_out clk

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 15 / 35

slide-32
SLIDE 32

proposed architecture

Summary

1 Context of the system 2 Vision system 3 Software prototype 4 Hardware data flow chain 5 proposed architecture 6 a3 example 7 synthesis results 8 current prototypes

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 16 / 35

slide-33
SLIDE 33

proposed architecture

global system view

SUBSAMPLE SUBSAMPLE EMBEDDED PROCESSOR SORT KEYPOINT SEARCH SORT KEYPOINT SEARCH SORT KEYPOINT SEARCH SORT KEYPOINT SEARCH SORT KEYPOINT SEARCH SORT KEYPOINT SEARCH CAMERA GRADIENT GAUSSIAN FILTER (1) GAUSSIAN FILTER ( 2 ) GAUSSIAN FILTER ( 2 ) GAUSSIAN FILTER (1) GAUSSIAN FILTER ( 2 ) FILTER (1) GAUSSIAN GAUSSIAN FILTER (1) MODULE ETHERNET EXTERNAL MEMORY MEMORY CONTROLLER FPGA

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 17 / 35

slide-34
SLIDE 34

proposed architecture

hardware acceleration

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 18 / 35

slide-35
SLIDE 35

proposed architecture

hardware acceleration

The IPs produce one grayscale pixel per clock cycle

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 18 / 35

slide-36
SLIDE 36

proposed architecture

hardware acceleration

The IPs produce one grayscale pixel per clock cycle 1920 × 1080 pixels : 2.07 M pixels

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 18 / 35

slide-37
SLIDE 37

proposed architecture

hardware acceleration

The IPs produce one grayscale pixel per clock cycle 1920 × 1080 pixels : 2.07 M pixels 25 full-HD fps ⇒ 51.84 M pixels per second

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 18 / 35

slide-38
SLIDE 38

proposed architecture

hardware acceleration

The IPs produce one grayscale pixel per clock cycle 1920 × 1080 pixels : 2.07 M pixels 25 full-HD fps ⇒ 51.84 M pixels per second 100 MHz OK for the proposed IPs

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 18 / 35

slide-39
SLIDE 39

proposed architecture

hardware acceleration

The IPs produce one grayscale pixel per clock cycle 1920 × 1080 pixels : 2.07 M pixels 25 full-HD fps ⇒ 51.84 M pixels per second 100 MHz OK for the proposed IPs The rest depends on :

the embedded processor (Features + Ethernet) the camera interface (RAW ? ⇒ grayscale)

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 18 / 35

slide-40
SLIDE 40

a3 example

Summary

1 Context of the system 2 Vision system 3 Software prototype 4 Hardware data flow chain 5 proposed architecture 6 a3 example 7 synthesis results 8 current prototypes

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 19 / 35

slide-41
SLIDE 41

a3 example

Gaussian filter - 2d convolution

2 solutions for Gaussian filtering :

1 2d window convolution

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 20 / 35

slide-42
SLIDE 42

a3 example

Gaussian filter - 2d convolution

2 solutions for Gaussian filtering :

1 2d window convolution

large number of multiplications (w2

window)

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 20 / 35

slide-43
SLIDE 43

a3 example

Gaussian filter - 2d convolution

2 solutions for Gaussian filtering :

1 2d window convolution

large number of multiplications (w2

window) 2 vertical + horizontal 1d windows convolutions

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 20 / 35

slide-44
SLIDE 44

a3 example

Gaussian filter - 2d convolution

2 solutions for Gaussian filtering :

1 2d window convolution

large number of multiplications (w2

window) 2 vertical + horizontal 1d windows convolutions

fewer multiplications (2 × wwindow)

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 20 / 35

slide-45
SLIDE 45

a3 example

Traditional architecture :

R R R R R R R R R R R R R R R R R R R R R R R R R R O(x−6,y−1) C(0,0) I(x,y) I(x,y−1) I(x,y−2) C(1,0) C(−1,0) C(−1,1) C(0,1) C(1,1) C(1,−1) C(0,−1) C(−1,−1) SHIFT REGISTER SHIFT REGISTER

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 21 / 35

slide-46
SLIDE 46

a3 example

MACC convolution operators : Very low latency for the sum of products Regular structure, generic code easier to write

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 22 / 35

slide-47
SLIDE 47

a3 example

MACC-based architecture :

R R R R R R R R R R R R O(x−3,y−1) C(−1,1) C(1,1) C(0,1) R R R R R R C(−1,−1) C(0,−1) C(1,−1) C(−1,0) C(0,0) C(1,0) I(x,y) REGISTER SHIFT REGISTER SHIFT

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 23 / 35

slide-48
SLIDE 48

a3 example

1D vertical+horizontal :

R R R R R R O(x−7,y−1) R C(0) R C(−1) R C(1) R R V(x−4,y−1) C(−1) C(0) C(1) I(x,y) SHIFT REGISTER SHIFT REGISTER HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 24 / 35

slide-49
SLIDE 49

a3 example

1D vertical+horizontal :

R R R R R R O(x−7,y−1) R C(−1) R C(1) R C(0) R R REGISTER SHIFT REGISTER SHIFT V(x−4,y−1) C(−1) C(0) C(1) I(x,y) HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 24 / 35

slide-50
SLIDE 50

synthesis results

Summary

1 Context of the system 2 Vision system 3 Software prototype 4 Hardware data flow chain 5 proposed architecture 6 a3 example 7 synthesis results 8 current prototypes

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 25 / 35

slide-51
SLIDE 51

synthesis results

Synthesized for Xilinx Virtex 6 family (equivalent results for Kintex 7 family)

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 26 / 35

slide-52
SLIDE 52

synthesis results

Synthesized for Xilinx Virtex 6 family (equivalent results for Kintex 7 family)

IP Registers LUTs 36k BRAM DSP48E1 Gradient 160 <1% 281 <1% 1 <1% Gaussian filter 5639 8% 5494 7% 21 15% 91 100% Subsample 288 <1% 259 <1% 2 2% DoG 192 <1% 624 <1% 6 4% Keypoint search 58352 81% 63550 77% 106 78% Sorting 7032 10% 12048 15% Total 71663 100% 82256 100% 136 100% 91 100%

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 26 / 35

slide-53
SLIDE 53

synthesis results

Synthesized for Xilinx Virtex 6 family (equivalent results for Kintex 7 family)

IP Registers LUTs 36k BRAM DSP48E1 Gradient 160 <1% 281 <1% 1 <1% Gaussian filter 5639 8% 5494 7% 21 15% 91 100% Subsample 288 <1% 259 <1% 2 2% DoG 192 <1% 624 <1% 6 4% Keypoint search 58352 81% 63550 77% 106 78% Sorting 7032 10% 12048 15% Total 71663 100% 82256 100% 136 100% 91 100% Chip Registers LUTs 36k BRAM DSP48E1 301440 150720 416 768 Virtex 6 LX240T FPGA 24% 55% 33% 12% 445200 222600 715 1440 Kintex 7 K355T FPGA 16% 37% 19% 6% 448000 224000 545 900 Zynq Z-7045 (Kintex 7 FPGA) 16% 37% 25% 10%

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 26 / 35

slide-54
SLIDE 54

synthesis results

Influence of parameters

Gaussian Filter IP - 4 parameters : Bus width of pixels and coefficients

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 27 / 35

slide-55
SLIDE 55

synthesis results

Influence of parameters

Gaussian Filter IP - 4 parameters : Bus width of pixels and coefficients Width of coefficient window

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 27 / 35

slide-56
SLIDE 56

synthesis results

Influence of parameters

Gaussian Filter IP - 4 parameters : Bus width of pixels and coefficients Width of coefficient window σ coefficient

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 27 / 35

slide-57
SLIDE 57

synthesis results

Influence of parameters

number of coefficients per window / σ value

200 400 600 800 1000 1200 2 3 4 5 6 7 8 9 10 LUTs Number of coefficients σ²=1 σ²=2

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 27 / 35

slide-58
SLIDE 58

synthesis results

Influence of parameters

number of coefficients per window / σ value

1 2 3 4 5 2 3 4 5 6 7 8 9 10 36k BRAM blocks Number of coefficients σ²=1 σ²=2

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 27 / 35

slide-59
SLIDE 59

synthesis results

Influence of parameters

number of coefficients per window / σ value

2 4 6 8 10 12 14 16 2 3 4 5 6 7 8 9 10 DSP48E1 blocks Number of coefficients σ²=1 σ²=2

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 27 / 35

slide-60
SLIDE 60

synthesis results

Influence of parameters

Bus width : pixels / coefficients

600 800 1000 1200 1400 10 15 20 25 30 LUTs Coefficient bus width (bits) 20 bits 19 bits 18 bits 17 bits 16 bits 14 bits 13 bits 12 bits 11 bits 10 bits 9 bits 8 bits

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 27 / 35

slide-61
SLIDE 61

synthesis results

Influence of parameters

Bus width : pixels / coefficients

2 3 4 5 6 7 10 15 20 36kb BRAM blocks Pixel bus width (bits) 32 bits 10 bits

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 27 / 35

slide-62
SLIDE 62

synthesis results

Influence of parameters

Bus width : pixels / coefficients

10 15 20 25 10 15 20 25 30 DSP48E1 blocks Coefficient bus width (bits) 20 bits 19 bits 18 bits 17 bits 16 bits 14 bits 13 bits 12 bits 11 bits 10 bits 9 bits 8 bits

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 27 / 35

slide-63
SLIDE 63

synthesis results

Influence of parameters

Keypoint search - 2 IP parameters : Bus width of pixels

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 28 / 35

slide-64
SLIDE 64

synthesis results

Influence of parameters

Keypoint search - 2 IP parameters : Bus width of pixels Search radius

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 28 / 35

slide-65
SLIDE 65

synthesis results

Influence of parameters

Keypoint search - 2 IP parameters : Bus width of pixels Search radius 1 pixel within the search radius ⇒ 1 comparator

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 28 / 35

slide-66
SLIDE 66

synthesis results

Influence of parameters

Keypoint search - 2 IP parameters : Bus width of pixels Search radius 1 pixel within the search radius ⇒ 1 comparator R=20 ⇒ 1256 comparators

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 28 / 35

slide-67
SLIDE 67

synthesis results

Influence of parameters

Pixel bus width / search radius

10000 20000 30000 40000 50000 60000 5 10 15 20 25 30 LUTs Search radius (pixels) 20 bits 19 bits 18 bits 17 bits 16 bits 15 bits 14 bits 13 bits 12 bits 11 bits 10 bits 9 bits

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 28 / 35

slide-68
SLIDE 68

synthesis results

Influence of parameters

Pixel bus width / search radius

10 20 30 40 50 60 70 80 10 15 20 36k BRAM blocks Pixel bus width (bits) R=30 R=20 R=10 R=5

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 28 / 35

slide-69
SLIDE 69

current prototypes

Summary

1 Context of the system 2 Vision system 3 Software prototype 4 Hardware data flow chain 5 proposed architecture 6 a3 example 7 synthesis results 8 current prototypes

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 29 / 35

slide-70
SLIDE 70

current prototypes

Embedded platform

Validation of the integration in the robot

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 30 / 35

slide-71
SLIDE 71

current prototypes

Embedded platform

Validation of the integration in the robot Neural network ⇔ Vision system (through Ethernet)

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 30 / 35

slide-72
SLIDE 72

current prototypes

Embedded platform

Validation of the integration in the robot Neural network ⇔ Vision system (through Ethernet) Only one spatial frequency band (small FPGA)

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 30 / 35

slide-73
SLIDE 73

current prototypes

Embedded platform

Validation of the integration in the robot Neural network ⇔ Vision system (through Ethernet) Only one spatial frequency band (small FPGA) 9 fps (320 × 240 pixels), limited by the NIOS II networking

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 30 / 35

slide-74
SLIDE 74

current prototypes

Embedded platform

... CAMERA GRADIENT GAUSSIAN PYRAMID NEIGHBORHOOD EXTRACTION NORMALIZATION SORT KEYPOINT TO THE ROBOT LOG/POLAR TRANSFORM SEARCH GAUSSIAN FILTER (1) GAUSSIAN FILTER ( 2 ) GAUSSIAN FILTER ( 2 ) GAUSSIAN FILTER (1) GAUSSIAN FILTER ( 2 ) FILTER (1) GAUSSIAN GAUSSIAN FILTER (1) SUBSAMPLE SUBSAMPLE

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 30 / 35

slide-75
SLIDE 75

current prototypes

Embedded platform

RAM NIOS II ETHERNET (TO/FROM ROBOT) CAMERA GRADIENT SORT KEYPOINT GAUSSIAN GAUSSIAN FILTER (1) FILTER (1) SEARCH FPGA

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 30 / 35

slide-76
SLIDE 76

current prototypes

Demonstration platform

Vision system ⇒ screen : VGA through SDRAM (Terasic IPs)

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 31 / 35

slide-77
SLIDE 77

current prototypes

Demonstration platform

Vision system ⇒ screen : VGA through SDRAM (Terasic IPs) Only one spatial frequency band

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 31 / 35

slide-78
SLIDE 78

current prototypes

Demonstration platform

Vision system ⇒ screen : VGA through SDRAM (Terasic IPs) Only one spatial frequency band No keypoint search (no image to display after the DoGs)

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 31 / 35

slide-79
SLIDE 79

current prototypes

Demonstration platform

Vision system ⇒ screen : VGA through SDRAM (Terasic IPs) Only one spatial frequency band No keypoint search (no image to display after the DoGs) 18 fps (640 × 480 pixels), limited by the camera module

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 31 / 35

slide-80
SLIDE 80

current prototypes

Demonstration platform

CONTROLLER VGA SDRAM CAMERA GRADIENT GAUSSIAN GAUSSIAN FILTER (1) FILTER (1) CONTROLLER SDRAM SCREEN FPGA HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 31 / 35

slide-81
SLIDE 81

current prototypes

Synthesis results (Embedded PF)

Terasic DE2-115 board - Cyclone IV 4CE115 Images : Grayscale 320 × 240 pixels Parameters : pixels on 16 bits, convolution coefficients on 16 bits, R=20, convolution windows : 7 coefficients Logic Elements : 72k (on 114k - 63%) FPGA Memory : 1956kb (on 3981kb - 49%) 9-bit multipliers : 4 (on 532 - <1%)

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 32 / 35

slide-82
SLIDE 82

current prototypes

Conclusion

The proposed architecture matches all the constraints of the system (time, volume, energy) Hardware IPs allow for the processing of Full-HD at over 50 fps The main IPs have been profiled in regard to their parameters IP genericity permits to change these parameters to aim for

  • ptimal performance for various FPGAs

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 33 / 35

slide-83
SLIDE 83

current prototypes

Future works

Implement the full system (multi-resolution, decent frame size) on a camera/FPGA couple Validate the behavior of the system and tune the parameters for each use case Design and implement log-polar transform as hardware IPs

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 34 / 35

slide-84
SLIDE 84

current prototypes

Thanks for your attention. Do you have any question ?

HW/SW architecture of a bio-inspired robotic vision system |

  • T. Lefebvre

| 5 avril 2012 35 / 35