A 240x180 120dB 10mW 12us latency sparse output vision sensor for - PowerPoint PPT Presentation

A 240x180 120dB 10mW 12us ‐ latency sparse output vision sensor for mobile applications Raphael Berner, Christian Brandli, Minhao Yang, Shih ‐ Chii Liu and Tobi Delbruck Inst. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland 1

Machine Vision for Mobile Applications • ENERGY matters because it is limited • LATENCY is crucial for real time interaction With conventional imager • Low frame rate: low energy / high latency • High frame rate: low latency / high energy Most of the time and energy spent on redundant information in and between frames WE NEED SMARTER WAYS OF SENSING & PROCESSING 2

Simplified Dynamic Vision Sensor ( DVS ) pixel architecture Lichtsteiner et al. JSSC 2009 3

Example DVS application ‐ RoboGoalie 4

Example DVS application – RoboGoalie Achieves 550Hz update rate at 4% laptop CPU load Achieves 550fps (equivalent) at 4% CPU load 5 Delbruck et al., 2009

The apsDVS chip adds an intensity readout to the DVS Intensity reset threshold I ON Intensity Change value events log I OFF Event comparators reset (ganglion cells) I change amplifier (bipolar cells) photoreceptor 6

apsDVS pixel (aps=Active Pixel Sensor) 7

The apsDVS chip SBRET10 • 0.18um 1P6M CMOS • MiM capacitors • 240x180 pixels • 22% fill factor • 12mW power Row APS readout Column APS readout 8

The apsDVS can enhance machine vision in several ways • Low APS frame rate  Low power • DVS output  Low latency, sparse data Combined advantages: 1. DVS delivers motion features for free 2. Continuous tracking solves correspondence problem 3. DVS can be used to control frame rate or ROI Slide 12

The apsDVS in Comparison This work (apsDVS) ATIS (Posch et al.) Intensity readout APS Time domain Image Dynamic Range 57 dB 125 dB Image FPN 1 % <0.25 % Pixel size [um 2 ] 18.5 x 18.5 30 x 30 Fill factor 22% 15% Pixel complexity 44 FET, 2 C, 1 PD 77 FET, 2 C, 2 PD Fixed integration time YES NO Power Consumption ~10mW (240x180) ~100mW (304x240) 13

Thank you Christian Dr. habil. Dr. Raphael Prof. Tobi Minhao Brandli Shih ‐ Chii Liu Berner Delbruck Yang sensors.ini.uzh.ch Funding Organizations University of Zurich ETH Zurich EU 7th framework program (SeeBetter) Swiss National Science Foundation (NCCR Robotics) 14

SBRET10 pixel layout 22% fill factor 18.5um 15

Technology UMC 180 6M 1P Chip size 5x5mm Design APS-DVS array size 240x180 Pixel size 18.5 x18.5 um 2 Supply voltage(s) 3.3V/1.8V Pixel architecture apsDVS, 44T+2C. Fill factor 22% Power consumption 7.4mW low activity 13.5mW high activity DVS event threshold limit 12% contrast Measured DVS illumination operating range 120dB (down to 0.1lux) DVS latency 12us APS readout frame rate 40 FPS APS FPN (midlevel) 1% of full signal swing Readout Noise <2mV Output Voltage Range 1.1V APS Dynamic Range ~55dB 16

Simplified Dynamic Vision Sensor ( DVS ) pixel architecture A random variation  log I Lichtsteiner et al. JSSC 2009 17

A 240x180 120dB 10mW 12us latency sparse output vision sensor for - PowerPoint PPT Presentation

A 240x180 120dB 10mW 12us latency sparse output vision sensor for mobile applications Raphael Berner, Christian Brandli, Minhao Yang, Shih Chii Liu and Tobi Delbruck Inst. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland

The Input/Output Complexity of Sparse Matrix Multiplication Rasmus Pagh, Morten St ockel IT

Output Perception Colour models Managing output 1 Human Elements of Graphical Output

Output Perception Colour models Managing output 1 Human Elements of Graphical Output

Output-sensitive algorithms for sumset and sparse polynomial multiplication Daniel S. Roche

Networks 1a. Define a latency procedure that takes one input, a list of lists of bit arrival times,

The Input/Output Complexity of Sparse Matrix Multiplication Rasmus Pagh 1 , Morten St ockel 2 1

Sparse Matrices Example Of Sparse Matrices diagonal tridiagonal sparse many elements are

Sparse Matrices sparse many elements are zero dense few elements are zero Example Of

Sparse Flat Neighborhood Networks (SFNNs): Scalable Guaranteed Pairwise Bandwidth & Unit

Reconstructing Objects with Sparse Boundaries: Total Variation vs. Discrete Tomography Willem Jan

Enc Encoding ding, F , Fas ast and Slo t and Slow: w: Low-Latency Video Processing Using

16. Recursion 2 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20 Output: 12

Real-Time Structure and Object- Model Aware Sparse SLAM ARC Centre of Excellence for Robotic

Fundamentals for Low Latency Communications H. Vincent Poor Princeton University Supported in

Is It Latency or Do You Just Suck? Latency Can Kill: Precision and Deadline in Online Games

Chapter 12 Overview Devices and Output Visual Output Dynamic Visualizations Sound

A Potpourri of Nonlinear Algebra Chris Hillar 2 Det 2 , 2 , 2 ( A ) = 1

Silicon retina technology Tobi Delbruck Inst. of Neuroinformatics , University of Zurich and ETH

Convolu'on NEU 466M Instructor: Professor Ila R. Fiete Spring 2016 Convolu'on g: a 'me-varying

From retina to statistical physics Bruno Cessac NeuroMathComp Team,INRIA Sophia Antipolis,France.

Image Processing Todays Class Image Representations: Matrices Image Representations: RGB,

~I ~ ~ ~ -~ 'PA~fc lf~fc. ~ g.&~ e.s . . i 'i ~ . ,.., 'j VlifHJ_ I , cl. c

for Emergency Room Personnel Daphne W. Denham, MD FACS FACCWS Healing with Hyperbarics of North

Clostridium perfringens Clostridium perfringens Dean O. Cliver (materials from M. N. Hajmeer)

A 240x180 120dB 10mW 12us latency sparse output vision sensor for - PowerPoint PPT Presentation

A 240x180 120dB 10mW 12us latency sparse output vision sensor for mobile applications Raphael Berner, Christian Brandli, Minhao Yang, Shih Chii Liu and Tobi Delbruck Inst. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland

The Input/Output Complexity of Sparse Matrix Multiplication Rasmus Pagh, Morten St ockel IT

Output Perception Colour models Managing output 1 Human Elements of Graphical Output

Output Perception Colour models Managing output 1 Human Elements of Graphical Output

Output-sensitive algorithms for sumset and sparse polynomial multiplication Daniel S. Roche

Networks 1a. Define a latency procedure that takes one input, a list of lists of bit arrival times,

The Input/Output Complexity of Sparse Matrix Multiplication Rasmus Pagh 1 , Morten St ockel 2 1

Sparse Matrices Example Of Sparse Matrices diagonal tridiagonal sparse many elements are

Sparse Matrices sparse many elements are zero dense few elements are zero Example Of

Sparse Flat Neighborhood Networks (SFNNs): Scalable Guaranteed Pairwise Bandwidth &amp; Unit

Reconstructing Objects with Sparse Boundaries: Total Variation vs. Discrete Tomography Willem Jan

Enc Encoding ding, F , Fas ast and Slo t and Slow: w: Low-Latency Video Processing Using

16. Recursion 2 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20 Output: 12

Real-Time Structure and Object- Model Aware Sparse SLAM ARC Centre of Excellence for Robotic

Fundamentals for Low Latency Communications H. Vincent Poor Princeton University Supported in

Is It Latency or Do You Just Suck? Latency Can Kill: Precision and Deadline in Online Games

Chapter 12 Overview Devices and Output Visual Output Dynamic Visualizations Sound

A Potpourri of Nonlinear Algebra Chris Hillar 2 Det 2 , 2 , 2 ( A ) = 1

Silicon retina technology Tobi Delbruck Inst. of Neuroinformatics , University of Zurich and ETH

Convolu'on NEU 466M Instructor: Professor Ila R. Fiete Spring 2016 Convolu'on g: a 'me-varying

From retina to statistical physics Bruno Cessac NeuroMathComp Team,INRIA Sophia Antipolis,France.

Image Processing Todays Class Image Representations: Matrices Image Representations: RGB,

~I ~ ~ ~ -~ 'PA~fc lf~fc. ~ g.&amp;~ e.s . . i 'i ~ . ,.., 'j VlifHJ_ I , cl. c

for Emergency Room Personnel Daphne W. Denham, MD FACS FACCWS Healing with Hyperbarics of North

Clostridium perfringens Clostridium perfringens Dean O. Cliver (materials from M. N. Hajmeer)

Sparse Flat Neighborhood Networks (SFNNs): Scalable Guaranteed Pairwise Bandwidth & Unit

~I ~ ~ ~ -~ 'PA~fc lf~fc. ~ g.&~ e.s . . i 'i ~ . ,.., 'j VlifHJ_ I , cl. c