Algorithm-SoC Co-Design for Mobile Continuous Vision Yuhao Zhu - - PowerPoint PPT Presentation

Algorithm-SoC Co-Design for Mobile Continuous Vision

Yuhao Zhu

Department of Computer Science University of Rochester with Anand Samajdar, Georgia Tech Matthew Mattina, ARM Research Paul Whatmough, ARM Research

Mobile Continuous Vision: Excessive Energy Consumption

Mobile Continuous Vision: Excessive Energy Consumption

720p, 30 FPS

Mobile Continuous Vision: Excessive Energy Consumption

Energy Budget: (under 3 W TDP) 109 nJ/pixel

720p, 30 FPS

Mobile Continuous Vision: Excessive Energy Consumption

Energy Budget: (under 3 W TDP) 109 nJ/pixel Object Detection Energy Consumption 1400 nJ/pixel

720p, 30 FPS

Application Drivers for Continuous Vision

3

Autonomous Drones

Application Drivers for Continuous Vision

3

Autonomous Drones ADAS

Application Drivers for Continuous Vision

3

Autonomous Drones Augmented Reality ADAS

Application Drivers for Continuous Vision

3

Autonomous Drones Augmented Reality ADAS Security Camera

Expanding the Scope

4

Vision Kernels

RGB Frames Semantic Results Conventional Scope

Expanding the Scope

4

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons Conventional Scope

Expanding the Scope

Our Scope

4

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

Expanding the Scope

Our Scope

4

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons Motion Metadata

Expanding the Scope

Our Scope

4

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons Motion Metadata

f(xt) =

Expanding the Scope

Our Scope

4

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons Motion Metadata diff (motion)

f(xt) = (xt ⊖ xt-1)

Expanding the Scope

Our Scope

4

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons Motion Metadata diff (motion)

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

Expanding the Scope

Our Scope

4

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons Motion Metadata diff (motion) synthesis

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

Expanding the Scope

Our Scope

4

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons Motion Metadata diff (motion) synthesis cheap

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

Expanding the Scope

Our Scope

4

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons Motion Metadata diff (motion) synthesis Motion-based Synthesis cheap

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

Getting Motion Data

5

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons Motion Metadata

Getting Motion Data

5

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

Conversion Demosaic … Bayer Domain Dead Pixel Correction

…

YUV Domain Temporal Denoising

…

Getting Motion Data

5

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

Conversion Demosaic … Bayer Domain Dead Pixel Correction

…

YUV Domain Temporal Denoising

…

Getting Motion Data

5

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

Conversion Demosaic … Bayer Domain Dead Pixel Correction

…

YUV Domain Temporal Denoising

…

Frame k

Getting Motion Data

5

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

Conversion Demosaic … Bayer Domain Dead Pixel Correction

…

YUV Domain Temporal Denoising

…

Frame k

<u, v>

Getting Motion Data

5

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

Conversion Demosaic … Bayer Domain Dead Pixel Correction

…

YUV Domain Temporal Denoising

…

Frame k-1 Frame k

<u, v> <x, y>

Getting Motion Data

5

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

Conversion Demosaic … Bayer Domain Dead Pixel Correction

…

YUV Domain Temporal Denoising

…

Frame k-1 Frame k

<u, v> <x, y>

Motion Vector = <x - u, y - v>

Getting Motion Data

5

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

Conversion Demosaic … Bayer Domain Dead Pixel Correction

…

YUV Domain Temporal Denoising

…

Motion Info. Frame k-1 Frame k

<u, v> <x, y>

Motion Vector = <x - u, y - v>

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors

Inference (I-Frame) Extrapolation (E-Frame) Inference (I-Frame) Extrapolation (E-Frame)

Extrapolation Window = 2

Extrapolation (E-Frame)

Extrapolation Window = 3 t4 t0 t1 t2 t3

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors

Inference (I-Frame) Extrapolation (E-Frame) Inference (I-Frame) Extrapolation (E-Frame)

Extrapolation Window = 2

Extrapolation (E-Frame)

Extrapolation Window = 3 t4 t0 t1 t2 t3

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors

Inference (I-Frame) Extrapolation (E-Frame) Inference (I-Frame) Extrapolation (E-Frame)

Extrapolation Window = 2

Extrapolation (E-Frame)

Extrapolation Window = 3 t4 t0 t1 t2 t3

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors ▸ Address three challenges:

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors ▸ Address three challenges:

▹ Handle deformable parts

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors ▸ Address three challenges:

▹ Handle deformable parts ▹ Filter motion noise

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors ▸ Address three challenges:

▹ Handle deformable parts ▹ Filter motion noise ▹ When to inference vs. extrapolate?

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors ▸ Address three challenges:

▹ Handle deformable parts ▹ Filter motion noise ▹ When to inference vs. extrapolate? ▹ See paper for details!

Synthesis Operation

6

f(xt) = f(x1, …, t-1) ⊕ (xt ⊖ xt-1)

diff (motion) synthesis Motion-based Synthesis

▸ Synthesis operation: Extrapolate based on motion vectors ▸ Address three challenges:

▹ Handle deformable parts ▹ Filter motion noise ▹ When to inference vs. extrapolate? ▹ See paper for details!

Computationally efficient:

Extrapolation: 10K operations/frame CNN Inference: 50B operations/frame

7

Euphrates

An Algorithm-SoC Co-Designed System for Energy-Efficient Mobile Continuous Vision

7

Euphrates

An Algorithm-SoC Co-Designed System for Energy-Efficient Mobile Continuous Vision

Algorithm

Motion-based tracking and detection synthesis.

7

Euphrates

An Algorithm-SoC Co-Designed System for Energy-Efficient Mobile Continuous Vision

SoC

Exploits synergies across IP

• blocks. Enables task autonomy.

Algorithm

Motion-based tracking and detection synthesis.

7

Euphrates

An Algorithm-SoC Co-Designed System for Energy-Efficient Mobile Continuous Vision

Results

66% energy saving & 1% accuracy loss with RTL/measurement.

SoC

Exploits synergies across IP

• blocks. Enables task autonomy.

Algorithm

Motion-based tracking and detection synthesis.

7

Euphrates

An Algorithm-SoC Co-Designed System for Energy-Efficient Mobile Continuous Vision

Results

66% energy saving & 1% accuracy loss with RTL/measurement.

SoC

Exploits synergies across IP

• blocks. Enables task autonomy.

Algorithm

Motion-based tracking and detection synthesis.

SoC Architecture

8

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

SoC Architecture

8

CNN Accelerator

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

SoC Architecture

8

Image Signal Processor CNN Accelerator

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

SoC Architecture

8

Image Signal Processor CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

Vision Kernels

RGB Frames Semantic Results

Imaging

Photons

SoC Architecture

9

Image Signal Processor CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

DRAM Display

SoC Architecture

9

Image Signal Processor CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine

SoC

DRAM Display Frame Buffer

SoC Architecture

9

Image Signal Processor CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine

SoC

DRAM Display Frame Buffer

SoC Architecture

9

Image Signal Processor CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine

SoC

DRAM Display Frame Buffer

SoC Architecture

9

Image Signal Processor CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine

SoC

DRAM Display Frame Buffer

SoC Architecture

9

Image Signal Processor CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine

SoC

DRAM Display Frame Buffer

SoC Architecture

9

CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine Image Signal Processor

SoC

DRAM Display Frame Buffer

SoC Architecture

9

CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine Image Signal Processor

Metadata

SoC

DRAM Display Frame Buffer

SoC Architecture

9

CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine Image Signal Processor

Metadata

1

SoC

DRAM Display Frame Buffer

SoC Architecture

9

CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine Image Signal Processor

Motion Controller Metadata

1 2

DRAM Display Frame Buffer

SoC Architecture

9

CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine Image Signal Processor

Motion Controller Metadata

1 2

DRAM Display Frame Buffer

SoC Architecture

9

CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine Image Signal Processor

Motion Controller Metadata

1 2

DRAM Display Frame Buffer

SoC Architecture

9

CNN Accelerator

Camera Sensor Sensor Interface

On-chip Interconnect

CPU (Host)

Memory Controller DMA Engine Image Signal Processor

Motion Controller Metadata

1 2

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC

10

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC ▸ Design decision: transfer MVs through DRAM

10

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC ▸ Design decision: transfer MVs through DRAM

▹ One 1080p frame: 8KB MV traffic vs. ~6MB pixel data

10

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC ▸ Design decision: transfer MVs through DRAM

▹ One 1080p frame: 8KB MV traffic vs. ~6MB pixel data ▹ Easy to piggyback on the existing SoC communication scheme

10

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC ▸ Design decision: transfer MVs through DRAM

▹ One 1080p frame: 8KB MV traffic vs. ~6MB pixel data ▹ Easy to piggyback on the existing SoC communication scheme

▸ Light-weight modification to ISP Sequencer

10

Temporal Denoising Stage

Motion Estimation Motion Compensation SRAM DMA Demosaic Color Balance

ISP Internal Interconnect SoC Interconnect

ISP Pipeline

Frame Buffer (DRAM) ISP Sequencer

Noisy Frame Denoised Frame Prev. Noisy Frame Prev. Denoised Frame

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC ▸ Design decision: transfer MVs through DRAM

▹ One 1080p frame: 8KB MV traffic vs. ~6MB pixel data ▹ Easy to piggyback on the existing SoC communication scheme

▸ Light-weight modification to ISP Sequencer

10

Temporal Denoising Stage

Motion Estimation Motion Compensation SRAM DMA Demosaic Color Balance

ISP Internal Interconnect SoC Interconnect

ISP Pipeline

Frame Buffer (DRAM) ISP Sequencer

Noisy Frame Denoised Frame Prev. Noisy Frame Prev. Denoised Frame

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC ▸ Design decision: transfer MVs through DRAM

▹ One 1080p frame: 8KB MV traffic vs. ~6MB pixel data ▹ Easy to piggyback on the existing SoC communication scheme

▸ Light-weight modification to ISP Sequencer

10

Temporal Denoising Stage

Motion Estimation Motion Compensation SRAM DMA Demosaic Color Balance

ISP Internal Interconnect SoC Interconnect

ISP Pipeline

Frame Buffer (DRAM) ISP Sequencer

Noisy Frame Denoised Frame Prev. Noisy Frame Prev. Denoised Frame

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC ▸ Design decision: transfer MVs through DRAM

▹ One 1080p frame: 8KB MV traffic vs. ~6MB pixel data ▹ Easy to piggyback on the existing SoC communication scheme

▸ Light-weight modification to ISP Sequencer

10

Temporal Denoising Stage

Motion Estimation Motion Compensation SRAM DMA Demosaic Color Balance

ISP Internal Interconnect SoC Interconnect

ISP Pipeline

Frame Buffer (DRAM) ISP Sequencer

Noisy Frame Denoised Frame Prev. Noisy Frame Prev. Denoised Frame

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC ▸ Design decision: transfer MVs through DRAM

▹ One 1080p frame: 8KB MV traffic vs. ~6MB pixel data ▹ Easy to piggyback on the existing SoC communication scheme

▸ Light-weight modification to ISP Sequencer

10

MVs

Temporal Denoising Stage

Motion Estimation Motion Compensation SRAM DMA Demosaic Color Balance

ISP Internal Interconnect SoC Interconnect

ISP Pipeline

Frame Buffer (DRAM) ISP Sequencer

Noisy Frame Denoised Frame Prev. Noisy Frame Prev. Denoised Frame

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC ▸ Design decision: transfer MVs through DRAM

▹ One 1080p frame: 8KB MV traffic vs. ~6MB pixel data ▹ Easy to piggyback on the existing SoC communication scheme

▸ Light-weight modification to ISP Sequencer

10

MVs

Temporal Denoising Stage

Motion Estimation Motion Compensation SRAM DMA Demosaic Color Balance

ISP Internal Interconnect SoC Interconnect

ISP Pipeline

Frame Buffer (DRAM) ISP Sequencer

Noisy Frame Denoised Frame Prev. Noisy Frame Prev. Denoised Frame

ISP Augmentation

▸ Expose motion vectors to the rest of the SoC ▸ Design decision: transfer MVs through DRAM

▹ One 1080p frame: 8KB MV traffic vs. ~6MB pixel data ▹ Easy to piggyback on the existing SoC communication scheme

▸ Light-weight modification to ISP Sequencer

10

MVs

Motion Controller IP

11

Extrapolation Unit

Motion Vector Buffer DMA Sequencer (FSM)

ROI Selection

ROI 4-Way SIMD Unit Scalar MVs New ROI

MMap Regs

ROI Winsize Base Addrs

Conf

Motion Controller IP

11

Extrapolation Unit

Motion Vector Buffer DMA Sequencer (FSM)

ROI Selection

ROI 4-Way SIMD Unit Scalar MVs New ROI

MMap Regs

ROI Winsize Base Addrs

Conf

Motion Controller IP

11

Extrapolation Unit

Motion Vector Buffer DMA Sequencer (FSM)

ROI Selection

ROI 4-Way SIMD Unit Scalar MVs New ROI

MMap Regs

ROI Winsize Base Addrs

Conf

Motion Controller IP

11

Extrapolation Unit

Motion Vector Buffer DMA Sequencer (FSM)

ROI Selection

ROI 4-Way SIMD Unit Scalar MVs New ROI

MMap Regs

ROI Winsize Base Addrs

Conf

Motion Controller IP

11

Extrapolation Unit

Motion Vector Buffer DMA Sequencer (FSM)

ROI Selection

ROI 4-Way SIMD Unit Scalar MVs New ROI

MMap Regs

ROI Winsize Base Addrs

Conf

Motion Controller IP

▸ Why not directly augment the CNN accelerator, but a new IP?

▹Independent of vision algo./arch implementation

11 Extrapolation Unit

Motion Vector Buffer DMA Sequencer (FSM)

ROI Selection

ROI 4-Way SIMD Unit Scalar MVs New ROI

MMap Regs

ROI Winsize Base Addrs

Conf

Motion Controller IP

▸ Why not directly augment the CNN accelerator, but a new IP?

▹Independent of vision algo./arch implementation

▸ Why not synthesize in CPU, but a new IP?

▹Switch-off CPU to enable “always-on” vision

11 Extrapolation Unit

Motion Vector Buffer DMA Sequencer (FSM)

ROI Selection

ROI 4-Way SIMD Unit Scalar MVs New ROI

MMap Regs

ROI Winsize Base Addrs

Conf

Motion Controller CNN Accelerator

Motion Controller IP

12 Extrapolation Unit

Motion Vector Buffer DMA Sequencer (FSM)

ROI Selection

ROI 4-Way SIMD Unit Scalar MVs New ROI

MMap Regs

ROI Winsize Base Addrs

Conf

ISP

SoC Interconnect

▸ Why not directly augment the CNN accelerator, but a new IP?

▹Independent of vision algo./arch implementation

▸ Why not synthesize in CPU, but a new IP?

▹Switch-off CPU to enable “always-on” vision

13

Euphrates

An Algorithm-SoC Co-Designed System for Energy-Efficient Mobile Continuous Vision

Algorithm

Motion-based tracking and detection synthesis.

SoC

Exploits synergies across IP

• blocks. Enables task autonomy.

Results

66% energy saving & 1% accuracy loss with RTL/measurement.

Experimental Setup

▸ In-house simulator modeling a commercial mobile SoC: Nvidia Tegra X2

▹ Real board measurement

14

Experimental Setup

▸ In-house simulator modeling a commercial mobile SoC: Nvidia Tegra X2

▹ Real board measurement

▸ Develop RTL models for IPs unavailable on TX2

▹ CNN Accelerator (651 mW, 1.58 mm2) ▹ Motion Controller (2.2 mW, 0.035 mm2)

14

Experimental Setup

▸ In-house simulator modeling a commercial mobile SoC: Nvidia Tegra X2

▹ Real board measurement

▸ Develop RTL models for IPs unavailable on TX2

▹ CNN Accelerator (651 mW, 1.58 mm2) ▹ Motion Controller (2.2 mW, 0.035 mm2)

14

▸ Evaluate on Object Tracking and Object Detection

▹ Important domains that are building blocks for many vision applications ▹ IP vendors have started shipping standalone tracking/detection IPs

Experimental Setup

▸ In-house simulator modeling a commercial mobile SoC: Nvidia Tegra X2

▹ Real board measurement

▸ Develop RTL models for IPs unavailable on TX2

▹ CNN Accelerator (651 mW, 1.58 mm2) ▹ Motion Controller (2.2 mW, 0.035 mm2)

14

▸ Evaluate on Object Tracking and Object Detection

▹ Important domains that are building blocks for many vision applications ▹ IP vendors have started shipping standalone tracking/detection IPs

Experimental Setup

▸ In-house simulator modeling a commercial mobile SoC: Nvidia Tegra X2

▹ Real board measurement

▸ Develop RTL models for IPs unavailable on TX2

▹ CNN Accelerator (651 mW, 1.58 mm2) ▹ Motion Controller (2.2 mW, 0.035 mm2)

14

▸ Evaluate on Object Tracking and Object Detection

▹ Important domains that are building blocks for many vision applications ▹ IP vendors have started shipping standalone tracking/detection IPs

▸ Object Detection

▹ Baseline CNN: YOLOv2 (state-of-the-art detection results)

Experimental Setup

▸ In-house simulator modeling a commercial mobile SoC: Nvidia Tegra X2

▹ Real board measurement

▸ Develop RTL models for IPs unavailable on TX2

▹ CNN Accelerator (651 mW, 1.58 mm2) ▹ Motion Controller (2.2 mW, 0.035 mm2)

14

▸ Evaluate on Object Tracking and Object Detection

▹ Important domains that are building blocks for many vision applications ▹ IP vendors have started shipping standalone tracking/detection IPs

▸ Object Detection

▹ Baseline CNN: YOLOv2 (state-of-the-art detection results)

▸ SCALESim: A systolic array-based, cycle-accurate CNN accelerator

• simulator. https://github.com/ARM-software/SCALE-Sim.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 Y O L O v 2

Evaluation Results

15

Accuracy

0.1 0.2 0.3 0.4 0.5 0.6 0.7 Y O L O v 2 0.25 0.5 0.75 1 Y O L O v 2

Evaluation Results

15

Accuracy

• Norm. Energy

0.1 0.2 0.3 0.4 0.5 0.6 0.7 Y O L O v 2 0.25 0.5 0.75 1 Y O L O v 2 Y O L O v 2 E W

• 2

E W

• 4

E W

• 8

E W

• 1

6 E W

• 3

2

Evaluation Results

15

Accuracy

• Norm. Energy

EW = Extrapolation Window

0.1 0.2 0.3 0.4 0.5 0.6 0.7 Y O L O v 2 0.25 0.5 0.75 1 Y O L O v 2 Y O L O v 2 E W

• 2

E W

• 4

E W

• 8

E W

• 1

6 E W

• 3

2 Y O L O v 2 E W

• 2

E W

• 4

E W

• 8

E W

• 1

6 E W

• 3

2

Evaluation Results

15

Accuracy

• Norm. Energy

EW = Extrapolation Window

0.1 0.2 0.3 0.4 0.5 0.6 0.7 Y O L O v 2 0.25 0.5 0.75 1 Y O L O v 2 Y O L O v 2 E W

• 2

E W

• 4

E W

• 8

E W

• 1

6 E W

• 3

2 Y O L O v 2 E W

• 2

E W

• 4

E W

• 8

E W

• 1

6 E W

• 3

2

Evaluation Results

15

Accuracy

• Norm. Energy

66% system energy saving with ~ 1% accuracy loss.

EW = Extrapolation Window

Scale-down CNN

0.1 0.2 0.3 0.4 0.5 0.6 0.7 Y O L O v 2 0.25 0.5 0.75 1 Y O L O v 2 Y O L O v 2 E W

• 2

E W

• 4

E W

• 8

E W

• 1

6 E W

• 3

2 Y O L O v 2 E W

• 2

E W

• 4

E W

• 8

E W

• 1

6 E W

• 3

2 Y O L O v 2 E W

• 4

E W

• 1

6 T i n y Y O L O

Evaluation Results

15

Accuracy

• Norm. Energy

66% system energy saving with ~ 1% accuracy loss.

EW = Extrapolation Window

0.1 0.2 0.3 0.4 0.5 0.6 0.7 Y O L O v 2 0.25 0.5 0.75 1 Y O L O v 2 Y O L O v 2 E W

• 2

E W

• 4

E W

• 8

E W

• 1

6 E W

• 3

2 Y O L O v 2 E W

• 2

E W

• 4

E W

• 8

E W

• 1

6 E W

• 3

2 Y O L O v 2 E W

• 4

E W

• 1

6 T i n y Y O L O

Evaluation Results

15

Accuracy

• Norm. Energy

66% system energy saving with ~ 1% accuracy loss. More efficient than simply scaling-down the CNN.

EW = Extrapolation Window

Conclusions

16

Conclusions

16

▸ We must expand our focus from isolated accelerators to holistic SoC architecture.

Conclusions

16

▸ We must expand our focus from isolated accelerators to holistic SoC architecture.

Conclusions

16

▸ Euphrates co-designs the SoC with a motion-based synthesis algorithm. ▸ We must expand our focus from isolated accelerators to holistic SoC architecture.

Conclusions

16

▸ Euphrates co-designs the SoC with a motion-based synthesis algorithm. ▸ We must expand our focus from isolated accelerators to holistic SoC architecture. ▸ 66% SoC energy savings with ~1% accuracy

• loss. More efficient than scaling-down CNNs.

Thank you!

17

Georgia Tech Anand Samajdar Paul Whatmough ARM Research Matt Mattina ARM Research