Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms - - PowerPoint PPT Presentation

▶
real time in situ intelligent video
SMART_READER_LITE
LIVE PREVIEW

Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms - - PowerPoint PPT Presentation

Jul 28, 2023 238 likes •547 views

Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms Christiaan Gribble Applied Technology Operation SURVICE Engineering GTC 2017 Acknowledgments University of Washington Steve Brunton, PhD Ben Erichson, PhD

slide-1
SLIDE 1

Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

Christiaan Gribble

Applied Technology Operation SURVICE Engineering GTC 2017

slide-2
SLIDE 2
  • University of Washington

– Steve Brunton, PhD – Ben Erichson, PhD – Nathan Kutz, PhD

  • SURVICE Engineering

– Rob Baltrusch – Mark Butkiewicz – Shawn Recker, PhD

Acknowledgments

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

2

slide-3
SLIDE 3

Overview

Combines modern data reduction & analysis techniques with machine learning via DNNs and massively parallel computing architectures to enable next-gen ISR

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

3

slide-4
SLIDE 4

Key elements

NVIDIA GPUs Advanced UIs Machine learning Compressed DMD

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

4

slide-5
SLIDE 5

Key elements

NVIDIA GPUs Advanced UIs Machine learning Compressed DMD

  • Provides least-squares fitting of temporal data
  • Enables real-time performance with compression
  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

5

slide-6
SLIDE 6

Key elements

Compressed DMD NVIDIA GPUs Advanced UIs Machine learning

  • Employs deep neural networks
  • Supports advanced object detection tasks
  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

6

slide-7
SLIDE 7

Key elements

Compressed DMD Machine learning Advanced UIs

  • Permit fast training for design & optimization
  • Enable inference on mobile platforms

NVIDIA GPUs

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

7

slide-8
SLIDE 8

Key elements

Compressed DMD Machine learning NVIDIA GPUs Advanced UIs

  • Extend users’ natural abilities
  • Reduce cognitive burden
  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

8

slide-9
SLIDE 9

Dynamic mode decomposition

  • Frame Xt → snapshot of dynamics
  • Xt+1 = AXt for A : Rn → Rn

[Grosek & Kutz 2014]

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

9

slide-10
SLIDE 10

Dynamic mode decomposition

  • Frame Xt → snapshot of dynamics
  • Xt+1 = AXt for A : Rn → Rn

DMD estimates A and its eigenvalues to characterize system dynamics

[Grosek & Kutz 2014]

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

10

slide-11
SLIDE 11

Dynamic mode decomposition

video stream space … modes reshaped video … space time amplitudes

≈

dynamic mode decomposition evolution time … flattened frame [Erichson et al. 2016]

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

11

slide-12
SLIDE 12

Background/foreground separation

[Grosek & Kutz 2014]

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

12

slide-13
SLIDE 13

Compressed DMD

Accelerates DMD by operating on compressed data

[Erichson et al. 2016]

X, X’

data modes full DMD

𝚾Y, 𝚳Y

cDMD

Y, Y’

compressed C

𝚾, 𝚳

𝚾 = X’VY SY

  • 1WY
  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

13

slide-14
SLIDE 14

Compressed DMD

→

compressed video [Erichson et al. 2016] … compression matrix reshaped video …

×

video stream reshaped video … space time … flattened frame

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

14

slide-15
SLIDE 15

Compressed DMD

space … modes compressed video … space time amplitudes

≈

dynamic mode decomposition evolution time [Erichson et al. 2016]

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

15

slide-16
SLIDE 16

Background/foreground separation

[Erichson et al. 2016]

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

16

slide-17
SLIDE 17

Implementation

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

17

slide-18
SLIDE 18

Intel Core i5

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

18

Compression Level

50 100 150 200 250 300 350 400 450 500 3840x2160 1920x1080 1280x720 1024x768 640x480

Runtime Performance (frames per second) Video Resolution

1 0.5 0.1 0.01 0.001

slide-19
SLIDE 19

NVIDIA Tesla K40c

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

19

Compression Level

100 200 300 400 500 600 700 800 900 3840x2160 1920x1080 1280x720 1024x768 640x480

Runtime Performance (frames per second) Video Resolution

1 0.5 0.1 0.01 0.001

slide-20
SLIDE 20

Jetson TX1 – CPU

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

20

Compression Level

5 10 15 20 25 30 35 40 45 50 1920x1080 1280x720 1024x768 640x480

Runtime Performance (frames per second) Video Resolution

1 0.5 0.1 0.01 0.001

slide-21
SLIDE 21

Jetson TX1 – GPU

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

21

Compression Level

10 20 30 40 50 60 70 80 90 100 1920x1080 1280x720 1024x768 640x480

Runtime Performance (frames per second) Video Resolution

1 0.5 0.1 0.01 0.001

slide-22
SLIDE 22

Object detection

DMD Haar classifier Traditional approach DNN approach

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

22

CNN

Mini-quad DJI
slide-23
SLIDE 23

DetectNet

[Image source: NVIDIA Parallel Forall]

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

23

Input FCN Output Error

slide-24
SLIDE 24

Initial results

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

24

slide-25
SLIDE 25

In progress

  • Enhance DMD
  • Fabricate IVA modules
  • Integrate advanced UIs
  • Deploy on mobile platforms

[Kutz et al. 2016]

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

25

slide-26
SLIDE 26

In progress

  • Enhance DMD
  • Fabricate IVA modules
  • Integrate advanced UIs
  • Deploy on mobile platforms

Sentinel-enabled IVA module

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

26

slide-27
SLIDE 27

In progress

  • Enhance DMD
  • Fabricate IVA modules
  • Integrate advanced UIs
  • Deploy on mobile platforms
  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

27

slide-28
SLIDE 28

In progress

  • Enhance DMD
  • Fabricate IVA modules
  • Integrate advanced UIs
  • Deploy on mobile platforms

JTARV – Hoverbike

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

28

slide-29
SLIDE 29

Brunton, B. W., Brunton, S. L., Proctor, J. L. & Kutz, J. N. (2015) Optimal sensor placement and enhanced sparsity for classification. SIAM Journal on Applied Mathematics, https://arxiv.org/abs/1310.4217. Erichson, N. B., Brunton, S. L., & Kutz, J. N. (2015) Compressed dynamic mode decomposition for background modeling. Journal of Real-Time Image Processing, https://arxiv.org/abs/1512.04205v2. Grosek, J. & Kutz, J. N. (2014) Dynamic mode decomposition for real-time background/foreground separation in video. IEEE Transactions on Pattern Analysis & Machine Learning, https://arxiv.org/abs/1404.7592. Kutz, J. N., Fu, X., & Brunton, S. L. (2016) Multi-resolution dynamic mode decomposition. SIAM Journal on Applied Dynamical Systems, https://arxiv.org/abs/1506.00564.

Key DMD references

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

29

slide-30
SLIDE 30

Contact information

Address

Applied Technology Operation SURVICE Engineering 4695 Millennium Drive Belcamp, MD 21017

E-mail

christiaan.gribble@survice.com

Web

http://www.survice.com/employees/~cgribble/

  • C. Gribble, Real-Time In-Situ Intelligent Video Analytics for Mobile Platforms

30