Approximate Query Service on Autonomous IoT Cameras Mengwei Xu 1 , - - PowerPoint PPT Presentation

▶
approximate query service on autonomous iot cameras
SMART_READER_LITE
LIVE PREVIEW

Approximate Query Service on Autonomous IoT Cameras Mengwei Xu 1 , - - PowerPoint PPT Presentation

Mar 25, 2024 579 likes •991 views

Approximate Query Service on Autonomous IoT Cameras Mengwei Xu 1 , Xiwen Zhang 2 , Yunxin Liu 3 Gang Huang 1 , Xuanzhe Liu 1 , Felix Xiaozhu Lin 2 1 Peking University, 2 Purdue University, 3 Microsoft Research, 1 Video Analytics is a Killer App

slide-1
SLIDE 1

Approximate Query Service on Autonomous IoT Cameras

Mengwei Xu1, Xiwen Zhang2, Yunxin Liu3 Gang Huang1, Xuanzhe Liu1, Felix Xiaozhu Lin2

1Peking University, 2Purdue University, 3Microsoft Research,

1

slide-2
SLIDE 2

Video Analytics is a Killer App

  • Busy cross roads
  • Retailing store
  • Sports stadium
  • Parking lots
  • …

2

slide-3
SLIDE 3

Video Analytics is a Killer App

  • Busy cross roads
  • Retailing store
  • Sports stadium
  • Parking lots
  • …

Urban, residential areas ü Wired electricity ü Good internet

3

slide-4
SLIDE 4

Video Analytics is a Killer App

  • Busy cross roads
  • Retailing store
  • Sports stadium
  • Parking lots
  • …

Urban, residential areas Rural, off-grid areas

  • Construction sites
  • Cattle farms
  • Highways
  • Wildlifes
  • …

?

4

slide-5
SLIDE 5

Autonomous Camera

  • Energy-independent and Compute-independent

5

slide-6
SLIDE 6

Autonomous Camera

  • Energy-independent and Compute-independent

Commodity SoCs, RPI-like, chargeable battery

6

slide-7
SLIDE 7

Autonomous Camera

  • Energy-independent and Compute-independent

Commodity SoCs, RPI-like, chargeable battery Small-sized energy harvester

e.g., “10Wh today”

7

slide-8
SLIDE 8

Autonomous Camera

  • Energy-independent and Compute-independent

Commodity SoCs, RPI-like, chargeable battery Small-sized energy harvester LPWAN like LoRaWAN, At most a few Kbps

e.g., “10Wh today”

8

slide-9
SLIDE 9

Autonomous Camera

  • Energy-independent and Compute-independent

Commodity SoCs, RPI-like, chargeable battery Small-sized energy harvester LPWAN like LoRaWAN, At most a few Kbps Concise, numerical video summaries

e.g., “10Wh today”

9

slide-10
SLIDE 10

Elf for Autonomous Cameras

  • Target video query: object counting

Object Counts

10

slide-11
SLIDE 11

Elf for Autonomous Cameras

  • Target video query: object counting

Query: (car, 30 mins) Install

11

slide-12
SLIDE 12

Elf for Autonomous Cameras

  • Target video query: object counting

Query: (car, 30 mins) Install Sample & capture

12

slide-13
SLIDE 13

Elf for Autonomous Cameras

  • Target video query: object counting

Query: (car, 30 mins) Install

7:00AM-7:30AM [500 + 100] Cars 7:30AM-8:00AM [700 + 140] Cars 8:00AM-8:30AM [800 + 180] Cars 8:30AM-9:00AM [400 + 100] Cars 9:30AM-10:00AM [200 + 80] Cars

Sample & capture

13

slide-14
SLIDE 14

Elf for Autonomous Cameras

  • Target video query: object counting with confidence interval (CI)

Query: (car, 30 mins) Install

7:00AM-7:30AM [500 + 100] Cars 7:30AM-8:00AM [700 + 140] Cars 8:00AM-8:30AM [800 + 180] Cars 8:30AM-9:00AM [400 + 100] Cars 9:30AM-10:00AM [200 + 80] Cars

Sample & capture 200

  • 80

+80

14

slide-15
SLIDE 15

Elf for Autonomous Cameras

  • Target video query: object counting with confidence interval (CI)
  • The central problem: planning constrained energy for counting
  • Energy model: a budget that cannot be exceeded in a horizon (e.g., 24 hrs)
  • Trade-offs: frame sampling and NN selection
  • Target: smallest mean CI widths across all (30-min) windows in a horizon

15

slide-16
SLIDE 16

Elf Overview

Camera Operating System Reinforcement Learning Planner

Sampled frames

Aggregator with error Integration

Selected Neural Net

Object counts with CIs

16

Check our paper for details Focus of the talk

slide-17
SLIDE 17

Elf tech #1: per-window energy/CI fronts

  • What’s the best count action for a window?
  • A count action: determining (1) an NN and (2) # of frames to process

100 150 200 250 300 350

2 4

YOLOv3

17

CI width (smaller, better) Energy Consumption (kJ)

Energy Consumption = E(NN) * frame_num

slide-18
SLIDE 18

Elf tech #1: per-window energy/CI fronts

  • What’s the best count action for a window?
  • A count action: determining (1) an NN and (2) # of frames to process

100 150 200 250 300 350

2 4

YOLOv3 YOLOv2 ssd-inception-v2 ssd-mobilenet-v2

18

CI width (smaller, better) Energy Consumption (kJ)

Energy Consumption = E(NN) * frame_num

slide-19
SLIDE 19

100 150 200 250 300 350

2 4

YOLOv3 YOLOv2 ssd-inception-v2 ssd-mobilenet-v2

CI width (smaller, better) Energy Consumption (kJ)

Elf tech #1: per-window energy/CI fronts

  • What’s the best count action for a window? No silver bullet.
  • A count action: determining (1) an NN and (2) # of frames to process

When energy is low: cheaper NNs win

  • Bottlenecked by sampling error (frame quantity)

19

slide-20
SLIDE 20

Elf tech #1: per-window energy/CI fronts

  • What’s the best count action for a window? No silver bullet.
  • A count action: determining (1) an NN and (2) # of frames to process

When energy is low: cheaper NNs win

  • Bottlenecked by sampling error (frame quantity)

When energy is low: more accurate NNs win

  • Bottlenecked by NN error (frame quality)

20

100 150 200 250 300 350

2 4

YOLOv3 YOLOv2 ssd-inception-v2 ssd-mobilenet-v2

CI width (smaller, better) Energy Consumption (kJ)

slide-21
SLIDE 21

Elf tech #1: per-window energy/CI fronts

  • What’s the best count action for a window? No silver bullet.
  • A count action: determining (1) an NN and (2) # of frames to process

When energy is low: cheaper NNs win

  • Bottlenecked by sampling error (frame quantity)

When energy is low: more accurate NNs win

  • Bottlenecked by NN error (frame quality)

21

100 150 200 250 300 350

2 4

YOLOv3 YOLOv2 ssd-inception-v2 ssd-mobilenet-v2

CI width (smaller, better) Energy Consumption (kJ)

slide-22
SLIDE 22

Elf tech #1: per-window energy/CI fronts

  • What’s the best count action for a window? No silver bullet.
  • A count action: determining (1) an NN and (2) # of frames to process

Energy/CI front: the combination of all “optimal” count actions with varied energy

22

100 150 200 250 300 350

2 4

YOLOv3 YOLOv2 ssd-inception-v2 ssd-mobilenet-v2

CI width (smaller, better) Energy Consumption (kJ)

When energy is low: cheaper NNs win When energy is low: more accurate NNs win

slide-23
SLIDE 23

Elf tech #1: per-window energy/CI fronts

  • What’s the best count action for a window? No silver bullet.
  • A count action: determining (1) an NN and (2) # of frames to process

Energy/CI front: the combination of all “optimal” count actions with varied energy

  • How to construct? Error integration
  • Depends on the video characteristics

23

100 150 200 250 300 350

2 4

YOLOv3 YOLOv2 ssd-inception-v2 ssd-mobilenet-v2

CI width (smaller, better) Energy Consumption (kJ)

When energy is low: cheaper NNs win When energy is low: more accurate NNs win

slide-24
SLIDE 24

Elf tech #1: per-window energy/CI fronts

  • What’s the best count action for a window? No silver bullet.
  • A count action: determining (1) an NN and (2) # of frames to process

Energy/CI front: the combination of all “optimal” count actions with varied energy

  • How to construct? Error integration
  • Depends on the video characteristics

24

Different windows have different energy/CI fronts

100 150 200 250 300 350

2 4

YOLOv3 YOLOv2 ssd-inception-v2 ssd-mobilenet-v2

100 150 200 250 300 350

2 4

9:00AM-9:30AM 7:00PM-7:30PM

When energy is low: cheaper NNs win When energy is low: more accurate NNs win

slide-25
SLIDE 25

Elf tech #2: across-window joint planning

  • An Oracle Planner: best performance but unrealistic
  • knows all energy/CI fronts

25

slide-26
SLIDE 26

Elf tech #2: across-window joint planning

  • An Oracle Planner: best performance but unrealistic
  • knows all energy/CI fronts

26

A greedy approach: giving energy to the window with the most benefit (i.e., CI width reduction).

[50080] [45070] [60095] [5055] An oracle planner

Energy: 100% Allocate 10%

[58075]

slide-27
SLIDE 27
  • An Oracle Planner: best performance but unrealistic
  • knows all energy/CI fronts

[58075]

Elf tech #2: across-window joint planning

27

[50080] [45070] [5055] An oracle planner

Energy: 90% Allocate 10%

[51060]

A greedy approach: giving energy to the window with the most benefit (CI width reduction)

slide-28
SLIDE 28
  • An Oracle Planner: best performance but unrealistic
  • knows all energy/CI fronts

[58075]

Elf tech #2: across-window joint planning

28

[50060] [45070] [5055] An oracle planner

Energy: 80% Allocate 10%

[40055]

A greedy approach: giving energy to the window with the most benefit (CI width reduction)

slide-29
SLIDE 29
  • An Oracle Planner: best performance but unrealistic
  • knows all energy/CI fronts

[58075]

Elf tech #2: across-window joint planning

29

[50060] [40055] [5055] An oracle planner

Energy: 80%

A greedy approach: giving energy to the window with the most benefit (CI width reduction) unknown

slide-30
SLIDE 30

Elf tech #2: across-window joint planning

  • An Oracle Planner: best performance but unrealistic
  • knows all energy/CI fronts
  • planned offline
  • A learning-based planner: imitating the oracle planner
  • basis: reinforcement learning
  • rationale: daily and temporal patterns

30

Classification Agent Regression Agent Counter # Frames Reward Oracle Planner

𝜈𝑢 𝜏𝑢 …… ……

Observations

(M + N windows) Compared with

  • racle’s decision

for 𝑿𝒖 and run reward function 𝜈𝑢−4 𝜏𝑢−4 𝜈𝑢−3 𝜏𝑢−3 𝜈𝑢−2 𝜏𝑢−2 𝜈𝑢−1 𝜏𝑢−1

Recent N days Recent M windows

slide-31
SLIDE 31

Elf tech #2: across-window joint planning

  • An Oracle Planner: best performance but unrealistic
  • knows all energy/CI fronts
  • planned offline
  • A learning-based planner: imitating the oracle planner
  • basis: reinforcement learning
  • rationale: daily and temporal patterns
  • offline training -> online prediction
  • Two agents: NN selection and # of frames
  • Observations: knowledge of past windows
  • Penalty: deviation from oracle’s decision

31

Classification Agent Regression Agent Counter # Frames Reward Oracle Planner

𝜈𝑢 𝜏𝑢 …… ……

Observations

(M + N windows) Compared with

  • racle’s decision

for 𝑿𝒖 and run reward function 𝜈𝑢−4 𝜏𝑢−4 𝜈𝑢−3 𝜏𝑢−3 𝜈𝑢−2 𝜏𝑢−2 𝜈𝑢−1 𝜏𝑢−1

Recent N days Recent M windows

slide-32
SLIDE 32

Elf tech #2: across-window joint planning

  • An Oracle Planner: best performance but unrealistic
  • knows all energy/CI fronts
  • planned offline
  • A learning-based planner: imitating the oracle planner
  • basis: reinforcement learning
  • rationale: daily and temporal patterns
  • offline training -> online prediction
  • Enforce energy budget: make reservation for future windows
  • 30 frames to be statistically meaningful

32

slide-33
SLIDE 33

Elf Implementation

  • Capture & processing decoupled for higher energy efficiency
  • Processing batched at the end of each window

33

slide-34
SLIDE 34

Elf Evaluation

  • Over 1,000-hr videos
  • Public, 2-week long each stream
  • Baselines
  • 1. GoldenNN: most accurate NN
  • 2. UniNN: one fixed best NN
  • 3. Oracle: offline planned
  • Small solar panel
  • 10Wh~30Wh per day

Auburn, AL Hampton, NY Jackson, WY Taipei Taipei

34

slide-35
SLIDE 35

Elf Evaluation

  • Average: 11% error, valid and 17%-width CI
  • 95% confidence level

Ground Truth Error: 11% CI width: 17%

GoOGen11 8ni11 2UacOe 2uUs GoOGen11 8ni11 2UacOe 2uUs GoOGen11 8ni11 2UacOe 2uUs GoOGen11 8ni11 2UacOe 2uUs GoOGen11 8ni11 2UacOe 2uUs 20 40 60 80 95 100 CI CoveUage 3UobabiOity (%)

92 91 92 92 93 95 96 95 94 94 96 97 96 95 95 95 96 95 94 95

Jackson Hampton Cross TaipHi Auburn

Cis cover ground truth with 95% probability (specified)

35

slide-36
SLIDE 36

Elf Evaluation

  • Average: 11% error, valid and 17%-width CI
  • Significant improvements over baselines in CI widths
  • 66.6%, 59.8%, and 56.2% smaller over GoldenNN (up to 3.4x)
  • 41.1%, 16.6%, and 9.7% smaller over UniNN

10Wh per day 20Wh per day 30Wh per day

36

slide-37
SLIDE 37

Elf Evaluation

  • Average: 11% error, valid and 17%-width CI
  • Significant improvements over baselines in CI widths
  • Very close to Oracle
  • < 5% wider CI
  • Well imitating the oracle planner

37

Oracle planner Elf’s learning- based planner

slide-38
SLIDE 38

Elf Evaluation

  • Average: 11% error, valid and 17%-width CI
  • Significant improvements over baselines in CI widths
  • Very close to Oracle
  • What if we have AI accelerators?
  • CIs are reduced noticeably (by 22.1%–33.1%)
  • Still cannot process every frame (short of energy)

5 10 15 20 25 30 RPI 4 Intel NCS2 Jetson Nano Edge TPU Mean CI Width (%)

  • urs

higher E_cap lower E_cap

38

slide-39
SLIDE 39

Summary

  • Autonomous camera: expanding the geo-frontier of video analytics
  • Energy-independent and compute-independent
  • Elf: the first runtime for autonomous camera
  • Target query: object counting
  • Key idea: count planning per- and across-windows
  • Prototyped on heterogeneous hardware
  • Evaluated on over 1,000-hr videos
  • 11% error, 17% CI width

39