Data Ma Mana nagement for r Vide deo Ana nalyti tics Video - - PDF document

▶
data ma mana nagement for r vide deo ana nalyti tics
SMART_READER_LITE
LIVE PREVIEW

Data Ma Mana nagement for r Vide deo Ana nalyti tics Video - - PDF document

Jun 16, 2023 525 likes •649 views

7/26/2019 Data Ma Mana nagement for r Vide deo Ana nalyti tics Video data is everywhere. Brandon Haynes, Maureen Daum, Amrita Mazumdar, Magdalena Balazinsk ska, Luis Ceze, & Alvin Cheung 1 2 1 2 Width in pixels Height in pixels

slide-1
SLIDE 1

7/26/2019 1

1

Data Ma Mana nagement for r Vide deo Ana nalyti tics

Brandon Haynes, Maureen Daum, Amrita Mazumdar, Magdalena Balazinsk ska, Luis Ceze, & Alvin Cheung

2

Video data is everywhere.

3

Existing systems treat video data like it’s the 20th century

4

Height in pixels Width in pixels

5

Many recent video applications require jointly querying multiple cameras, reasoning about position and orientation, or querying complex metadata.

6

1 2 3 4 5 6

slide-2
SLIDE 2

7/26/2019 2

7

SELECT Metadata.Location FROM Cameras WHERE ‘James Bridle’ IN Metadata.Name

8

A Video Data Management Benchmark

Visual Road

A Database System for Virtual & Augmented Reality Video Applications Optimized storage and retrieval of video data

Video File System Metastore

Efficient querying of rich video content (Maureen Daum)

Data Ma Mana nagement for r Vide deo Ana nalyti tics

Modern

LightDB

A Database System for Virtual & AugmentedReality Video Applications

How would you write this application?

12

20th century Imperative Code

LightDB Query

source = Scan(“kittens”) detection = source.Map(𝑒𝑓𝑢𝑓𝑑𝑢) result = Union(source, detection) result.Save(“output.mp4”)

7 8 9 10 11 12

slide-3
SLIDE 3

7/26/2019 3

13

source = Scan(“kittens”) detection = source.Map(𝑒𝑓𝑢𝑓𝑑𝑢) result = Union(source, detection) result.Save(“output.mp4”)

LightDB Query:

14

Logical Plan

Scan Map (Grayscale) Encode

HEVC

Scan Map Union Save

source = Scan(“kittens”) detection = source.Map(𝑒𝑓𝑢𝑓𝑑𝑢) result = Union(source, detection) result.Save(“output.mp4”)

15

GPU Overlay GPU Overlay ScanSingle File SaveSingle File GPUMap GPUMap GPUEncode GPUEncode GPUTee GPUTee GPU ToCPU GPU ToCPU GPUQueued Decode Homomorphic Union

Physical Plan

15 30 45

Timelapse Venice Coaster Cats

FPS LightDB Ffmpeg Scanner OpenCV

(Light Field)

Object Overlay AR Application

Brandon Haynes bhaynes@cs.washington.edu lightdb.uwdb.io

Key Features:

  • Data management system for VR & AR video applications
  • Unified data model for panoramic (360°) and light field video
  • Declarative queries with automatic optimization

Key Contributions:

  • ∼ Τ

1 10 lines of code

  • Up to 4× performance for real-world workloads
  • Reduced client bandwidth & power requirements

Scan(“LEGOS”) .Map(GRAYSCALE) .Store(“GRAYLEGOS”)

A Database System for Virtual & Augmented Reality Video Applications A Video Data Management Benchmark

Visual Road

Optimized storage and retrieval of video data

Video File System Metastore

Efficient querying of rich video content (Maureen Daum)

Data Ma Mana nagement for r Vide deo Ana nalyti tics

Modern

13 14 15 16 17 18

slide-4
SLIDE 4

7/26/2019 4

19

The perfor

  • rmance of video systems is evaluated

like it’s still the 20th century

20

Video System # Distinct Videos LightDB (2018) 4 Chameleon (2018) 5 BlazeIt (2018) 6 NoScope (2017) 7 Focus (2018) 14 DeepLens (2019) ∼16 Scanner (2018) >100 (only 14 joined)

# Distinct videos performance teste stedby system

21

UA-DETRAC Dataset (2015) 10 hours, 24 Locations

Video by Wen et al.

Manually annotated with 1,210,000 hand- drawn boxes!

22

Type Benchmarks Test Data Video (Visual Road) Ad Hoc OLTP (TPC-H) Synthetic OLAP (SSB, DWEB) Synthetic Streaming (Linear Road, DTDW) Synthetic NoSQL (YCSB) Synthetic Graph (LDBC) Synthetic Privacy (SDV) Synthetic Synthetic

23

Benchmark Queries & Verifier Video Dataset Generator Hyperparameters: scale, resolution, duration, seed

Configure Generate

Synthetic Dataset Commercial Gaming Engine Autonomous Driving Simulator Video Data Management System

Execute

Q1 Q2 Q2b Q2c Q2d Q3 Q4 Q5 Q6a Q6b Q7 Q8 Q9 Q10

Verify

Pregenerated Datasets

24

19 20 21 22 23 24

slide-5
SLIDE 5

7/26/2019 5

25 26

Automatic ground truth generation

27

Traffic Camera 𝑊

𝑘:

Result for query instance 𝑗:

(4 × 𝑇𝑑𝑏𝑚𝑓 query instances)

⋮

Query 7: Object Detection

28

1 2

Q2(a)

8 16

Q2(d)

6 12

Q2(b)

8 16

Q3

1 2

Q1

36 72

Q2(c) Scale Factor

1 2 2 4 6 8

Q5

2 4 2 4 6 8

Q6(a)

4 8 2 4 6 8

Q4

`

29

visualroad.uwdb.io Key Features:

  • Video data management benchmark
  • Synthetic dataset generation
  • Unlimited scale, resolution, duration, and overlap
  • Extensible suite of computer vision, VR, and microbenchmarks
  • Applicable to both general and specialized video DBMSs

A Database System for Virtual & Augmented Reality Video Applications A Video Data Management Benchmark

Visual Road

Optimized storage and retrieval of video data

Video File System Metastore

Efficient querying of rich video content (Maureen Daum)

Data Ma Mana nagement for r Vide deo Ana nalyti tics

Modern

25 26 27 28 29 30

slide-6
SLIDE 6

7/26/2019 6

31

visualroad.uwdb.io Key Features:

  • Video data management benchmark
  • Synthetic dataset generation
  • Unlimited scale, resolution, duration
  • Extensible suite of computer vision, VR, and

microbenchmark queries

Key Features:

  • DBMS for VR & AR video applications
  • Unified data model
  • Declarative queries
  • Automatic optimization

lightdb.uwdb.io

A Database System for Virtual & Augmented Reality Video Applications A Video Data Management Benchmark

Visual Road

Optimized storage and retrieval of video data

Video File System Metastore

Efficient querying of rich video content (Maureen Daum)

Data Ma Mana nagement for r Vide deo Ana nalyti tics

Modern

Towards ds Efficient t Querying of Rich h Vide deo Con

  • ntent

Maureen Daum

Motivation

  • We want to enable rich, content-based queries over video data
  • Existing systems optimize running object detection over videos
  • As a result, they focus on simple queries only
  • We want to use this metadata to enable more complex queries

Metadata

  • Object labels
  • Weather conditions
  • Descriptions
  • License plate numbers
  • Aggregates

35

SELECT pixels FROM video WHERE dog

36

31 32 33 34 35 36

slide-7
SLIDE 7

7/26/2019 7

Overlay the dog pixels from video 1 onto the background in video 2

37

Select sequences of frames that contain increasing numbers of cats

38

Metadata Model

t y x

Label Volume Dog

𝑢0,𝑢1 , 𝑦0, 𝑦1 , [𝑧0,𝑧1]

39

Executing Queries

40

  • Videos are stored in a compressed format
  • Encoding and decoding are expensive operators

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Select frames with bicycles Select frames with buses Select frames with people Draw boxes over people

Perc rcent nt of Que uery ry Time Spe pent in n Enc ncod

  • de or
  • r Decode

de

Encode and decode

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA

41

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA

42

37 38 39 40 41 42

slide-8
SLIDE 8

7/26/2019 8

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA

43

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA

44

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA

45

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA

46

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA

47

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA

48

43 44 45 46 47 48

slide-9
SLIDE 9

7/26/2019 9

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA

49

SELECT frames FROM video WHERE dog

This Photo by Unknown Author is licensed under CC BY-SA

50

This Photo by Unknown Author is licensed under CC BY-SA

51

SELECT frames FROM video WHERE dog

Random access point Encoded as a delta

This Photo by Unknown Author is licensed under CC BY-SA

52

SELECT frames FROM video WHERE dog

Random access point Encoded as a delta

This Photo by Unknown Author is licensed under CC BY-SA

53

SELECT frames FROM video WHERE dog

Random access point Encoded as a delta

This Photo by Unknown Author is licensed under CC BY-SA

54

SELECT frames FROM video WHERE dog

Random access point Encoded as a delta

49 50 51 52 53 54

slide-10
SLIDE 10

7/26/2019 10

This Photo by Unknown Author is licensed under CC BY-SA

55

SELECT frames FROM video WHERE dog

Random access point Encoded as a delta

This Photo by Unknown Author is licensed under CC BY-SA

56

SELECT frames FROM video WHERE dog

Random access point Encoded as a delta

57

SELECT pixels FROM video WHERE dog

58

SELECT pixels FROM video WHERE dog

59

SELECT pixels FROM video WHERE dog

60

SELECT pixels FROM video WHERE dog

55 56 57 58 59 60

slide-11
SLIDE 11

7/26/2019 11

61

SELECT pixels FROM video WHERE dog

62

SELECT pixels FROM video WHERE dog

Add Metadata

Putting Things Together

Similar to DB cracking, incrementally partition the video and add indices Original Video Execute Query Update Layout

63

Ongoing Work

  • Measure the effectiveness of selection optimization techniques
  • Investigate optimization techniques for more compute-heavy queries
  • Determine how to effectively layout videos with a lot of metadata
  • Possibly store multiple versions of a video with different layouts

64

Conclusion

  • Deep learning opens the door to rich queries over video data
  • Videos are large and slow to process
  • Database techniques can accelerate such queries
  • Partitioning
  • Indexing
  • Incremental physical tuning
  • Indexing must be balanced with maintaining reasonable storage sizes

61 62 63 64 65