for Mobile Augmented Reality David Chen 1 , Mina Makar 1,2 , Andre - - PowerPoint PPT Presentation

▶
for mobile augmented reality
SMART_READER_LITE
LIVE PREVIEW

for Mobile Augmented Reality David Chen 1 , Mina Makar 1,2 , Andre - - PowerPoint PPT Presentation

Feb 01, 2024 562 likes •806 views

Interframe Coding of Global Image Signatures for Mobile Augmented Reality David Chen 1 , Mina Makar 1,2 , Andre Araujo 1 , Bernd Girod 1 1 Department of Electrical Engineering, Stanford University 2 Qualcomm Inc. 1 Chen et al. , Interframe Coding

slide-1
SLIDE 1

1

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Interframe Coding of Global Image Signatures for Mobile Augmented Reality

David Chen1, Mina Makar1,2, Andre Araujo1, Bernd Girod1

1Department of Electrical Engineering, Stanford University 2Qualcomm Inc.

slide-2
SLIDE 2

2

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Samsung Galaxy S3 Smartphone

slide-3
SLIDE 3

3

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

On-Device Image Matching System

Extract Local Features Generate Global Signature Match with Database Global Signatures Perform Geometric Verification Local Database

… … … …

slide-4
SLIDE 4

4

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Hybrid Image Matching System

Extract Local Features Generate Global Signature Match with Database Global Signatures Perform Geometric Verification Local Database

Wireless Network

Remote Database Match with Database Global Signatures Send compact interframe coded stream of global signatures in uplink Send labels, local features, and global signatures for top-ranked database candidates in downlink

0.51 0.62 0.50 0.49

…

slide-5
SLIDE 5

5

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Outline

  • Related work on feature compression
  • Interframe coding of global signatures

– Selective codeword propagation – Selective frame propagation – Selective frame propagation + local search – Global signature embedding

  • Coding and retrieval results
slide-6
SLIDE 6

6

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Related Work on Feature Compression

Intraframe Coding Interframe Coding Global Image Signatures Local Image Features

Random projections [Yeo, 2008] Transform coding [Chandrasekhar, 2009] Location histogram coding [Tsai, 2009] CHoG [Chandrasekhar, 2009] Bag of hash bits [He, 2011] Trace transform [Brasnett, 2007] Tree histogram coding [Chen, 2009] Residual vectors [Perronnin, 2010] [Jegou, 2010] [Chen, 2011] Temporally coherent keypoint detector [Makar, 2012] Descriptor and location predictive coding [Makar, 2012] [Baroffio, 2014]

Our Current Work

slide-7
SLIDE 7

7

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Forward Propagation (FP) Frame Detection (D) Frame

Temporally Coherent Keypoint Detection

Makar et al., 2012

time

t

slide-8
SLIDE 8

8

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Generating Feature Residuals

k = 3

slide-9
SLIDE 9

9

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Generating Feature Residuals

k = 3

slide-10
SLIDE 10

10

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Residual Enhanced Visual Vector (REVV)

Chen et al., 2011

Extract Local Features Quantize Feature Descriptors k codewords Normalize Feature Residuals Query Image Regularize with Power Law Perform Cell-Specific LDA Binarize Components from Sign Compute Weighted Correlations Database Signatures Ranked List

0.74 0.73 0.72 0.70 0.63 0.62

…

Reduce Dimensions by PCA Normalize Correlation Scores

  • 1

+1

5 10 15 20 25 30 0.05 0.1 0.15 0.2 Hamming Distance Probability Matching Images Non-matching Images 5 10 15 20 25 30 0.2 0.4 0.6 0.8 1 Hamming Distance Weights
slide-11
SLIDE 11

11

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Continuous stream of REVV signatures

Interframe Coding of REVV

R1 S1 Extract REVV Extract REVV Extract REVV Extract REVV R2 Predictively Encode S2 Rn+1 Sn+1 Mobile Device Database REVV Signatures Compute Weighted Correlations Wireless Network Server Decode REVV Signatures Previous REVV Signatures Recognition results + Features for top candidates D-Frame D-Frame FP-Frame FP-Frame

Keypoints Keypoints Keypoints

S1

…

Rn Predictively Encode Sn S2 Sn-1

… …

slide-12
SLIDE 12

12

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Interframe Coding of REVV

R1 S1 Extract REVV Extract REVV Extract REVV Extract REVV R2 Predictively Encode S2 Rn+1 Sn+1 D-Frame D-Frame FP-Frame FP-Frame

Keypoints Keypoints Keypoints

S1

…

Rn Predictively Encode Sn S2 Sn-1

… …

Ut,1 = 1 Ut,2 = 1 Ut,4 = 1 Ut,6 = 1 Ut,k = 1 … Rt,1 Rt,2 Rt,4 Rt,6 Rt,k Ut,3 = 0 Ut,5 = 0 Vt,1 = 1 Vt,2 = 1 Vt,4 = 1 Vt,6 = 1 Vt,k = 1 … St,1 St,2 St,4 St,6 St,k Vt,3 = 0 Vt,5 = 0

Extracted REVV Signature Transmitted REVV Signature

slide-13
SLIDE 13

13

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Selective Codeword Propagation (SCP)

Ut,1 = 1 Ut,2 = 1 Ut,4 = 1 Ut,8 = 1 Rt,1 Rt,2 Ut,3 = 0 Ut,5 = 1 Ut,6 = 1 Ut,7 = 0 … Rt,4 Rt,5 Rt,6 Rt,8 Frame t D-Frame Extracted Wireless Network Mobile Device Server Vt,1 = 1 Vt,2 = 1 Vt,4 = 1 Vt,8 = 1 St,1 St,2 Vt,3 = 0 Vt,5 = 1 Vt,6 = 1 Vt,7 = 0 St,4 St,5 St,6 St,8 Frame t D-Frame Received … Ut+1,1 = 1 Ut+1,2 = 1 Ut+1,3 = 1 Ut+1,4 = 1 Ut+1,6 = 1 Ut+1,8 = 1 Ut+1,7 = 0 Ut+1,5 = 0 Frame t+1 FP-Frame Extracted Rt+1,1 Rt+1,2 Rt+1,4 Rt+1,3 Rt+1,6 Rt+1,8 Frame t+1 FP-Frame Sent Vt+1,1 = 1 Vt+1,2 = 1 Vt+1,4 = 1 Vt+1,8 = 1 St+1,1 St+1,2 Vt+1,3 = 0 Vt+1,5 = 0 Vt+1,6 = 1 Vt+1,7 = 0 … St+1,4 St+1,6 St+1,8 AND AND AND AND AND AND AND AND Frame t+1 FP-Frame Received Vt+1,1 = 1 Vt+1,2 = 1 Vt+1,4 = 1 Vt+1,8 = 1 St+1,1 St+1,2 Vt+1,3 = 0 Vt+1,5 = 0 Vt+1,6 = 1 Vt+1,7 = 0 … St+1,4 St+1,6 St+1,8 …

slide-14
SLIDE 14

14

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Selective Frame Propagation (SFP)

Ut,1 = 1 Ut,2 = 1 Ut,4 = 1 Ut,8 = 1 Rt,1 Rt,2 Ut,3 = 0 Ut,5 = 1 Ut,6 = 1 Ut,7 = 0 … Rt,4 Rt,5 Rt,6 Rt,8 Frame t D-Frame Extracted Mobile Device Ut+1,1 = 1 Ut+1,2 = 1 Ut+1,3 = 1 Ut+1,4 = 1 Ut+1,6 = 1 Ut+1,8 = 1 Ut+1,7 = 0 Ut+1,5 = 0 Frame t+1 FP-Frame Extracted Rt+1,1 Rt+1,2 Rt+1,4 Rt+1,3 Rt+1,6 Rt+1,8

 

 

, 1, 1, 1 1

, 1 AND ,

k k k t j t j t j j j

r t t U U U

   

 

SCP Encoding Vt+1,1 = 1 Vt+1,2 = 1 Vt+1,4 = 1 Vt+1,8 = 1 St+1,1 St+1,2 Vt+1,3 = 0 Vt+1,5 = 0 Vt+1,6 = 1 Vt+1,7 = 0 … St+1,4 St+1,6 St+1,8

Interframe Codeword Similarity rk > tr ? Yes No

SFP Encoding … Vt+1,1 = 1 Vt+1,2 = 1 Vt+1,4 = 1 Vt+1,8 = 1 St+1,1 St+1,2 Vt+1,3 = 0 Vt+1,5 = 0 Vt+1,6 = 1 Vt+1,7 = 0 St+1,4 St+1,6 St+1,8 St+1,5 …

slide-15
SLIDE 15

15

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

SFP + Local Search (SFP + LS)

Local Database Remote Database Wireless Network Mobile Device Server

Ngeo ³ tgeo

Terminate query locally

  • n mobile device

Send REVV stream to server by SFP coding

Ngeo < tgeo

Send local features and REVV signatures for top ranked database candidates to mobile device Number of inliers in geometric verification

slide-16
SLIDE 16

16

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Embedded Global Signatures

Codeword 1 Codeword 2 Codeword 3 Codeword 4 Codeword 5 Codeword 6 Codeword 7 Codeword 8

Level 1  Highest Bitrate Level 3  Lowest Bitrate Level 2  Medium Bitrate

slide-17
SLIDE 17

17

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Outline

  • Related work on feature compression
  • Interframe coding of global signatures

– Selective codeword propagation – Selective frame propagation – Selective frame propagation + local search – Global signature embedding

  • Coding and retrieval results
slide-18
SLIDE 18

18

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Analysis of Retrieval Performance

slide-19
SLIDE 19

19

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Mobile Augmented Reality

Stanford Streaming MAR Dataset

32 VGA-resolution query videos recorded with a camera phone Database of 23 labeled objects + 1M distractor images

[Makar et al., 2013]

slide-20
SLIDE 20

20

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Experimental Setup

  • Interframe coding parameters

– ND-Frames = 1 and NFP-Frames = 29 for frame rate of 30 fps – Interframe codeword similarity threshold: tr = 0.9 – RANSAC threshold: tgeo = 25 feature matches

  • REVV signature parameters

– 250 SIFT features extracted for every D-Frame – Dimensionality reduction to dLDA = 32 – Codebook of k = 190 codewords

  • Retrieval accuracy vs. uplink bitrate comparison
slide-21
SLIDE 21

21

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Retrieval Accuracy vs. Uplink Bitrate

Embedding: Level 1 Embedding: Level 2 Embedding: Level 3 14x 24x 88x < 2 kbps

slide-22
SLIDE 22

22

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Conclusions

  • Developed efficient methods for interframe coding of a

continuous stream of global REVV signatures

– Adapts bitrate in response to current viewfinder contents – Combines advantages of local and remote database search – Performs local database update at same time as query expansion

  • Analyzed how retrieval accuracy varies with bitrate

– Models distributions of REVV correlation scores – Quantifies bitrate savings for all coding methods

  • Achieved substantial bitrate savings

– Reduces uplink bitrate by almost 2 orders of magnitude – Requires only a small downlink bitrate

slide-23
SLIDE 23

23

Chen et al., Interframe Coding of Global Image Signatures for Mobile Augmented Reality

Thank You

dmchen@stanford.edu