Novelty Detection from an Ego- Centric Perspective Omid Aghazadeh, - - PowerPoint PPT Presentation

▶
novelty detection from an ego centric perspective
SMART_READER_LITE
LIVE PREVIEW

Novelty Detection from an Ego- Centric Perspective Omid Aghazadeh, - - PowerPoint PPT Presentation

Sep 15, 2022 195 likes •496 views

Novelty Detection from an Ego- Centric Perspective Omid Aghazadeh, Josephine Sullivan, and Stefan Carlsson Presented by Randall Smith Friday, November 16, 12 1 Outline Introduction Sequence Alignment Appearance Based Cues

slide-1
SLIDE 1

Novelty Detection from an Ego- Centric Perspective

Omid Aghazadeh, Josephine Sullivan, and Stefan Carlsson Presented by Randall Smith

1 Friday, November 16, 12

slide-2
SLIDE 2

Outline

  • Introduction
  • Sequence Alignment
  • Appearance Based Cues
  • Geometric Similarity
  • Example
  • Dynamic Time Warping
  • Algorithm
  • Evaluation of Similarity Matching
  • Results
  • Conclusion

2 Friday, November 16, 12

slide-3
SLIDE 3

Introduction

  • Problem: Select relevant visual input from worn, mobile camera.
  • Motivation:
  • Routine Recognition [Blanke & Schiele 2009]
  • Life Logging [Doherty & Smeaton 2010]

[Schiele et. al. 2007]

  • Memory assistance [Hodges et. al. 2006]

Image: CVPR 2011, Aghazadeh et. al.,link

3 Friday, November 16, 12

slide-4
SLIDE 4

Introduction : Memory Selection

  • We must decide what visual inputs to remember.
  • How should this be done?
  • Novelty detection.
  • What is novelty detection?

4 Friday, November 16, 12

slide-5
SLIDE 5

Introduction : Novelty Detection

  • Novelty = All Inputs - Known Inputs
  • Novelty detection: identification of inputs that differ from previously

seen inputs.

  • Novelty detection can help decide on what is worth remembering.

All Inputs Known Inputs

5 Friday, November 16, 12

slide-6
SLIDE 6

Introduction : Setup

  • Heuristic: detect novelty as deviation from background.
  • Context: collect video sequences from from daily commute to work.
  • Equipment: 4cm camera + memory stick.

Image: CVPR 2011, Aghazadeh et. al.,link

6 Friday, November 16, 12

slide-7
SLIDE 7

Introduction : Dataset

Image: CVPR 2011, Aghazadeh et. al.,link

7 Friday, November 16, 12

slide-8
SLIDE 8

Sequence Alignment

  • Novelty is defined as a failure to register a sequence with a set of stored

reference sequences (25 Hz videos sampled at 1 Hz.)

  • Accomplished by sequence alignment, via Dynamic Time Warping (DTW).

Image: CVPR 2011, Aghazadeh et. al.,link

8 Friday, November 16, 12

slide-9
SLIDE 9

Sequence Alignment : Discussion

  • Could we define or detect novelty in some other way?

9 Friday, November 16, 12

slide-10
SLIDE 10

Sequence Alignment : Dynamic Time Warping

M Time Series B Time Series A

10 Friday, November 16, 12

slide-11
SLIDE 11

Sequence Alignment : Similarity

  • In order to use DTW, need to define some cost function
  • This can by defining a measure of similarity between each pair of frames.
  • Can use appearance based cues (SIFT, VLAD) to do this.

Image: link

11 Friday, November 16, 12

slide-12
SLIDE 12

Appearance Based Cues

  • Can compute a fixed length vector each frame and use a kernel in order to

compare similarity.

  • Use SIFT or VLAD/SIFT to compute Bag of Features (BoF).
  • VLAD: Vector of Locally Aggregated Descriptors:
  • (1) get k-means code book, and
  • (2) for each codeword C
  • take the L2-normalized sum of all the vectors assigned to it.

Image: link

12 Friday, November 16, 12

slide-13
SLIDE 13

Geometric Similarity

  • Appearance based cues alone are not accurate enough.
  • Need to match local structures in a geometrically consistent way.
  • Need a transformation that will do this: fundamental matrix.
  • The measure of similarity will be the percentage of inliers in an initial set of

putative matches, w.r.t to estimated fundamental matrix.

  • Match against homography mapping to assess correctness of hypothetical

fundamental matrix

13 Friday, November 16, 12

slide-14
SLIDE 14

Geometric Similarity : Discussion

  • Could we supplement or substitute some other measure of similarity?
  • How could different similarity measures affect novelty?

14 Friday, November 16, 12

slide-15
SLIDE 15

Example

15 Friday, November 16, 12

slide-16
SLIDE 16

Example

Image: CVPR 2011, Aghazadeh et. al.,link

16 Friday, November 16, 12

slide-17
SLIDE 17

Dynamic Time Warping

  • Define a path:
  • s.t. (1) , and

(2)

  • Define a cost function
  • Let
  • Want .
  • Solved via dynamic programming.

p∗ = argminpCp c(i, j) ≥ 0 Cp =

Kp

X

k=1

c(ik, jk) (iK, jK) = (M, N) (i1, j1) = (1, 1) p = {(i1, j1), . . . , (iK, jK)} pk+1 − pk ∈ {(0, 1), (1, 0), (1, 1)}

M N Time Series B Time Series A

17 Friday, November 16, 12

slide-18
SLIDE 18

Algorithm

  • compute features , and nearest neighbor distance ratio
  • keep best matches based on this ordering
  • compute loose homography and inliers
  • compute 5 point fundamental matrix from and inliers
  • compute similarity

HL PH E PH N P PHE fs = min(1, α max(0, |PHR| |P| − β))

Image: CVPR 2011, Aghazadeh et. al.,link

F1 F2

18 Friday, November 16, 12

slide-19
SLIDE 19

Algorithm : Cost Matrix

  • Need to compute similarity matrix for sequences and .
  • Convert to cost matrix via zero-mean Gaussian with standard deviation .
  • Why? Noise?
  • Use DTW to find optimal alignment!
  • Problem: this is expensive.

s1 s2 σc

19 Friday, November 16, 12

slide-20
SLIDE 20

Algorithm : Optimization

  • Optimization: for each frame in find the k nearest neighbors in .
  • Evaluate only the k nearest neighbors instead.

s1 s2

Image: CVPR 2011, Aghazadeh et. al.,link

20 Friday, November 16, 12

slide-21
SLIDE 21

Algorithm : Match Cost

  • Let correspond to frame indices in and to frame indices in .
  • Let be the minimum cost path from DTW.
  • The match cost for a frame in to is
  • where is the value of the cost matrix at .

δs1,s2 i j λ(i, δs1,s2) = ( Cik,jk if ∃(ik, jk) ∈ δs1,s2 s.t. i = ik 1

  • therwise

s1 s2 λ(i, δs1,s2) i s1 s2 Cik,jk (ik, jk)

21 Friday, November 16, 12

slide-22
SLIDE 22

Algorithm : Novelty Detection

  • Compute the minimum match cost for each frame in the query sequence:
  • where contains all reference sequences.
  • Threshold the minimum match cost to find novelties.
  • Smoothing: Gaussian mask applied to prior to matching with and using

threshold . E(s(i)

t ) = min sr∈S λ(i, δsq,sr)

S σN ΘN = e

−

1 23σ2 c

22 Friday, November 16, 12

slide-23
SLIDE 23

Algorithm : Discussion

  • How else could we implement memory selection or novelty detection?
  • How does this scale with the number of stored sequences?

23 Friday, November 16, 12

slide-24
SLIDE 24

Evaluation of Similarity Matching

  • minimum intersection kernel for BoF and degree one polynomial kernel for

VLAD/SIFT

  • VLAD + BoF + Dense (gray + color) -> 88% = best

Image: CVPR 2011, Aghazadeh et. al.,link

24 Friday, November 16, 12

slide-25
SLIDE 25

Results : Detecting Novelty

Image: CVPR 2011, Aghazadeh et. al.,link

25 Friday, November 16, 12

slide-26
SLIDE 26

Results : Precision Recall Curves and Matches

Image: CVPR 2011, Aghazadeh et. al.,link

26 Friday, November 16, 12

slide-27
SLIDE 27

Conclusion

  • The scalability of this algorithm seems to be an issue.
  • It would be interesting to explore alternative measures of similarity or novelty.
  • Could this be converted to purely use clustering and only store clips for

reference (by the user).

  • The dataset is quite small, which is understandable given their technique, but

perhaps an improved technique could make this work better?

27 Friday, November 16, 12

slide-28
SLIDE 28

References

  • H. Jegou, M. Douze, C. Schmid, and P

. Perez. Aggregating local descriptors into a compact image representation. In CVPR, 2010.

  • M. Muller. Information retrieval for music and motion. Springer-Verlag New

York Inc, 2007.

  • Novelty Detection from an Egocentric Perspective. O. Aghazadeh, J. Sullivan,

and S. Carlsson. CVPR 2011

28 Friday, November 16, 12