Jointly Aligning and Segmenting Multiple Web Photo Streams for the - - PowerPoint PPT Presentation

▶
jointly aligning and segmenting multiple web photo
SMART_READER_LITE
LIVE PREVIEW

Jointly Aligning and Segmenting Multiple Web Photo Streams for the - - PowerPoint PPT Presentation

Jan 02, 2023 324 likes •620 views

Jointly Aligning and Segmenting Multiple Web Photo Streams for the Inference of Collective Photo Storylines Gunhee Kim Eric P. Xing School of Computer Science, Carnegie Mellon University June 19, 2013 1 Outline Problem

slide-1
SLIDE 1

Jointly Aligning and Segmenting Multiple Web Photo Streams for the Inference of Collective Photo Storylines

Gunhee Kim Eric P. Xing

1

School of Computer Science, Carnegie Mellon University June 19, 2013

slide-2
SLIDE 2
  • Problem Statement
  • Algorithm
  • Dataset and preprocessing
  • Alignment of Multiple Photo Streams
  • Large-scale Cosegmentation
  • Experiments
  • Conclusion

Outline

2

slide-3
SLIDE 3

3

Background

Query scuba+diving from Flickr

Any meaningful structural summary?

Likely to share common storylines Taken in different spatial, temporal, and personal perspective

slide-4
SLIDE 4

4

Our Ultimate Goal

Narrative structural summary vs. independently retrieved images

beach boat diving

  • n boat

underwater coral sunset dinner

cf) ranking and retrieval by Reconstructing photo storylines from large-scale online images An example of scuba+diving storyline

slide-5
SLIDE 5

5

Objective of This Paper

As a first technical step, jointly perform two crucial tasks...

PS14 User 2 at 03/19/2005 (Phuket, Thailand) PS3 User 3 at 08/27/2008 (Cozumel, Mexico) PS2 User 1 at 10/19/2008 (Cayman Islands)

Mutually rewarding!

Alignment Cosegmentation

Match images from

  • different photo streams

Segment K common regions

  • from aligned M images
slide-6
SLIDE 6

PS14 User 2 at 03/19/2005 (Phuket, Thailand) PS3 User 3 at 08/27/2008 (Cozumel, Mexico) PS2 User 1 at 10/19/2008 (Cayman Islands)

6

Objective of This Paper

As a first technical step, jointly perform two crucial tasks... Mutually rewarding!

Alignment Cosegmentation

  • Online images are too diverse to segment together at once
  • The alignment discovers the images that share common

regions

slide-7
SLIDE 7

7

Objective of This Paper

As a first technical step, jointly perform two crucial tasks... Mutually rewarding!

Alignment Cosegmentation

  • Improve image matching by a better image similarity measure

Closing a loop between the two tasks

slide-8
SLIDE 8
  • Problem Statement
  • Algorithm
  • Dataset and preprocessing
  • Alignment of Multiple Photo Streams
  • Large-scale Cosegmentation
  • Experiments
  • Conclusion

Outline

8

slide-9
SLIDE 9

9

Flickr Dataset

Flickr dataset of 15 outdoor recreational activities

  • Experiments with more than 100K images of 1K photo streams
  • Larger than those of previous work by orders of magnitude

# of images (1,514,976) # photo streams (13,157)

Surfing Beach Horse Riding RAfting YAcht Air Ballooning ROwing Scuba Diving Formula One SNow boarding Safari Park Mountain Camping Rock Climbing Tour de France London Marathon Fly Fishing

slide-10
SLIDE 10

10

Image Descriptor and Similarity Measure

Image description Image similarity measure :

  • 1. No segmentation available
  • 2. Segmentation available

Not robust against location/pose

  • changes
  • HSV color SIFT and HOG features on regular grid
  • L1 normalized spatial pyramid histogram using 300 visual words
  • (Our assumption) Segmentation enhances the image alignment.
  • Histogram intersection on SPH

Histogram intersection on the

  • best assignment of segments
slide-11
SLIDE 11
  • Problem Statement
  • Algorithm
  • Dataset and preprocessing
  • Alignment of Multiple Photo Streams
  • Large-scale Cosegmentation
  • Experiments
  • Conclusion

Outline

11

slide-12
SLIDE 12

12

Alignment of Photo Streams

Input: A set of photo streams (PS): P = {P1,…,PL}

Photo Stream: a set of photos taken in sequence by a single user

  • in a single day

P1 P2 P3 P4 Idea: Align all photo streams at once after building K-NN graph

  • Naïve-Bayes Nearest Neighbor(NBNN) [Boiman et al. 08] for similarity metric
slide-13
SLIDE 13

13

Alignment of Photo Streams

Input: A set of photo streams (PS): P = {P1,…,PL}

Photo Stream: a set of photos taken in sequence by a single user

  • in a single day

Idea: Align all photo streams at once after building K-NN graph P1 P2 P3 P4 For simplicity, first consider pairwise alignment of two photo streams

Pairwise alignment

  • Naïve-Bayes Nearest Neighbor(NBNN) [Boiman et al. 08] for similarity metric
slide-14
SLIDE 14

14

Pairwise Alignment

Goal of alignment: find a matching btw a pair of PS Optimization: MRF-based energy minimization

  • means I in P1 has no match in P2.
  • Solved by discrete BP
  • Flexibility: Various energy terms

f : P1 → P2 ∪{∅}

E(P1,P2) = d(Ii, ˆ Ii)

Ii∈P1

∑

+ ηmin(t(Ii)−t( ˆ Ii),τ )

Ii∈P1

∑

+ ρ min(t( ˆ Ii)−t( ˆ I j),ν)

(Ii,I j )∈δ

∑

f(I) = ∅

slide-15
SLIDE 15

15

Pairwise Alignment

Objective function

Data term : The matched image pairs should be visually similar. Time term : The matched image pairs should be temporally similar. Smoothness term : The matched images to neighbors in P1 should be neighbors in P2. 9AM 10AM 6PM

E(P1,P2) = d(Ii, ˆ Ii)

Ii∈P1

∑

+ ηmin(t(Ii)−t( ˆ Ii),τ )

Ii∈P1

∑

+ ρ min(t( ˆ Ii)−t( ˆ I j),ν)

(Ii,I j )∈δ

∑

slide-16
SLIDE 16

P1 P2 P3 P4

16

Alignment of Multiple Photo Streams

Objective : MRF-based energy minimization

Pairwise alignment Pairwise alignment Pairwise alignment Pairwise alignment

: All pairs of NN photo streams

Message-passing based optimization

  • until convergence or for fixed iterations

EAll = E(Pi, P j)

∑

(Pi,P j) ∈ Ξ

slide-17
SLIDE 17

P1 P2 P3 P4

17

Alignment of Multiple Photo Streams

Objective : MRF-based energy minimization

: All pairs of NN photo streams

Message-passing based optimization

  • until convergence or for fixed iterations

EAll = E(Pi, P j)

∑

(Pi,P j) ∈ Ξ

slide-18
SLIDE 18
  • Problem Statement
  • Algorithm
  • Dataset and preprocessing
  • Alignment of Multiple Photo Streams
  • Large-scale Cosegmentation
  • Experiments
  • Conclusion

Outline

18

slide-19
SLIDE 19

19

Build an Image Graph

Idea: Connect the images that are similar enough to be cosegmented Image Graph G = (I, E)

  • E = EB U Ew

EB : Edges between different photo streams (results of alignment) EW : Edges within a photo stream

  • I : The set of images. E : The set of edges.

EB

For each image I, links I with the K-NN of I (EW). consider the images such that

Ew

t(I)−t(Ii) ≤δ

slide-20
SLIDE 20

20

Scalable Cosegmentation

Iteratively run the MFC algorithm [Kim and Xing, 2012] on the image graph Review of MFC algorithm Foreground Modeling Region Assignment

Learn appearance models of K FGs

  • and BG

Allocate the regions

  • f image into one of
  • K FGs or BG

Iterate

  • Ex. Gaussian mixture on
  • RGB, linear SVM on

SPH Any region classifiers

  • or their combination

Very efficiently solve using the idea of

  • combinatorial auction

Cosegmentation: Jointly segment M images into K+1 regions

  • (K foregrounds (FG) + background (BG))
slide-21
SLIDE 21

21

Scalable Cosegmentation on Image Graph

FG 2 (road) FG 3 (BG) FG 1 (car) Message-passing based optimization

  • Learn FG Models from neighbors of Ii.
  • Run region assignment on Ii.

Iteratively solve…

slide-22
SLIDE 22

22

Scalable Cosegmentation on Image Graph

FG 1 (car) FG 3 (BG) FG 2 (road) Message-passing based optimization

  • Learn FG Models from neighbors of Ii.
  • Run region assignment on Ii.

Initialization Iteratively solve…

  • Supervised: start from seed

labels Unsupervised: use the algorithm

  • of CoSand [Kim et al. 2011].
slide-23
SLIDE 23
  • Problem Statement
  • Algorithm
  • Dataset and preprocessing
  • Alignment of Multiple Photo Streams
  • Large-scale Cosegmentation
  • Experiments
  • Conclusion

Outline

23

slide-24
SLIDE 24

24

Evaluation for Alignment Evaluation for cosegmentation

Evaluation – Two Experiments

Task: Temporal localization (inspired by geo-location estimation) Task: Foreground detection

  • We manually annotate 100 images per class

Very hard to obtain groundtruth!

  • Correspondences btw two sets of thousands of images?
  • Accuracy is measured by intersection-over-union

[Hays and Efros. 2008]

Where is it likely to be taken? When are they likely to be taken?

P1

Timeline

Acc = GT

i ∩ R i

GT

i ∪ R i

slide-25
SLIDE 25

25

Evaluation of Alignment

Procedures of temporal localization Training (80%)

Test (20%)

  • 1. Given a set of photo streams,

randomly split training and test sets

  • 2. Run alignment
  • 3. Estimate timestamps of all images

in test photo streams

  • 4. Temporal localization is correct if

Baselines

Popular multiple sequence alignment – Image similarity only (the simplest) Justify closing a loop

  • BPS: Our Alignment + Cosegmentation
  • BP: Our alignment only
  • KNN: K-nearest neighbors
  • HMM: Hidden Markov Models
  • DTW: Dynamic Time Windows

Better temporal localization ≠ Better Alignment

tgt −t

e st ≤ε

slide-26
SLIDE 26

26

BPS: Our Alignment + Cosegmentation BP: Our alignment only KNN: K-nearest neighbors HMM: Hidden Markov Models DTW: Dynamic Time Windows

Evaluation of Alignment

  • Temporal localization is correct

if

= 60 min.

tgt −t

e st ≤ε

ε ε

slide-27
SLIDE 27

27

Evaluation of Cosegmentation

Task: Foreground detection Examples

BP+MFC: (Proposed) Alignment + Cosegmentation MFC: Our cosegmentation without alignment COS : Submodular optimization [Kim et al. ICCV11] LDA: LDA-based localization [Russell et al. CVPR06]

slide-28
SLIDE 28
  • Problem Statement
  • Algorithm
  • Dataset and preprocessing
  • Alignment of Multiple Photo Streams
  • Large-scale Cosegmentation
  • Experiments
  • Conclusion

Outline

28

slide-29
SLIDE 29

29

Conclusion

Ultimate goal: building photo storylines from large-scale online images safari+park horse+riding