Columbia HLF: TRECVID2006 TRECVID TRECVID TRECVID 2005 2005 - - PowerPoint PPT Presentation

▶
columbia hlf trecvid2006
SMART_READER_LITE
LIVE PREVIEW

Columbia HLF: TRECVID2006 TRECVID TRECVID TRECVID 2005 2005 - - PowerPoint PPT Presentation

Nov 25, 2022 445 likes •875 views

DVMM lab d igital video | multimedia laboratory Coping with Video Domain Change Analysis of Cross-Domain Learning Methods for High-Level Visual Concept Detection Eric Zavesky, Wei Jiang, Akira Yanagawa, Shih-Fu Chang TRECVID HLF 2007 Columbia

slide-1
SLIDE 1

DVMMlab

digital video | multimedia laboratory

Coping with Video Domain Change

Eric Zavesky, Wei Jiang, Akira Yanagawa, Shih-Fu Chang TRECVID HLF 2007

Analysis of Cross-Domain Learning Methods for High-Level Visual Concept Detection

slide-2
SLIDE 2

Coping with Video Domain Change; Zavesky 2007

2

Columbia HLF: TRECVID2006

TRECVID 2005 (development) Columbia 374 (baseline models) Concept fusion (explore concept relations) BCRF (20 improved models) TRECVID 2005 (development) Columbia 374 (baseline models) TRECVID 2005 (development) Columbia 374 (baseline models) Concept fusion (explore concept relations) BCRF (20 improved models) TRECVID 2006 (test)
slide-3
SLIDE 3

Coping with Video Domain Change; Zavesky 2007

3

Columbia HLF: TRECVID2007

Concept fusion (explore concept relations) TRECVID 2005 (development) Columbia 374 (baseline models) TRECVID 2007 (test) BCRF (20 improved models)

1

Concept fusion (explore concept relations) TRECVID 2005 (development) Columbia 374 (baseline models) TRECVID 2007 (development) New baseline TRECVID 2007 (test) BCRF (20 improved models)

1 2

Concept fusion (explore concept relations) TRECVID 2005 (development) Columbia 374 (baseline models) TRECVID 2007 (development) Cross domain New baseline TRECVID 2007 (test) BCRF (20 improved models) Model adaptation

1 2 3

slide-4
SLIDE 4

Coping with Video Domain Change; Zavesky 2007

4

cross-domain learning

slide-5
SLIDE 5

Coping with Video Domain Change; Zavesky 2007

Definition:

  • Domain: set of content with same production/capture

method and content quality

Problem:

  • Not all data sets are created equal; classifiers trained on one

domain often do not work well on others Goal:

  • Achieve robust detection in new domain with minimal

additional complexity

5

documentary (new domain) TRECVID 2007 news (old domain) TRECVID 2005

slide-6
SLIDE 6

Coping with Video Domain Change; Zavesky 2007

Cross-Domain Problem: What is it?

Approach:

  • Leverage pre-trained existing models
  • Optimal weighted combination of data from both domains

Data:

  • TRECVID2005 (broadcast news @ 100 hours),
  • TRECVID2007 (documentaries @ 60 hours)

6

slide-7
SLIDE 7

Coping with Video Domain Change; Zavesky 2007

Cross-Domain Problem: Common approaches

7

increasing time & complexity

method

  • ld

new applicable condition

use old model all

  • ld domain very

similar to new domain training data

slide-8
SLIDE 8

Coping with Video Domain Change; Zavesky 2007

Case 1: old model works best

Studio

8

learn new domain, test new domain learn old domain, test old domain learn old domain, test new domain

  • top 5 detection results
slide-9
SLIDE 9

Coping with Video Domain Change; Zavesky 2007

Cross-Domain Problem: Common approaches

9

increasing time & complexity

train new domain model

  • all

new and old domains very dissimilar

method

  • ld

new applicable condition

use old model all

  • ld domain very

similar to new domain training data

slide-10
SLIDE 10

Coping with Video Domain Change; Zavesky 2007

Case 2: new model works best

Waterscape

10

learn new domain, test new domain learn old domain, test old domain learn old domain, test new domain

  • top 5 detection results
slide-11
SLIDE 11

Coping with Video Domain Change; Zavesky 2007

Cross-Domain Problem: Common approaches

11

increasing time & complexity

adapt old model small all new and old domains slightly dissimilar

method

  • ld

new applicable condition

use old model all

  • ld domain very

similar to new domain train new model

  • all

new and old domains very dissimilar training data

slide-12
SLIDE 12

Coping with Video Domain Change; Zavesky 2007

Case 3: old model adaptation works best

Charts

12

learn new domain, test new domain learn old domain, test old domain adapt old domain+new domain, test new domain

  • top 5 detection results
slide-13
SLIDE 13

Coping with Video Domain Change; Zavesky 2007

Cross-Domain Problem: Common approaches

13

increasing time & complexity

method

  • ld

new applicable condition

use old model all

  • ld domain very

similar to new domain train new model

  • all

new and old domains very dissimilar adapt old model small all new and old domains slightly dissimilar training data train combined new+old model all all

  • ld and new domains

similar; sparse new domain

  • r strong old model
slide-14
SLIDE 14

Coping with Video Domain Change; Zavesky 2007

Case 4: combined model works best

Sports

14

learn new domain, test new domain learn old domain, test new domain learn old domain+new domain, test new domain

  • top 5 detection results
slide-15
SLIDE 15

Coping with Video Domain Change; Zavesky 2007

Cross-Domain Problem: Common approaches

15

method

  • ld

new applicable condition

use old model all

  • ld domain very

similar to new domain train new model

  • all

new domain and old domains very dissimilar adapt old model small all new and old domains slightly dissimilar train combined new+old model all all

  • ld and new domains

similar; sparse new domain

  • r strong old model

increasing time & complexity

training data adapt old model small all new and old domains slightly dissimilar train combined new+old model all all

  • ld and new domains

similar; sparse new domain

  • r strong old model
slide-16
SLIDE 16

Coping with Video Domain Change; Zavesky 2007

Topic Review: Support Vector Machine (SVM)

16

+ + + + + + Old Feature Space + positive

  • ld samples

+ + +

+ + +

+ + + + + + Old Feature Space + positive

  • ld samples

+ + + x x x x x x x x x x xx x negative

  • ld samples

+ + +

decision boundary (hyperplane)

  • ld support vectors

(evidence for decision boundary) +

+

+ + + + + Old Feature Space + positive

  • ld samples

+ + + x x x x x x x x x x

x

x negative

  • ld samples

margin distance 1 || w ||

x + +

New Feature Space positive new samples negative new samples +

+ + x x

x x + + +

NOTE: new domain can be sparse

x x x x new samples

  • ld

support vectors classification errors (using old hyperplane) New Feature Space +

+ + x

x x + + + x x x x

+ x x + +

new samples

  • ld

samples

helpful support vector from old domain new samples

  • ld

support vectors New Feature Space +

+ + x

x x + + + x x x x

+ x x + +

ignored support vector from old domain adapted hyperplane

slide-17
SLIDE 17

Coping with Video Domain Change; Zavesky 2007

  • Idea: includes all data (new and old) in training of new

domain models

  • Kernel matrix: equal weights for all samples

Combined model: Uniform sample importance

17

  • ld

new

  • ld

1x new

learn new models new samples

  • ld

samples

features

slide-18
SLIDE 18

Coping with Video Domain Change; Zavesky 2007

  • ld

new

  • ld

1x new

  • Idea: augment feature vector to learn intra-domain weights

across many dimensions

  • Cross-domain training may be quite dissimilar
  • Trust intra-domain similarity more
  • Intelligent method for feature expansion

Replication model: Kernel matrix replication

18

  • H. Daume III, “Frustratingly easy domain adaptation”, Proc. the 45th Annual Meeting of the Association of Computational Linguistics, 2007

learn new models new samples

  • ld

samples

features

  • ld

new

  • ld

2x 1x new 1x 2x

replication processor

slide-19
SLIDE 19

Coping with Video Domain Change; Zavesky 2007

  • Idea: augment feature vector to learn intra-domain

weights across many dimensions

Replication model: Kernel matrix replication

19

  • H. Daume III, “Frustratingly easy domain adaptation”, Proc. the 45th Annual Meeting of the Association of Computational Linguistics, 2007

D*3 features

+1 0.2 0.4 ...

  • ld

+1 0.9 0.9 ...

new D features M samples

+1 0.4 0.2 0.4 0.2 0.0 0.0 ... +1 0.9 0.9 0.0 0.0 0.9 0.9 ...

N samples M + N samples

general feats

  • ld

features new features

slide-20
SLIDE 20

Coping with Video Domain Change; Zavesky 2007

  • Idea: trust old domain model more than new domain
  • Perturb old model within some tolerance with

weighted new samples and a constant offset

Adaptive SVM (A-SVM): Constrained model adaptation

20

  • ld

models

extract support vectors

learn new models

  • ld

support vectors new samples

  • ptimize choice
  • f new samples

and old support vectors

J. Yang, et al., “Cross-domain video concept detection using adaptive svms”, ACM Multimedia, 2007.

f(x) = f old(x) + ∆f(x)

slide-21
SLIDE 21

Coping with Video Domain Change; Zavesky 2007

Adaptive SVM (A-SVM): Constrained model adaptation

21

J. Yang, et al., “Cross-domain video concept detection using adaptive svms”, ACM Multimedia, 2007. decision boundary (hyperplane)
  • ld support vectors
(evidence for decision boundary)

+

Old Feature Space

x

margin distance 1 || w ||

x + +

New Feature Space +

+ + x

x x + + + x x x x

New Feature Space

+ + x

x x + + + x x x x

+ x x + +

margin distance 1 || w || constrained adjustment

new samples
  • ld
support vectors adapted hyperplane

+

slide-22
SLIDE 22

Coping with Video Domain Change; Zavesky 2007

  • Idea: trust support vectors from trained old domain

model as best observations in old domain

  • Weigh SVs then combine with new data and retrain

Cross-domain SVM (CD-SVM): Adapting prior models

22

  • ld

models

extract support vectors

learn new models

new samples

  • ld

support vectors compute similarity

features samples weighted samples

Submitted: W. Jiang, E. Zavesky, S.F. Chang, A. Loui, “Cross-domain learning methods for high-level concept classification,” ICASSP 2008.
slide-23
SLIDE 23

Coping with Video Domain Change; Zavesky 2007

Cross-domain SVM (CD-SVM): Adapting prior models

23

Submitted: W. Jiang, E. Zavesky, S.F. Chang, A. Loui, “Cross-domain learning methods for high-level concept classification,” ICASSP 2008.

min

w

1 2||w||2

2 + C

|Dnew|

i=1

ǫi +C M

j=1 σ(vold j

, Dnew)ǫj σ(vold

j

, Dnew) = 1 |Dnew|

  • (xi,yi)∈Dnew exp
  • −β||vold

j

− xi||2

2

  • decision boundary
(hyperplane)
  • ld support vectors
(evidence for decision boundary)

+

Old Feature Space

x

margin distance 1 || w ||

x + +

helpful support vector from old domain new samples
  • ld
support vectors New Feature Space +

+ + x

x x + + + x x x x

+ x x + +

ignored support vector from old domain adapted hyperplane
slide-24
SLIDE 24

Coping with Video Domain Change; Zavesky 2007

Cross-domain methods: Observed speed trends

24 method

  • ld

new example training cost

  • ld model

all

  • 0x

combined all all 3x replication all all 9x CDSVM small all 1.25x

increasing observed performance

Theoretical training cost with 40k samples in old domain, 20k in new domain (similar to TRECVID problem)

new model

  • all

1x

training data

slide-25
SLIDE 25

Coping with Video Domain Change; Zavesky 2007

Choosing an approach...

  • No single approach is always optimal, but

predictions can be found in a piece-wise manner

  • Based on available statistics
  • Positive new domain samples strongly relates to

ideal training conditions for each approach...

25

slide-26
SLIDE 26

Coping with Video Domain Change; Zavesky 2007

Performance comparison: High positive frequency

26

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Person Face Outdoor Crowd Vegetation Sky Building Walking_Running Urban Office Road Waterscaoe Concept AP (Positive freq > 0.05) AP target combined replication CDSVM A-SVM
  • No clear winners
slide-27
SLIDE 27

Coping with Video Domain Change; Zavesky 2007

Performance comparison: Mid positive frequency

27

shaded regions best per concept (5% relative improvement over all others)

  • Clear differentiation seen
0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Meeting Car Animal Computer_TV Studio Military People-Marching Boat_Ship Sports Police_Security Truck Concept AP (0.05 <= Positive freq <= 0.01) AP new combined replication CDSVM A-SVM
slide-28
SLIDE 28

Coping with Video Domain Change; Zavesky 2007

0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Charts Snow Mountain Weather Court Maps Explosion_Fire Prisoner Desert Bus Natural-Disaster Airplane Flag-US Concept AP (Positive freq. < 0.01) AP

Performance comparison: Low positive frequency

28

shaded regions best per concept (5% relative improvement over all others)

  • More differentiation, but less reliable in for low performance
target combined replication CDSVM A-SVM

Observed differences but performance too low for significant conclusions?

slide-29
SLIDE 29

Coping with Video Domain Change; Zavesky 2007

Which approach to choose to obtain good new domain performance

  • Decision based on frequency of positive samples

and performance of old model...

  • High frequency (old or new more than 5%)

select CDSVM (adapts old to well-defined new domain)

  • person, sky, road, ...

29

slide-30
SLIDE 30

Coping with Video Domain Change; Zavesky 2007

Which approach to choose to obtain good new domain performance

  • Decision based on frequency of positive samples

and performance of old model...

  • High frequency (old or new more than 5%)
  • Mid-frequency (new < 5%, new > 1%)
  • If performance (AP) of old model was high,

select replication (learn combined trends)

  • truck, car, people-marching
  • If AP was too low,

select new domain only (not enough evidence)

30

slide-31
SLIDE 31

Coping with Video Domain Change; Zavesky 2007

Which approach to choose to obtain good new domain performance

  • Decision based on frequency of positive samples

and performance of old model...

  • High frequency (old or new more than 5%)
  • Mid-frequency (new < 5%, new > 1%)
  • Low-frequency (new < 1%)
  • If sparse old (old < %1)

select new (sparsity risk too high)

  • boats, computer-tv, map, explosion-fire

31

slide-32
SLIDE 32

Coping with Video Domain Change; Zavesky 2007

Which approach to choose to obtain good new domain performance

  • Decision based on frequency of positive samples

and performance of old model...

  • High frequency (old or new more than 5%)
  • Mid-frequency (new < 5%, new > 1%)
  • Low-frequency (new < 1%)
  • Otherwise, choose default model...

32

slide-33
SLIDE 33

Coping with Video Domain Change; Zavesky 2007

Approach selection: Empirical rule set

  • Aggregating these intuitions, we can create a ruleset to

choose an approach that optimizes new domain performance

33

strong old model high frequency new and old too sparse default choice

slide-34
SLIDE 34

Coping with Video Domain Change; Zavesky 2007

Approach selection: Rule-based benefits

34

Observed MAP improvement over new model alone

high frequency mid frequency low frequency

8.7% 29.8% 24.6% shaded regions best per concept (5% relative improvement over all others)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Person Face Outdoor Crowd Vegetation Sky Building Walking_Running Urban Office Road Waterscaoe Concept AP (Positive freq > 0.05) AP 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Meeting Car Animal Computer_TV Studio Military People-Marching Boat_Ship Sports Police_Security Truck Concept AP (0.05 <= Positive freq <= 0.01) 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Charts Snow Mountain Weather Court Maps Explosion_Fire Prisoner Desert Bus Natural-Disaster Airplane Flag-US Concept AP (Positive freq. < 0.01) target combined replication CDSVM A-SVM
slide-35
SLIDE 35

Coping with Video Domain Change; Zavesky 2007

35

TRECVID 2007 high level features

slide-36
SLIDE 36

Coping with Video Domain Change; Zavesky 2007

36

Columbia HLF: TRECVID2007

Concept fusion (explore concept relations) TRECVID 2005 (development) Columbia 374 (baseline models) TRECVID 2007 (development) Cross domain New baseline TRECVID 2007 (test) BCRF (20 improved models) Model adaptation

1 2 3

slide-37
SLIDE 37

Coping with Video Domain Change; Zavesky 2007

Empirical Results: TRECVID2007

  • 4 of 6 runs in top 20
  • Less than 0.005 MIAP difference between new models and

replicated models

  • Only replication model

was submitted

  • Cross-domain fusion

improved performance for most concepts

  • Color moment,

edge direction histogram Gabor texture

37

Concept fusion (explore concept relations) TRECVID 2005 (development) Columbia 374 (baseline models) TRECVID 2007 (development) Cross domain New baseline TRECVID 2007 (test) BCRF (20 improved models) Model adaptation 1 2 3
slide-38
SLIDE 38

Coping with Video Domain Change; Zavesky 2007

0.05 0.10 0.15 0.20 0.25 0.30 0.35 MIAP sports weather
  • ffice
meeting desert mountain waterscape police military animal computer_tv flag-us airplane car truck boat_ship marching explosion_fire maps charts IAP

Empirical performance:

Method comparisons; new vs. replication

38

R5: replicate R6: new R4: bcrf+new R3: bcrf+replicate+new R2: bcrf+replicate+new+old R1: adaptive selection

replication replication

slide-39
SLIDE 39

Coping with Video Domain Change; Zavesky 2007

conclusions & next steps

39

slide-40
SLIDE 40

Coping with Video Domain Change; Zavesky 2007

Conclusions

  • Cross-domain helps to cope with domain change
  • When new domain model is weak, good to use old domain

data and models

  • Move models into new domain with minimal complexity

increase and maintain performance

  • Explore different different model approaches
  • No universally superior approach
  • Performance predictors: frequency of new and old

domain and domain similarity

  • Prediction using domain properties works reasonably well

40

slide-41
SLIDE 41

Coping with Video Domain Change; Zavesky 2007

Next Steps: Technical questions for adaptation

  • When to adapt vs. training new model
  • Rules are first step, but deeper data

distribution analysis is underway

  • Next problem: few or no labels on new domain
  • Leveraging large concept ontology (LSCOM)
  • Adaptation needed for concept-based

approaches on new data

41

slide-42
SLIDE 42

Coping with Video Domain Change; Zavesky 2007

Thanks for your time.

42