V V ISUAL ISUAL G RAPH RAPH M M ODELING AND R R ETRIEVAL ODELING AND - - PowerPoint PPT Presentation

▶
v
SMART_READER_LITE
LIVE PREVIEW

V V ISUAL ISUAL G RAPH RAPH M M ODELING AND R R ETRIEVAL ODELING AND - - PowerPoint PPT Presentation

Feb 17, 2023 28 likes •533 views

V V ISUAL ISUAL G RAPH RAPH M M ODELING AND R R ETRIEVAL ODELING AND ETRIEVAL A L A L ANGUAGE ANGUAGE M M ODEL ODEL A A PPROACH PPROACH FOR S CENE CENE R R ECOGNITION FOR ECOGNITION PHAM PHAM T PHAM T PHAM T RONG T RONG RONG RONG

slide-1
SLIDE 1

V VISUAL

ISUAL GRAPH RAPH M

MODELING

ODELING AND AND R

RETRIEVAL

ETRIEVAL

A L A LANGUAGE

ANGUAGE M

MODEL

ODEL A

APPROACH

PPROACH FOR FOR SCENE CENE R

RECOGNITION

ECOGNITION

PHAM PHAM T TRONG

RONG

T TÔN

ÔN

PHAM PHAM T TRONG

RONG

T TÔN

ÔN

MRIM TEAM & IPAL LAB

  • !"

##$!% & '#$ #( )*+!," -(# .(/(#$# 0))*,"-(# 1#"--*,,&,"2((

slide-2
SLIDE 2

'*,0"0'*,0",0$)." & +" -"0-&"3

Goal ((# &(##4((

4#(#

1((((

((#

Thesis focus +($((# +((# Applications *#### &#####(5(#

2

slide-3
SLIDE 3
  • *'! *3"-3&"6

3

Graph matching Query graph model Document graph model Image ranking

slide-4
SLIDE 4

*0&,"

(& 3(( (&& 3(( (&&& (&3

4

3(( 2 3(( /( (# '##

slide-5
SLIDE 5
  • 0 &3&. +-! &,"&,+

(& 3(( (&& 3(( (&&& (&3

5

7 .(#4( 7 3((2

3(( 2 3(( /( (# '##

slide-6
SLIDE 6

0&" & +" *&,0 *1 3&"6.

44%(#4(####(#8

(#8##

44/(4(2(##88#(/((

4(

Part I: Visual graph indexing 97.(#4(

6

  • #(#

+(# &#

slide-7
SLIDE 7

3"'0*-$)." & +" -"-".",00&*,

&4#2/(#(

: :(#

&( )(#4/(%#

Part I: Visual graph indexing 97.(#4(

:(# :#44 $ 4((# $ (/# $ (;#4((4#(#

7

slide-8
SLIDE 8

.0-'0-" & +" -"-".",00&*, '*,07

$##(((

( +##

Part I: Visual graph indexing 97.(#4(

8

  • :(#(#;(#

:<#(# $ #$%#(;

slide-9
SLIDE 9

.0-'0-" & +" -"-".",00&*, '*,07

((( ,((

1#(((4###

Part I: Visual graph indexing 97.(#4(

9

! "#

:(#(((# $ #$%#(;

slide-10
SLIDE 10

*- -**.3&. +-! *"&,+

visual graph (##visual concepts,

spatial relations ( weights/probabilities

Part I: Visual graph indexing 97.(#4(

top of87= top of87 left of87> ; 7?

  • 7=

( 7

10

top of87? left of87@ 2 /(

slide-11
SLIDE 11

3&. +-! &,"&,+ .'!" "

Part I: Visual graph indexing 973((2

11

slide-12
SLIDE 12

3&. +-! &,"&,+ .'!" "

Part I: Visual graph indexing 973((2

12

slide-13
SLIDE 13

& +" -*'"..&,+

Part I: Visual graph indexing 973((2

&(### 1(2(#

  • '###(
  • "#(
  • .&10A #$%&&&

13

  • A.&10.(&/((1(0(4#
slide-14
SLIDE 14

3&. +-! &,"&,+ .'!" "

Part I: Visual graph indexing 973((2

14

slide-15
SLIDE 15
  • /(%kmeans clustering
  • k#4/(#
  • */(# 4#(4(∈

∈ ∈ ∈

3&. '*,'"0 "-,&,+

Part I: Visual graph indexing 973((2

visual concepts

15

c1 c3 c2

' B

F CD'##8"8.&10E

3(#Cf

slide-16
SLIDE 16

3&. +-! &,"&,+ .'!" "

Part I: Visual graph indexing 973((2

16

slide-17
SLIDE 17

3&. +-! '*,.0-'0&*,

1#((I "2(##4weighted concept set

4#(/(4(f

"2(##4weighted relation set

%%##

WCI

f (WCI f’ 4#(((#l

($(#WCI

f = WCI f’

$(#WCI

f ≠ WCI f’

Part I: Visual graph indexing 973((2

17

&($(# &$(#

cooccurrence left_of top_of

slide-18
SLIDE 18

1# ( ( ## C .#4%#

SWC = ∪WCf

.#4%(#

S = ∪WE

3&. +-! '*,.0-'0&*, '*,07

Part I: Visual graph indexing 973((2

∪

SWE = ∪WEl

1#((I .#4%#

SI

WC = ∪WCf I

.#4%(#

SI

WE = ∪WEl I 18 18

slide-19
SLIDE 19

3&. +-! "1&,&0&*,

3((4#((&GI = <SI

WC, SI WE>

SI

(

,# c c B l ,w’

Part I: Visual graph indexing 973((2

19

SI

WC

SI

WE

"

(

c1 ci B B c1 w1 l1,w’1 3((GI l1 lj l2,w’2 l3,w’3 c2 w2 c3 w3

slide-20
SLIDE 20

" "

  • Part I: Visual graph indexing 973((2
  • = {(c1,3), (c2,2), (c3,2), (c4,2)}
  • = {(c1,c1,left_of,2), (c2,c2,left_of,1),

(c2,c3,left_of,1), (c4,c3,left_of,1), (c4,c4,left_of,1)}

20

slide-21
SLIDE 21
  • *'! *3"-3&"6

21

Graph matching Query graph model Document graph model Image ranking

slide-22
SLIDE 22
  • 0 &&3&. +-! -"0-&"3

(& 3(( (&& 3(( (&&& (&3

22

7 .(#4( 7 3((/(

3(( 2 3(( /( (# '##

slide-23
SLIDE 23

'--",0 0'!&,+ "0!*.

&2(((2D Multiresolution

Hidden Markov Model

Part II : Visual graph retrieval 97.(#4(

  • "(##4! (;#/ #(#

23

slide-24
SLIDE 24

'--",0 0'!&,+ "0!*. '*,07

Kernelbased graph clustering (#

  • Part II : Visual graph retrieval 97.(#4(

24

! "# (#4#(((

slide-25
SLIDE 25

'--",0 0'!&,+ "0!*. '*,07

(%language modeling (#bag of visual words n$(#n C8?8@

Part II : Visual graph retrieval 97.(#4(

25

' ( .(((#(4implicitly n$(<

3(

slide-26
SLIDE 26

,++" *" &, &,1*- 0&*, -"0-&"3

F;###(

F F #

  • ;##

P(Q|D)

)*+)%&&(

Part II : Visual graph retrieval 97.(#4(

(#G#(#

26

,

  • .

/ , . / , . , .

  • =

= ∏

∈

  • "44#4#2/(&-

.##<

slide-27
SLIDE 27

*- -**.3&. +-! 0'!&,+

+(((#(#language modeling

((;#(#

#4visual concepts # #4relations Weight/probability #4#((#

Part II : Visual graph retrieval 97.(#4(

27

000 & % top of87= top of87 left of87> ; 7?

  • 7=

( 7 top of87? left of87@

slide-28
SLIDE 28

3&. +-! -"0-&"3 .'!" "

Part II : Visual graph retrieval 973((/(

28

slide-29
SLIDE 29

3&. +-! 0'!&,+

& 8probability likelihood #4

(<( 4##(

  • Part II : Visual graph retrieval 973((/(

, 1 2 . , 2 . , 2 .

  • ×

=

'#

  • (#

29

%

  • <(
  • #(
slide-30
SLIDE 30
  • *))&&0H *1 '*,'"0 ."0.

'#independent hypothesis ∩fWCq = Ø

Sq

WC

WC1 WCf

B

Part II : Visual graph retrieval 973((/(

Multinomial distribution #4#WCq

30

∏

∈

∝

  • ,

1 . /

, 2 . , 2 .

∏

∈

=

  • ,

2 . , 2 .

slide-31
SLIDE 31

. **0!&,+ 0"'!,&F".

#I#J 4##

P(c | Gd) = 0 P (Gq | Gd) = 0

.##/((#(4###C

4#(I#J

Part II : Visual graph retrieval 973((/(

4#(I#J

*##(;$ ##&-

31

  • %

1

  • $)

, .-1 / , 1 . / , .-1 / , 1 . / , % . , 2 . ∈ + − =

  • λ

λ λ

slide-32
SLIDE 32
  • *))&&0H *1 -"0&*, ."0.
  • (#independent hypothesis ∩lWEq = Ø

#(##

∏

∈

=

  • ,

1 2 . , 1 2 .

Part II : Visual graph retrieval 973((/(

#(## ;$ ##

32

∏

× × ∈

= ∝

  • 3

, 1 3 1 . , 1 1 3 1 . /

, 1 3 1 2 , 3 1 . . , 1 2 . , 1-1 3 1 . / , 1 1 3 1 . / , 1-1 3 1 . / , 1 1 3 1 . / , % . , 1 3 1 2 , 3 1 . .

  • λ

λ + − = =

slide-33
SLIDE 33

3&. +-! 0'!&,+ " "

top of87= top of87 left of87> ' 7? ' 7= '? 7 top of87? left of87@ ' ? ' > top of8K left of8=

Part II : Visual graph retrieval 973((/(

33

7

#Gd FGq P(Gq|Gd)= P(C1|Gd)3.P(C2|Gd)6 × P(L(C1,C2)=top_of|WCq,Gd)7.P(L(C1,C2)=left_of|WCq,Gd)5 = (0.3)3. (0.5)6. (0.3)7. (0.4)5 = 0.59049.108 &(((;(#probability likelihoods

slide-34
SLIDE 34
  • *'! *3"-3&"6

34

Graph matching Query graph model Document graph model Image ranking

slide-35
SLIDE 35
  • 0 &&&&'0&*,.

(& 3(( (&& 3(( (&&& (&3

35

7 .## 7 -###(5(#

3(( 2 3(( /( (# '##

slide-36
SLIDE 36

. -H

7

*####

7

&#####(5(#

CORRIDOR

36

slide-37
SLIDE 37

& +"$)." *)&" 0*- +&"

Part III : Applications 97.##

37

slide-38
SLIDE 38

.0*&'$'*"'0&*,

Training Test Overall

&( ?LM >> 3849 .

  • 101

Part III : Applications 97.##

38

0.(#0#*N &4(#'##

44 *#(#/#N 3((##4/%#8( 3((##4##

slide-39
SLIDE 39

"30&*, "0!*.

./((#4#((<

Trained by I Trained by S

Part III : Applications 97.##

39

by I by S Query by I Query by S

slide-40
SLIDE 40

3&. +-! *".

mg #

  • Summary
  • =/(#
  • #WCmg # WCgg
  • ($(#WEleft_of , WEtop_of
  • Implemented models

∅

Part III : Applications 97.##

40

gg #

  • Implemented models
  • !"#$%= < {WCmg}, ∅ >

&

!"#'%< {WCmg},{WEleft_of, WEtop_of }>

(

""#$% = < {WCgg}, ∅ >

)

""#'%= < {WCgg}, {WEleft_of, WEtop_of } >

slide-41
SLIDE 41

""-& ",0 -".0.

Part III : Applications 97.##

0( F mgLM !"#'% ggLM ""#'% '(4(#(( &(((= TPi / Ni (Ni = >>) .((= TPs / Ns (Ns = )

VGMs %(((##4#LMs .4((#4multiple querying images (S)

41

0( F mgLM !"#'% ggLM ""#'% & & 7KLM 0.794 :7>O 7@L@ 7==:?7LO & . 7L 1.00 :7>O 7@>= 7K>:>?7LO . & 7=M =M@:7?O 7@KL 0.603 :>7O . . 1.00 1.00 7LM 7M:?7O

slide-42
SLIDE 42

'* -&.*, 6&0! 0!" .00"$*1$0!"$-0

.3 A #-)1;%#/((#

VGM #4##LM (SVM #

#class SVM LM VGM mgconcepts

  • 7K@@

7KLM:>7O 0.794 :>7?O

Part III : Applications 97.##

VGM #4##LM (SVM #

&(# C/C++ %LTILib #2(4# 3.0 GHz quadcore '(8.0 Gb #4# "2#0.22 seconds (

42

A.3 .#3# (

slide-43
SLIDE 43

. -H

7

*####

7

&#####(5(#

CORRIDOR

43

slide-44
SLIDE 44
  • 0 &3'*,'.&*,

(& 3(( (&& 3(( (&&& (&3

44

7 '## 7 /

3(( 2 3(( /( (# '##

slide-45
SLIDE 45

'*,0-&)0&*,.

graphbased image representation 4#

(2(/(

visual concept sets relation sets weight/probability #4/(#((#

Part IV : Conclusions 97'##

weight/probability #4/(#((# simple (effective ((

#

(#language modeling &- multinomial distribution (independent

hypotheses

generality (expendability 44#2

45

slide-46
SLIDE 46

'*,0-&)0&*,. '*,07

(####4($4scene

recognition

/((#((#mobile device #/((%spatial relations

Part IV : Conclusions 97'##

"2#robot localization

#/(4#(#

/((#(#%

illumination and environment changes

#4#4#(#4stateofthe

art SVM method

46

slide-47
SLIDE 47

.!*-0$0"- "-."'0&3".

'#(##42((#(/((

#(##4(%#;

1#/(#((((#

Part IV : Conclusions 97/

"/((##4##(#(#((

##

*N(4(#83*' 3#/(80-"'3&

47

40 5

slide-48
SLIDE 48

*,+$0"- "-."'0&3".

Relevance modeling 4#(#/ KullbackLeibler (KL) / #(

/%<#(##

Part IV : Conclusions 97/

"2##4probabilistic framework 4##4“soft” /(#(#fuzzy cmeans #

ExpectationMaximization clustering

48

EM clustering

slide-49
SLIDE 49

)&'0&*,.

Journal Peerreviewed Articles 1.

TrongTon Pham8 8# (#(8"+(8##$ !% 7Visual Graph Modeling for Scene Recognition and Robot Localization7 #(# (0##((#8(8.8((7

2.

TrongTon Pham8# (#(8 8"+(7Modèle de graphe et modèle de langue pour la reconnaissance de scènes visuelles7,# (/#,<83#?$L8(/#8 7

International Peerreviewed Conference Articles International Peerreviewed Conference Articles 1.

TrongTon Pham8 8# (#(7Spatial Relationships in Visual Graph Modeling for Image Categorization7##4??' .&+&-P8(KM$K?8+/(8.%5(87

2.

TrongTon Pham8 8# (#(8"+(7 Integration of Spatial Relationship in Visual Language Model for Scene Retrieval7 &"""L&(#(6#;##'#$)( (&2 ') &8>(8+#81(87

3.

TrongTon Pham8# (#(8 8"+(7Visual Language Model for Scene Recognition7.(#($1&.# .1(PM8L(8.(#8M7

4.

TrongTon Pham8,#( (#8##$!% 8($'/(7Latent Semantic Fusion Model for Image Retrieval and Annotation7' >'#4 #&4#(#(Q#% (('&Q 8(@?M$@@@8#(8 #(8K7

49

slide-50
SLIDE 50

0!,Q H*

F###R

50

S2(