Semantic Adaptation of Multimedia Documents S ebastien Laborie - - PowerPoint PPT Presentation

▶
semantic adaptation of multimedia documents
SMART_READER_LITE
LIVE PREVIEW

Semantic Adaptation of Multimedia Documents S ebastien Laborie - - PowerPoint PPT Presentation

Oct 04, 2022 18 likes •878 views

Semantic Adaptation of Multimedia Documents S ebastien Laborie Post-doc 1 S ebastien Laborie Semantic Adaptation of Multimedia Documents Introduction A multimedia document Composed of objects Text Image Audio Video Assembled by an

slide-1
SLIDE 1

Semantic Adaptation of Multimedia Documents

S´ ebastien Laborie Post-doc

1 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-2
SLIDE 2

Introduction

A multimedia document

Composed of objects Text Image Audio Video Assembled by an author Temporal dimension Spatial dimension Hypermedia dimension

2 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-3
SLIDE 3

Introduction

Multimedia document adaptation

3 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-4
SLIDE 4

Introduction

Multimedia document adaptation

Profile1 Profile2 Profile3

3 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-5
SLIDE 5

Introduction

Multimedia document adaptation

Profile1 Profile2 Profile3 Adaptation

3 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-6
SLIDE 6

Semantic adaptation

Outline

1

Semantic adaptation

2

Spatio-temporal and hypermedia specification

3

Spatio-temporal and hypermedia adaptation

4

SMIL adaptation

5

Semantic media adaptation

4 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-7
SLIDE 7

Semantic adaptation

Outline

1

Semantic adaptation

2

Spatio-temporal and hypermedia specification

3

Spatio-temporal and hypermedia adaptation

4

SMIL adaptation

5

Semantic media adaptation

5 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-8
SLIDE 8

Semantic adaptation

Current adaptation techniques

Specification of alternatives

based on target profiles Adaptation Initial document Adapted document Profile

6 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-9
SLIDE 9

Semantic adaptation

Current adaptation techniques

Specification of alternatives

based on target profiles based on content Adaptation Stylesheet Initial document Adapted document Profile

6 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-10
SLIDE 10

Semantic adaptation

Current adaptation techniques

Specification of alternatives

based on target profiles based on content both approaches Adaptation Initial document Adapted document Profile Stylesheet

6 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-11
SLIDE 11

Semantic adaptation

Current adaptation techniques

Specification of alternatives

based on target profiles based on content both approaches Adaptation Initial document Stylesheet

Hard to predict target profiles

6 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-12
SLIDE 12

Semantic adaptation

Current adaptation techniques

Specification of alternatives

based on target profiles based on content both approaches

Specification of transformation rules

Adaptation

T1:VideoToImages T2:TextToAudio . . .

Repository of transformation rules Profile Initial document Adapted document

6 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-13
SLIDE 13

Semantic adaptation

Current adaptation techniques

Specification of alternatives

based on target profiles based on content both approaches

Specification of transformation rules Specification of abstract models

Adaptation Profile Data Narrative structure

intro problem image section

present show before 6 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-14
SLIDE 14

Semantic adaptation

Current adaptation techniques

Specification of alternatives

based on target profiles based on content both approaches

Specification of transformation rules Specification of abstract models Requirements of an adaptation approach flexible and general

  • Cover all document dimensions
  • Transformations not explicit

Language independant

6 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-15
SLIDE 15

Semantic adaptation

Adaptation quality

”The basic assumption is that content transformation activities should be provided as non-destructive operations.” (DocEng’08) ⇒ One approach is to measure the adaptation transformations Different kind of metrics: Measuring the discourse evolution Measuring the composition degradation Measuring the content transformation

7 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-16
SLIDE 16

Semantic adaptation

Adaptation quality

”The basic assumption is that content transformation activities should be provided as non-destructive operations.” (DocEng’08) ⇒ One approach is to measure the adaptation transformations Different kind of metrics: Measuring the discourse evolution Measuring the composition degradation Measuring the content transformation

7 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-17
SLIDE 17

Semantic adaptation

Semantic adaptation of multimedia documents

An execution is considered as a set of objects

Objects may be composed temporally e4

Poster R´ esum´ e Personnages Bande-annonce Dates 8 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-18
SLIDE 18

Semantic adaptation

Semantic adaptation of multimedia documents

An execution is considered as a set of objects

Objects may be composed temporally e2 e4

Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates 8 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-19
SLIDE 19

Semantic adaptation

Semantic adaptation of multimedia documents

A document can be interpreted as a set of executions

Is e1 e3 e5 e2 e4

Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates 8 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-20
SLIDE 20

Semantic adaptation

Semantic adaptation of multimedia documents

A document can be interpreted as a set of executions

Some satisfy the author intention Is e1 e3 Ms e5 e2 e4

Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates 8 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-21
SLIDE 21

Semantic adaptation

Semantic adaptation of multimedia documents

The target device accepts only some executions

Profile: no more than 2 objects executed simultaneously Mp Is e1 e3 Ms e5 e2 e4

Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates 8 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-22
SLIDE 22

Semantic adaptation

Semantic adaptation of multimedia documents

Adapting amounts to find possible initial executions (Ms ∩ Mp)

Here, no adaptation is needed Mp Ms ∩ Mp Is e1 e3 Ms e5 e2 e4

Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates 8 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-23
SLIDE 23

Semantic adaptation

Semantic adaptation of multimedia documents

Other profiles can be considered

Profile: satisfy the presentation contiguity Mp Is e1 e3 Ms e5 e2 e4

Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates 8 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-24
SLIDE 24

Semantic adaptation

Semantic adaptation of multimedia documents

The adaptation selects some possible initial executions

Refining adaptation Mp Ms ∩ Mp Is e1 e3 Ms e5 e2 e4

Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates 8 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-25
SLIDE 25

Semantic adaptation

Semantic adaptation of multimedia documents

In other cases, no initial execution can be possible

Profile: execute one objets at a time Mp Is e1 e3 Ms e5 e2 e4

Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates 8 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-26
SLIDE 26

Semantic adaptation

Semantic adaptation of multimedia documents

The adaptation selects some executions close to initial ones

Transgressive adaptation Ms ∩ Mp = ∅ d =? Mp Is e1 e3 Ms e5 e2 e4

Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates Poster R´ esum´ e Personnages Bande-annonce Dates 8 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-27
SLIDE 27

Semantic adaptation

Our contribution

Bridge the gap between theory and application

Consider all multimedia document dimensions

Temporal dimension Spatial dimension Hypermedia dimension

Consider the combination of all dimensions

Adapting the spatio-temporal and hypermedia dimension

Consider standard multimedia documents

Adapting SMIL documents

Consider media adaptation

9 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-28
SLIDE 28

Spatio-temporal and hypermedia specification

Tables des mati` eres

1

Semantic adaptation

2

Spatio-temporal and hypermedia specification

3

Spatio-temporal and hypermedia adaptation

4

SMIL adaptation

5

Semantic media adaptation

10 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-29
SLIDE 29

Spatio-temporal and hypermedia specification

Multimedia document specification

Definition (Multimedia document specification) A multimedia document specification s = O, C is composed of a set of

  • bjects O and a set of relations (or constraints) C between the elements of

O. Example (Spatio-temporal and hypermedia specification) Acropole Agora Temple Mus´ ee l1 l2

0s 5s 10s 20s

11 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-30
SLIDE 30

Spatio-temporal and hypermedia specification

Multimedia document specification

Definition (Relation graph) A multimedia document specification s = O, C can be represented with a relation graph: O is the set of nodes and edges are labeled by elements from C. Example (Spatio-temporal and hypermedia relation graph) Acropole Agora Temple Mus´ ee l1 l2

11 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-31
SLIDE 31

Spatio-temporal and hypermedia specification

Multimedia document specification

Definition (Relation graph) A multimedia document specification s = O, C can be represented with a relation graph: O is the set of nodes and edges are labeled by elements from C. Example (Spatio-temporal and hypermedia relation graph) Acropole Agora Temple Mus´ ee l1 l2

r1 r2 r3 r5 r7 r8

11 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-32
SLIDE 32

Spatio-temporal and hypermedia specification

Multimedia document specification

Definition (Relation graph) A multimedia document specification s = O, C can be represented with a relation graph: O is the set of nodes and edges are labeled by elements from C. Example (Spatio-temporal and hypermedia relation graph) Acropole Agora Temple Mus´ ee l1 l2

r1 r2 r3 r5 r7 r8 r4 r6 r9 r10 r11 r12 r13 r14 r15

11 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-33
SLIDE 33

Spatio-temporal and hypermedia specification

Spatio-temporal and hypermedia specification

Definition (Spatio-temporal relation) A spatio-temporal relation r = rs, rt is composed of a spatial relation rs and a temporal relation rt. RCC8 spatial representation a b DC a b EC a b PO ab ab EQ b a TPP a b TPPi b a NTPP a b NTPPi Allen temporal representation

x r y x / y y r−1 x before (b) (bi) after meets (m) (mi) met-by during (d) (di) contains

  • verlaps (o)

(oi) overlapped-by starts (s) (si) started-by finishes (f ) (fi) finished-by equals (e) (e)

12 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-34
SLIDE 34

Spatio-temporal and hypermedia specification

Spatio-temporal and hypermedia specification

Definition (Relation graph) A multimedia document specification s = O, C can be represented with a relation graph: O is the set of nodes and edges are labeled by elements from C. Example (Spatio-temporal and hypermedia specification) Acropole Agora Temple Mus´ ee l1 l2

{e, DC} {m, EQ} {m, DC} {si, EQ} {fi, DC} {m, DC} {m, EQ} {si, DC} {fi, EQ} {e, DC} {bi, EQ} {mi, DC} {bi, DC} {mi, EQ} {m, DC}

13 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-35
SLIDE 35

Spatio-temporal and hypermedia specification

Spatio-temporal and hypermedia specification

Definition (Relation graph) A multimedia document specification s = O, C can be represented with a relation graph: O is the set of nodes and edges are labeled by elements from C. Example (Spatio-temporal and hypermedia specification) Acropole Agora Temple Mus´ ee l1 l2

{e, DC,fi, DC} {m, EQ} {m, DC, b, DC} {si, EQ} {fi, DC} {m, DC} {m, EQ, b, EQ} {si, DC} {fi, EQ} {e, DC,fi, DC} {bi, EQ} {mi, DC} {bi, DC} {mi, EQ} {m, DC}

13 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-36
SLIDE 36

Spatio-temporal and hypermedia adaptation

Outline

1

Semantic adaptation

2

Spatio-temporal and hypermedia specification

3

Spatio-temporal and hypermedia adaptation

4

SMIL adaptation

5

Semantic media adaptation

14 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-37
SLIDE 37

Spatio-temporal and hypermedia adaptation

Transgressive adaptation

Ms

e4 e5

Profile no simultaneous executions no side by side objects

Acropole Agora Mus´ ee Temple Acropole Agora Mus´ ee Temple 15 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-38
SLIDE 38

Spatio-temporal and hypermedia adaptation

Transgressive adaptation

Ms

e4 e5

Mp

e2 e3

Acropole Agora Mus´ ee Temple Acropole Agora Mus´ ee Temple Acropole Agora Mus´ ee Temple Acropole Agora Mus´ ee Temple

d = ? d = ?

15 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-39
SLIDE 39

Spatio-temporal and hypermedia adaptation

Transgressive adaptation

Ms

e4 e5

Mp

e2 e3

Acropole Agora Temple Mus´ ee

b, EQ m, EQ b, EQ m, EQ mi, EQ b, EQ

Acropole Agora Temple Mus´ ee

m, EQ b, EQ b, EQ b, EQ m, EQ m, EQ

Acropole Agora Temple Mus´ ee

e, DC m, EQ m, DC m, EQ m, DC e, DC

Acropole Agora Temple Mus´ ee

fi, DC m, EQ b, DC b, EQ m, DC fi, DC

d = ? d = ?

15 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-40
SLIDE 40

Spatio-temporal and hypermedia adaptation

Transgressive adaptation

Ms

e4 e5

Mp

e2 e3

Acropole Agora Temple Mus´ ee

b, EQ m, EQ b, EQ m, EQ mi, EQ b, EQ

Acropole Agora Temple Mus´ ee

m, EQ b, EQ b, EQ b, EQ m, EQ m, EQ

Acropole Agora Temple Mus´ ee

e, DC m, EQ m, DC m, EQ m, DC e, DC

Acropole Agora Temple Mus´ ee

fi, DC m, EQ b, DC b, EQ m, DC fi, DC

d = ? d = δ δ = ?

15 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-41
SLIDE 41

Spatio-temporal and hypermedia adaptation

Allen neighborhood graph

before meets

  • verlaps

finished-by starts contains equals during started-by finishes

  • verlapped-by

met-by after

16 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-42
SLIDE 42

Spatio-temporal and hypermedia adaptation

Allen neighborhood graph

before meets

  • verlaps

finished-by starts contains equals during started-by finishes

  • verlapped-by

met-by after

16 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-43
SLIDE 43

Spatio-temporal and hypermedia adaptation

Allen neighborhood graph

before meets

  • verlaps

finished-by starts contains equals during started-by finishes

  • verlapped-by

met-by after

16 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-44
SLIDE 44

Spatio-temporal and hypermedia adaptation

Allen neighborhood graph

before meets

  • verlaps

finished-by starts contains equals during started-by finishes

  • verlapped-by

met-by after

δ(before, overlaps) = 2

16 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-45
SLIDE 45

Spatio-temporal and hypermedia adaptation

RCC8 neighborhood graph

a b a b a b b a a b b a a b a b DC EC PO TPP TPPi NTPP NTPPi EQ

17 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-46
SLIDE 46

Spatio-temporal and hypermedia adaptation

RCC8 neighborhood graph

a b a b a b b a a b b a a b a b DC EC PO TPP TPPi NTPP NTPPi EQ

δ(DC, PO) = 2

17 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-47
SLIDE 47

Spatio-temporal and hypermedia adaptation

Spatio-temporal neighborhood

Definition (Distance between spatio-temporal relations) The distance between spatio-temporal relations is based on the graph product of the temporal and spatial neighborhood graphs. Example (Distance between spatio-temporal relations) Allen neighborhood graph

b m

  • fi

s di e d si f

  • i

mi bi ×

RCC8 neighborhood graph

DC EC PO TPP EQ TPPi NTPP NTPPi 18 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-48
SLIDE 48

Spatio-temporal and hypermedia adaptation

Spatio-temporal neighborhood

Definition (Distance between spatio-temporal relations) The distance between spatio-temporal relations is based on the graph product of the temporal and spatial neighborhood graphs. Example (Distance between spatio-temporal relations)

b DC m DC

  • DC

fi DC s DC di DC e DC d DC si DC f DC

  • i

DC mi DC bi DC b EC m EC

  • EC

fi EC s EC di EC e EC d EC si EC f EC

  • i

EC mi EC bi EC 18 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-49
SLIDE 49

Spatio-temporal and hypermedia adaptation

Spatio-temporal neighborhood

Definition (Distance between spatio-temporal relations) The distance between spatio-temporal relations is based on the graph product of the temporal and spatial neighborhood graphs. Example (Distance between spatio-temporal relations)

b DC m DC

  • DC

fi DC s DC di DC e DC d DC si DC f DC

  • i

DC mi DC bi DC b EC m EC

  • EC

fi EC s EC di EC e EC d EC si EC f EC

  • i

EC mi EC bi EC

δ(di, DC, bi, EC) = 5

18 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-50
SLIDE 50

Spatio-temporal and hypermedia adaptation

Spatio-temporal neighborhood

Property (Distance between spatio-temporal relations) Suppose two spatio-temporal relations r1 = r1

t , r1 s et r2 = r2 t , r2 s and two

neighborhood graphs with one temporal NX

t and the other one spatial NX ′ s .

The conceptual distance δ(r1, r2) is equal to the sum of δ(r1

t , r2 t ) in NX t and

δ(r1

s , r2 s ) in NX ′ s .

Example (Distance between spatio-temporal relations) δ(di, DC, bi, EC) = δ(di, bi) + δ(DC, EC) = 4 + 1 = 5

19 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-51
SLIDE 51

Spatio-temporal and hypermedia adaptation

Computation of adapted solutions

Ms Mp

e2 e3 e4 e5

Acropole Agora Temple Mus´ ee

b, EQ m, EQ b, EQ m, EQ mi, EQ b, EQ

Acropole Agora Temple Mus´ ee

m, EQ b, EQ b, EQ b, EQ m, EQ m, EQ

Acropole Agora Temple Mus´ ee

e, DC m, EQ m, DC m, EQ m, DC e, DC

Acropole Agora Temple Mus´ ee

fi, DC m, EQ b, DC b, EQ m, DC fi, DC

d = 27 d = 27

20 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-52
SLIDE 52

Spatio-temporal and hypermedia adaptation

Computation of adapted solutions

Ms Mp

e2 e3 e4 e5

Acropole Agora Temple Mus´ ee

b, EQ m, EQ b, EQ m, EQ mi, EQ b, EQ

Acropole Agora Temple Mus´ ee

m, EQ b, EQ b, EQ b, EQ m, EQ m, EQ

Acropole Agora Temple Mus´ ee

e, DC m, EQ m, DC m, EQ m, DC e, DC

Acropole Agora Temple Mus´ ee

fi, DC m, EQ b, DC b, EQ m, DC fi, DC

d = 27 d = 25 d = 25

20 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-53
SLIDE 53

Spatio-temporal and hypermedia adaptation

Computation of adapted solutions

Ms Mp

e2 e3 e4 e5

Acropole Agora Temple Mus´ ee

b, EQ m, EQ b, EQ m, EQ mi, EQ b, EQ

Acropole Agora Temple Mus´ ee

m, EQ b, EQ b, EQ b, EQ m, EQ m, EQ

Acropole Agora Temple Mus´ ee

e, DC m, EQ m, DC m, EQ m, DC e, DC

Acropole Agora Temple Mus´ ee

fi, DC m, EQ b, DC b, EQ m, DC fi, DC

d = 25

20 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-54
SLIDE 54

Spatio-temporal and hypermedia adaptation

Computation of adapted solutions

Ms Mp

e2 e3 e4 e5

Acropole Agora Temple Mus´ ee

b, EQ m, EQ b, EQ m, EQ mi, EQ b, EQ

Acropole Agora Temple Mus´ ee

m, EQ b, EQ b, EQ b, EQ m, EQ m, EQ

Acropole Agora Temple Mus´ ee

e, DC m, EQ m, DC m, EQ m, DC e, DC

Acropole Agora Temple Mus´ ee

fi, DC m, EQ b, DC b, EQ m, DC fi, DC

d = 25 d = 21 d = 21

20 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-55
SLIDE 55

Spatio-temporal and hypermedia adaptation

Computation of adapted solutions

Ms Mp

e2 e3 e4 e5

Acropole Agora Temple Mus´ ee

b, EQ m, EQ b, EQ m, EQ mi, EQ b, EQ

Acropole Agora Temple Mus´ ee

m, EQ b, EQ b, EQ b, EQ m, EQ m, EQ

Acropole Agora Temple Mus´ ee

e, DC m, EQ m, DC m, EQ m, DC e, DC

Acropole Agora Temple Mus´ ee

fi, DC m, EQ b, DC b, EQ m, DC fi, DC

d = 25 d = 21 d = 17 d = 17

20 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-56
SLIDE 56

Spatio-temporal and hypermedia adaptation

Computation of adapted solutions

Ms Mp

e2 e3 e4 e5

Acropole Agora Temple Mus´ ee

b, EQ m, EQ b, EQ m, EQ mi, EQ b, EQ

Acropole Agora Temple Mus´ ee

m, EQ b, EQ b, EQ b, EQ m, EQ m, EQ

Acropole Agora Temple Mus´ ee

e, DC m, EQ m, DC m, EQ m, DC e, DC

Acropole Agora Temple Mus´ ee

fi, DC m, EQ b, DC b, EQ m, DC fi, DC

d = 25 d = 17

20 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-57
SLIDE 57

Spatio-temporal and hypermedia adaptation

Computation of adapted solutions

Ms Mp

e2 e3 e4 e5

Acropole Agora Temple Mus´ ee

b, EQ m, EQ b, EQ m, EQ mi, EQ b, EQ

Acropole Agora Temple Mus´ ee

m, EQ b, EQ b, EQ b, EQ m, EQ m, EQ

Acropole Agora Temple Mus´ ee

e, DC m, EQ m, DC m, EQ m, DC e, DC

Acropole Agora Temple Mus´ ee

fi, DC m, EQ b, DC b, EQ m, DC fi, DC

d = 25 d = 17 Adapted solution

20 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-58
SLIDE 58

Spatio-temporal and hypermedia adaptation

Computation of adapted solutions

Ms Mp

e2 e3 e4 e5

Acropole Agora Mus´ ee Temple Acropole Agora Mus´ ee Temple Acropole Agora Mus´ ee Temple Acropole Agora Mus´ ee Temple

d = 17 Adapted solution d = 25

20 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-59
SLIDE 59

Spatio-temporal and hypermedia adaptation

Adaptation algorithm

In : An initial matrix Ii,j and a matrix with possible relations Pi,j. Out : A set of adapted matrix solutions. PathConsistency(P); If P does not contain an empty relation then Select a relation in Pi,j and decomposed Pi,j in r1, . . . , rk; If Pi,j can’t be decomposed then tmp ← d(I, P); If (tmp < Min) then Min ← tmp;S ← {P}; If (tmp = Min) then S ← S ∪ {P}; Otherwise for each rl (1 ≤ l ≤ k) do Pi,j ← rl; If (d(I, P) ≤ Min) then Adaptation(I,P);

21 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-60
SLIDE 60

Spatio-temporal and hypermedia adaptation

Adaptation algorithm

In : An initial matrix Ii,j and a matrix with possible relations Pi,j. Out : A set of adapted matrix solutions. PathConsistency(P); If P does not contain an empty relation then Select a relation in Pi,j and decomposed Pi,j in r1, . . . , rk; If Pi,j can’t be decomposed then tmp ← d(I, P); If (tmp < Min) then Min ← tmp;S ← {P}; If (tmp = Min) then S ← S ∪ {P}; Otherwise for each rl (1 ≤ l ≤ k) do Pi,j ← rl; If (d(I, P) ≤ Min) then Adaptation(I,P);

21 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-61
SLIDE 61

SMIL adaptation

Outline

1

Semantic adaptation

2

Spatio-temporal and hypermedia specification

3

Spatio-temporal and hypermedia adaptation

4

SMIL adaptation

5

Semantic media adaptation

22 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-62
SLIDE 62

SMIL adaptation

General strategy

Initial specification Adapted specification abstract layer Initial document Adapted document multimedia description languages Adaptation Abstraction Instantiation

23 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-63
SLIDE 63

Semantic media adaptation

Outline

1

Semantic adaptation

2

Spatio-temporal and hypermedia specification

3

Spatio-temporal and hypermedia adaptation

4

SMIL adaptation

5

Semantic media adaptation

24 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-64
SLIDE 64

Semantic media adaptation

Adaptation quality

”The basic assumption is that content transformation activities should be provided as non-destructive operations.” ⇒ One approach is to measure the adaptation transformations Different kind of metrics: Measuring the discourse evolution Measuring the composition degradation Measuring the content transformation

25 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-65
SLIDE 65

Semantic media adaptation

Semantic media adaptation

Mona Lisa Image

PNG format 560 x 864

Web

26 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-66
SLIDE 66

Semantic media adaptation

Semantic media adaptation

Mona Lisa Image

PNG format 560 x 864

Web

26 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-67
SLIDE 67

Semantic media adaptation

Semantic media adaptation

Mona Lisa Image

PNG format 560 x 864

Web Profile

Format = {JPG,BMP,3GP} Max resolution = 400 x 600

26 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-68
SLIDE 68

Semantic media adaptation

Semantic media adaptation

Mona Lisa Image

PNG format 560 x 864

Web Mona Lisa Image

JPG format 386 x 600

26 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-69
SLIDE 69

Semantic media adaptation

Software architecture

World Wide Web Descriptions

d3 d4 d5 d1 d2

Media items

m1 m2 m3 m

27 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-70
SLIDE 70

Semantic media adaptation

Software architecture

World Wide Web Descriptions

d3 d4 d5 d1 d2

Media items

m1 m2 m3 m

a b c d Adaptation component

27 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-71
SLIDE 71

Semantic media adaptation

Software architecture

World Wide Web Descriptions

d3 d4 d5 d1 d2

Media items

m1 m2 m3 m

a b c d Adaptation component

mi ,{ d0,. . .,dj}

m m1 m2

d1 height: “864” width: “560” format: PNG d2 title: “Mona Lisa” author: “Da Vinci” d3 date: “17/12/2007” title: “Mona Lisa” format: PNG d4 height: “600” width: “386” format: JPG d5 title: “Mona Lisa” author: “Da Vinci”

a

: Description association

27 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-72
SLIDE 72

Semantic media adaptation

Software architecture

World Wide Web Descriptions

d3 d4 d5 d1 d2

Media items

m1 m2 m3 m

a b c d Adaptation component

mi ,{ d0,. . .,dj} mi ,d′

i

m m1 m2

d′ height: “864” width: “560” format: PNG title: “Mona Lisa” author: “Da Vinci” d′

1

date: “17/12/2007” title: “Mona Lisa” format: PNG d′

2

height: “600” width: “386” format: JPG title: “Mona Lisa” author: “Da Vinci”

b

: Description aggregation

27 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-73
SLIDE 73

Semantic media adaptation

Software architecture

World Wide Web Descriptions

d3 d4 d5 d1 d2

Media items

m1 m2 m3 m

a b c d Adaptation component

mi ,{ d0,. . .,dj} mi ,d′

i

m m1 m2

d′ height: “864” width: “560” format: PNG title: “Mona Lisa” author: “Da Vinci” d′

1

date: “17/12/2007” title: “Mona Lisa” format: PNG d′

2

height: “600” width: “386” format: JPG title: “Mona Lisa” author: “Da Vinci” Sim(d′

k ,d′ l )

0.76 0.31 0.9

c

: Description similarity

27 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-74
SLIDE 74

Semantic media adaptation

Software architecture

World Wide Web Descriptions

d3 d4 d5 d1 d2

Media items

m1 m2 m3 m

profile a b c d Adaptation component

mi ,{ d0,. . .,dj} mi ,d′

i

m m2

d′ height: “864” width: “560” format: PNG title: “Mona Lisa” author: “Da Vinci” d′

2

height: “600” width: “386” format: JPG title: “Mona Lisa” author: “Da Vinci” Sim(d′

k ,d′ l )

0.9

d

: Description selection

27 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-75
SLIDE 75

Semantic media adaptation

Software architecture

World Wide Web Descriptions

d3 d4 d5 d1 d2

Media items

m1 m2 m3 m

profile a b c d Adaptation component d′

  • ut

mi ,{ d0,. . .,dj} mi ,d′

i

m m2

d′ height: “864” width: “560” format: PNG title: “Mona Lisa” author: “Da Vinci” d′

2

height: “600” width: “386” format: JPG title: “Mona Lisa” author: “Da Vinci” Sim(d′

k ,d′ l )

0.9

27 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-76
SLIDE 76

Semantic media adaptation

Media adaptation using Semantic Web technologies

d1 height: “864” width: “560” format: PNG

a

  • Indexing

28 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-77
SLIDE 77

Semantic media adaptation

Media adaptation using Semantic Web technologies

O1

Painting Portrait Abstract Landscape Painting ≡∃ hasPainter hasTitle d1 height: “864” width: “560” format: PNG d2 hasTitle: “Mona Lisa” hasPainter: “Da Vinci”

a

  • Indexing RDF

(+OWL ontologies)

28 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-78
SLIDE 78

Semantic media adaptation

Media adaptation using Semantic Web technologies

O1

Painting Portrait Abstract Landscape Painting ≡∃ hasPainter hasTitle d1 height: “864” width: “560” format: PNG d2 hasTitle: “Mona Lisa” hasPainter: “Da Vinci”

| = type: Painting

a

  • Indexing RDF

(+OWL ontologies)

28 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-79
SLIDE 79

Semantic media adaptation

Media adaptation using Semantic Web technologies

O1

Painting Portrait Abstract Landscape Painting ≡∃ hasPainter hasTitle

O2

ArtWork Drawing MasterPiece Movie Cartoon ⊥ author title d3 type: Cartoon author: “Sato” format: JPG d1 height: “864” width: “560” format: PNG d2 hasTitle: “Mona Lisa” hasPainter: “Da Vinci”

a

  • Indexing RDF

(+OWL ontologies)

28 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-80
SLIDE 80

Semantic media adaptation

Media adaptation using Semantic Web technologies

O1

Painting Portrait Abstract Landscape Painting ≡∃ hasPainter hasTitle

O2

ArtWork Drawing MasterPiece Movie Cartoon ⊥ author title d3 type: Cartoon author: “Sato” format: JPG d4 hasTitle: “La Gioconda” represents: ?X ?X type Woman type: Portrait d5 title: “Mona Lisa” author: “Da Vinci” type: MasterPiece d1 height: “864” width: “560” format: PNG d2 hasTitle: “Mona Lisa” hasPainter: “Da Vinci”

a

  • Indexing RDF

(+OWL ontologies)

28 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-81
SLIDE 81

Semantic media adaptation

Media adaptation using Semantic Web technologies

O1

Painting Portrait Abstract Landscape Painting ≡∃ hasPainter hasTitle

O2

ArtWork Drawing MasterPiece Movie Cartoon ⊥ author title d′

2

type: Cartoon author: “Sato” format: JPG d′

3

hasTitle: “La Gioconda” represents: ?X ?X type Woman type: Portrait title: “Mona Lisa” author: “Da Vinci” type: MasterPiece d′

1 height: “864”

width: “560” format: PNG hasTitle: “Mona Lisa” hasPainter: “Da Vinci”

b

  • RDF Merge

28 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-82
SLIDE 82

Semantic media adaptation

Media adaptation using Semantic Web technologies

O1

Painting Portrait Abstract Landscape Painting ≡∃ hasPainter hasTitle

O2

ArtWork Drawing MasterPiece Movie Cartoon ⊥ author title d′

2

type: Cartoon author: “Sato” format: JPG d′

3

hasTitle: “La Gioconda” represents: ?X ?X type Woman type: Portrait title: “Mona Lisa” author: “Da Vinci” type: MasterPiece d′

1 height: “864”

width: “560” format: PNG hasTitle: “Mona Lisa” hasPainter: “Da Vinci”

c

  • Similarity

? ? ?

28 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-83
SLIDE 83

Semantic media adaptation

Media adaptation using Semantic Web technologies

O1

Painting Portrait Abstract Landscape Painting ≡∃ hasPainter hasTitle

O2

ArtWork Drawing MasterPiece Movie Cartoon ⊥ author title ⊑ ⊑ ≡ d′

2

type: Cartoon author: “Sato” format: JPG d′

3

hasTitle: “La Gioconda” represents: ?X ?X type Woman type: Portrait title: “Mona Lisa” author: “Da Vinci” type: MasterPiece d′

1 height: “864”

width: “560” format: PNG hasTitle: “Mona Lisa” hasPainter: “Da Vinci”

c

  • Semantic similarity

based on

  • ntology matching

0.8 0.2

28 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-84
SLIDE 84

Semantic media adaptation

Media adaptation using Semantic Web technologies

O1

Painting Portrait Abstract Landscape Painting ≡∃ hasPainter hasTitle

O2

ArtWork Drawing MasterPiece Movie Cartoon ⊥ author title ⊑ ⊑ ≡ d′

3

hasTitle: “La Gioconda” represents: ?X ?X type Woman type: Portrait title: “Mona Lisa” author: “Da Vinci” type: MasterPiece d′

1 height: “864”

width: “560” format: PNG hasTitle: “Mona Lisa” hasPainter: “Da Vinci”

d

  • Semantic filtering

(e.g., only masterpieces)

0.8

28 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-85
SLIDE 85

Conclusion

Conclusion

Application of the semantic adaptation approach

Adaptation of all multimedia document dimensions

temporal, spatial and hypermedia spatio-temporal and hypermedia

Development of a SMIL adaptation strategy Implementation of a SMIL adaptation prototype Semantic media adaptation based on Semantic Web technologies

29 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents

slide-86
SLIDE 86

Conclusion

Discussion about the adaptation quality

”The basic assumption is that content transformation activities should be provided as non-destructive operations.” (DocEng’08) ⇒ One approach is to measure the adaptation transformations Different kind of metrics: Measuring the discourse evolution Measuring the composition degradation Measuring the content transformation How is it possible to mix all distances ?

30 S´ ebastien Laborie Semantic Adaptation of Multimedia Documents