C o o r d i n a t i n g S p e e c h D e l i v - - PowerPoint PPT Presentation

SLIDE 1

C

• r

d i n a t i n g S p e e c h D e l i v e r y t

• G

e s t u r e P r

• g

r e s s

T i m

• B

a u m a n n U n i v e r s i t ä t H a m b u r g b a u m a n n @ i n f

• r

m a t i k . u n i

• h

a m b u r g . d e w w w . t i m

• b

a u m a n n . d e / w

• r

k

SLIDE 2

C

• r

d i n a t i n g S p e e c h D e l i v e r y t

• G

e s t u r e P r

• g

r e s s

e . g . f

• r

d e i c t i c e x p r e s s i

• n

s : „ m

• v

e t h i s p i e c e

• v

e r t h e r e t h r

• u

g h t h a t g a t e . “

I m a g e c

• u

r t e s y

• f

V a n d e r b i l t U n i v e r s i t y .

SLIDE 3

C

• r

d i n a t i n g S p e e c h D e l i v e r y t

• G

e s t u r e P r

• g

r e s s

e . g . f

• r

d e i c t i c e x p r e s s i

• n

s : „ m

• v

e t h i s p i e c e

• v

e r t h e r e t h r

• u

g h t h a t g a t e . “

I m a g e c

• u

r t e s y

• f

V a n d e r b i l t U n i v e r s i t y .

P r

• b

l e m s w i t h F i x e d P l a n s

SLIDE 4

C

• r

d i n a t i n g S p e e c h D e l i v e r y t

• G

e s t u r e P r

• g

r e s s

e . g . f

• r

d e i c t i c e x p r e s s i

• n

s : „ m

• v

e t h i s p i e c e

• v

e r t h e r e t h r

• u

g h t h a t g a t e . “

I m a g e c

• u

r t e s y

• f

V a n d e r b i l t U n i v e r s i t y .

P r

• b

l e m s w i t h F i x e d P l a n s

• a

c t u a t

• r

s p e e d d e v i a t i

• n

s

• r

e f l e x e s t

• a

v

• i

d a c

• l

l i s i

• n
• r

e f e r e n t c h a n g e s p

• s

i t i

• n

SLIDE 5

C

• n

v e n t i

• n

a l A p p r

• a

c h : S t

• p

/ R e s u m e S p e e c h

„ m

• v

e . . . . . . . . t h i s p i e c e . . . . . . .

• v

e r . . . t h e r e t h r

• u

g h t h . . a t g a t e . “

• j

u s t p a u s i n g t h e a u d i

• s

t r e a m i s p s y c h

• l

i n g u i s t i c a l l y i m p l a u s i b l e

– e

f f e c t s a r

• u

n d p a u s e s w

• u

l d s t i l l h a v e t

• b

e m

• d

e l l e d

– i

t ' s n

• t

w h a t h u m a n s d

• a

l r e a d y r e q u i r e s d e l i v e r y p r

• g

r e s s i n f

• r

m a t i

• n
• n
• w

a y t

• s

p e e d u p s y n t h e s i s i f g e s t u r e i s a h e a d

• f

t i m e → s i m p l y s t

• p

p i n g / r e s u m i n g d

• e

s n ' t c u t i t

SLIDE 6

C

• r

d i n a t i n g S p e e c h D e l i v e r y t

• G

e s t u r e P r

• g

r e s s

. . . w i t h i n c r e m e n t a l s p e e c h s y n t h e s i s

• m
• r

e f l e x i b l e t h a n s t

• p

p i n g / r e s u m i n g s p e e c h :

– o

n l i n e s p e e c h t e m p

• a

d a p t a t i

• n

s ( s t r e t c h / c

• m

p r e s s )

– c

h a n g e c

• n

t e n t t h a t i s t

• b

e s p

• k

e n ( e . g . c h a n g e a r e f e r e n t )

– r

e f l e x i v e b e h a v i

• u

r , s u c h a s h e s i t a t i

• n

s

– p

r

• v

i d e d e t a i l e d f e e d b a c k

• n

d e l i v e r y p r

• g

r e s s

→ r e l a t i v e l y e a s y t

• p

e r f

• r

m i n t h e I U F r a m e w

• r

k

SLIDE 7

I n c r e m e n t a l P r

• c

e s s i n g i n t h e I U F r a m e w

• r

k

say(move(p, [x, y, z]) ∧ loc(x) ∧ loc(z) ∧ gate(y) ∧ loc(y)) move this

• ver

there through that piece gate move(p,../) ∧ loc(x) loc(z) gate(y) ∧ loc(y) pragmatic plan (conceptualization) syntactic plan/pattern formulation phonemisation motor planning (HSMM parameters) articulation (vocoding) m v u: ð ɪ s p

• d

a t a c

• m

e a s i n c r e m e n t s ( I U s ) , s m a l l e s t u n i t s

• f

i n f

• r

m a t i

• n
• n

a g i v e n l e v e l

• f

a b s t r a c t i

• n

t h a t a r e i n d i v i d u a l l y p r

• c

e s s a b l e

SLIDE 8

I n c r e m e n t a l P r

• c

e s s i n g i n t h e I U F r a m e w

• r

k

say(move(p, [x, y, z]) ∧ loc(x) ∧ loc(z) ∧ gate(y) ∧ loc(y)) move this

• ver

there through that piece gate move(p,../) ∧ loc(x) loc(z) gate(y) ∧ loc(y) pragmatic plan (conceptualization) syntactic plan/pattern formulation phonemisation motor planning (HSMM parameters) articulation (vocoding) m v u: ð ɪ s p

• I

U s a r e i n t e r c

• n

n e c t e d w i t h r e l a t e d I U s ( t h

• s

e t h a t a r e a b

• v

e / b e l

• w

,

• r
• n

t h e s a m e l e v e l )

– I

U s f

• r

m a n e t w

• r

k t h a t r e f l e c t s t h e s y s t e m s t a t e

SLIDE 9

I n c r e m e n t a l P r

• c

e s s i n g i n t h e I U F r a m e w

• r

k

say(move(p, [x, y, z]) ∧ loc(x) ∧ loc(z) ∧ gate(y) ∧ loc(y)) move this

• ver

there through that piece gate move(p,../) ∧ loc(x) loc(z) gate(y) ∧ loc(y) pragmatic plan (conceptualization) syntactic plan/pattern formulation phonemisation motor planning (HSMM parameters) articulation (vocoding) m v u: ð ɪ s p

• t

h e s y s t e m s t a t e i s c h a n g e d b y a d d i n g / r e m

• v

i n g I U s

• I

U s c a n b e m a n a g e d b y p r

• c

e s s

• r

s t h a t r e a c t t

• n

e t w

• r

k u p d a t e s ( a d d / r e v

• k

e / u p d a t e )

SLIDE 10

I n c r e m e n t a l P r

• c

e s s i n g i n t h e I U F r a m e w

• r

k

say(move(p, [x, y, z]) ∧ loc(x) ∧ loc(z) ∧ gate(y) ∧ loc(y)) move this

• ver

there through that piece gate move(p,../) ∧ loc(x) loc(z) gate(y) ∧ loc(y) pragmatic plan (conceptualization) syntactic plan/pattern formulation phonemisation motor planning (HSMM parameters) articulation (vocoding) m v u: ð ɪ s p

• t

h e s y s t e m s t a t e i s c h a n g e d b y a d d i n g / r e m

• v

i n g I U s

• I

U s c a n b e m a n a g e d b y p r

• c

e s s

• r

s t h a t r e a c t t

• n

e t w

• r

k u p d a t e s ( a d d / r e v

• k

e / u p d a t e )

../ just enough lookahead to model co-articulation ../ just enough to keep sound-card bufgers full

SLIDE 11

C

• r

d i n a t i n g S p e e c h D e l i v e r y t

• G

e s t u r e P r

• g

r e s s

→ r e l a t i v e l y e a s y t

• p

e r f

• r

m i n t h e I U F r a m e w

• r

k → r e q u i r e m e n t s f

• r

c

• r

d i n a t i

• n

/ c

• u

p l i n g :

– p

r

• v

i d e s y n c h r

• n

i z a t i

• n

p

• i

n t s b e t w e e n m

• t

i

• n

& s p e e c h

– s

y n c h r

• n

i z a t i

• n

s h

• u

l d b e a v a i l a b l e

• n

v a r i

• u

s l i n g u i s t i c l e v e l s ( p h r a s e s , w

• r

d s , s y l l a b l e s ) a n d

– s

y n c h r

• n

i z a t i

• n

s h

• u

l d i n t e g r a t e w i t h p r

• d

u c t i

• n

c a p a b i l i t i e s ( N L G , p r

• s
• d

y , a r t i c u l a t i

• n

, s y n t h e s i s )

SLIDE 12

I n t e r f a c i n g w i t h E x e c u t i

• n

C

• n

t r

• l
• a

n c h

• r

p

• i

n t s b e t w e e n g e s t u r e & s p e e c h t h a t a r e c

• p

l a n n e d i n a d v a n c e

– e

. g . s t a r t / c e n t e r / e n d

• f

s

• m

e I U

• r
• b
• t

c

• n

t r

• l

m

• n

i t

• r

s g e s t u r e d e l i v e r y a n d n

• t

i f i e s

• f

d e v i a t i

• n

s ( e . g . i n c l u d i n g t

e x p e c t e d

a n d e x p e c t a t i

• n

e r r

• r

)

– n

• t

i f y t h e I U , i t w i l l a u t

• m

a t i c a l l y d e t e r m i n e t h e r e l e v a n t p r

• c

e s s i n g s t e p s

– e

x p e c t a t i

• n

e r r

• r

c

• u

l d b e u s e d e . g . t

• d

e t e r m i n e w h e t h e r s t r e t c h i n g

• r

h e s i t a t i n g s h

• u

l d b e p e r f

• r

m e d

• s

p e e c h d e l i v e r y r e t u r n s h

• w

w e l l i t i s a b l e t

• m

e e t t h e n e w g

• a

l

SLIDE 13

A n e x a m p l e

• s

y n c h r

• n

i z a t i

• n
• n

m

• r

e a b s t r a c t l e v e l l e a v e s m

• r

e f r e e d

• m

( a n d r e s p

• n

s i b i l i t y ) t

• s

p e e c h d e l i v e r y :

– o

n w

• r

d l e v e l :

– o

n c

• n

c e p t l e v e l :

• l

e t s p e e c h d e l i v e r y d e c i d e

• n

t h e b e s t

• p

t i

• n

g i v e n t h e t i m i n g c

• n

s t r a i n t s

through that gate through that gate through that gate gate(y) ∧ loc(y) through the here gate(y) ∧ loc(y) gate

• ver

sync point sync point

SLIDE 14

S u m m a r y

C

• r

d i n a t e d S p e e c h D e l i v e r y i s w

• r

k

• i

n

• p

r

• g

r e s s

• t

e m p

• c

h a n g e s n

• t

y e t a r t i c u l a t

• r

i l l y p l a u s i b l e

– n

e e d t

• d

e t e r m i n e s t r e t c h a b i l i t y i n g i v e n c

• n

t e x t s

• h

e s i t a t i

• n

s a r e a v a i l a b l e ( b u t d

• n
• t

s

• u

n d g r e a t )

• s

i m p l e r e

• g

e n e r a t i

• n

w

• u

l d b e e a s y , b u t i n f l e x i b l e

– t

h

• r
• u

g h r e

• g

e n e r a t i

• n

i s s t i l l a n

• p

e n q u e s t i

• n
• g

e n e r i c i n t e r f a c e w i t h g e s t u r e i s y e t t

• b

e d e t e r m i n e d

– t

h a t ' s w h y I c a m e t

• t

h e w

• r

k s h

• p
• I

d

• n

' t h a v e a r

• b
• t

SLIDE 15

T h a n k y

• u

. b a u m a n n @ i n f

• r

m a t i k . u n i

• h

a m b u r g . d e , g e t t h e c

• d

e a t i n p r

• t

k . s f . n e t .

F u n d e d b y a D a i m l e r a n d B e n z F

• u

n d a t i

• n

P

• s

t D

• c

g r a n t .

SLIDE 16

R a u m f ü r N

• t

i z e n