C o o r d i n a t i n g S p e e c h D e l i v - - PowerPoint PPT Presentation

c o o r d i n a t i n g s p e e c h d e l i v e r y t o g
SMART_READER_LITE
LIVE PREVIEW

C o o r d i n a t i n g S p e e c h D e l i v - - PowerPoint PPT Presentation

C o o r d i n a t i n g S p e e c h D e l i v e r y t o G e s t u r e P r o g r e s s T i m o B a u m a n n U n i v e r s i t t H a m b u r g b a u m a n n @ i n f


slide-1
SLIDE 1

C

  • r

d i n a t i n g S p e e c h D e l i v e r y t

  • G

e s t u r e P r

  • g

r e s s

T i m

  • B

a u m a n n U n i v e r s i t ä t H a m b u r g b a u m a n n @ i n f

  • r

m a t i k . u n i

  • h

a m b u r g . d e w w w . t i m

  • b

a u m a n n . d e / w

  • r

k

slide-2
SLIDE 2

C

  • r

d i n a t i n g S p e e c h D e l i v e r y t

  • G

e s t u r e P r

  • g

r e s s

e . g . f

  • r

d e i c t i c e x p r e s s i

  • n

s : „ m

  • v

e t h i s p i e c e

  • v

e r t h e r e t h r

  • u

g h t h a t g a t e . “

I m a g e c

  • u

r t e s y

  • f

V a n d e r b i l t U n i v e r s i t y .

slide-3
SLIDE 3

C

  • r

d i n a t i n g S p e e c h D e l i v e r y t

  • G

e s t u r e P r

  • g

r e s s

e . g . f

  • r

d e i c t i c e x p r e s s i

  • n

s : „ m

  • v

e t h i s p i e c e

  • v

e r t h e r e t h r

  • u

g h t h a t g a t e . “

I m a g e c

  • u

r t e s y

  • f

V a n d e r b i l t U n i v e r s i t y .

P r

  • b

l e m s w i t h F i x e d P l a n s

slide-4
SLIDE 4

C

  • r

d i n a t i n g S p e e c h D e l i v e r y t

  • G

e s t u r e P r

  • g

r e s s

e . g . f

  • r

d e i c t i c e x p r e s s i

  • n

s : „ m

  • v

e t h i s p i e c e

  • v

e r t h e r e t h r

  • u

g h t h a t g a t e . “

I m a g e c

  • u

r t e s y

  • f

V a n d e r b i l t U n i v e r s i t y .

P r

  • b

l e m s w i t h F i x e d P l a n s

  • a

c t u a t

  • r

s p e e d d e v i a t i

  • n

s

  • r

e f l e x e s t

  • a

v

  • i

d a c

  • l

l i s i

  • n
  • r

e f e r e n t c h a n g e s p

  • s

i t i

  • n
slide-5
SLIDE 5

C

  • n

v e n t i

  • n

a l A p p r

  • a

c h : S t

  • p

/ R e s u m e S p e e c h

„ m

  • v

e . . . . . . . . t h i s p i e c e . . . . . . .

  • v

e r . . . t h e r e t h r

  • u

g h t h . . a t g a t e . “

  • j

u s t p a u s i n g t h e a u d i

  • s

t r e a m i s p s y c h

  • l

i n g u i s t i c a l l y i m p l a u s i b l e

– e

f f e c t s a r

  • u

n d p a u s e s w

  • u

l d s t i l l h a v e t

  • b

e m

  • d

e l l e d

– i

t ' s n

  • t

w h a t h u m a n s d

  • a

l r e a d y r e q u i r e s d e l i v e r y p r

  • g

r e s s i n f

  • r

m a t i

  • n
  • n
  • w

a y t

  • s

p e e d u p s y n t h e s i s i f g e s t u r e i s a h e a d

  • f

t i m e → s i m p l y s t

  • p

p i n g / r e s u m i n g d

  • e

s n ' t c u t i t

slide-6
SLIDE 6

C

  • r

d i n a t i n g S p e e c h D e l i v e r y t

  • G

e s t u r e P r

  • g

r e s s

. . . w i t h i n c r e m e n t a l s p e e c h s y n t h e s i s

  • m
  • r

e f l e x i b l e t h a n s t

  • p

p i n g / r e s u m i n g s p e e c h :

– o

n l i n e s p e e c h t e m p

  • a

d a p t a t i

  • n

s ( s t r e t c h / c

  • m

p r e s s )

– c

h a n g e c

  • n

t e n t t h a t i s t

  • b

e s p

  • k

e n ( e . g . c h a n g e a r e f e r e n t )

– r

e f l e x i v e b e h a v i

  • u

r , s u c h a s h e s i t a t i

  • n

s

– p

r

  • v

i d e d e t a i l e d f e e d b a c k

  • n

d e l i v e r y p r

  • g

r e s s

→ r e l a t i v e l y e a s y t

  • p

e r f

  • r

m i n t h e I U F r a m e w

  • r

k

slide-7
SLIDE 7

I n c r e m e n t a l P r

  • c

e s s i n g i n t h e I U F r a m e w

  • r

k

say(move(p, [x, y, z]) ∧ loc(x) ∧ loc(z) ∧ gate(y) ∧ loc(y)) move this

  • ver

there through that piece gate move(p,../) ∧ loc(x) loc(z) gate(y) ∧ loc(y) pragmatic plan (conceptualization) syntactic plan/pattern formulation phonemisation motor planning (HSMM parameters) articulation (vocoding) m v u: ð ɪ s p

  • d

a t a c

  • m

e a s i n c r e m e n t s ( I U s ) , s m a l l e s t u n i t s

  • f

i n f

  • r

m a t i

  • n
  • n

a g i v e n l e v e l

  • f

a b s t r a c t i

  • n

t h a t a r e i n d i v i d u a l l y p r

  • c

e s s a b l e

slide-8
SLIDE 8

I n c r e m e n t a l P r

  • c

e s s i n g i n t h e I U F r a m e w

  • r

k

say(move(p, [x, y, z]) ∧ loc(x) ∧ loc(z) ∧ gate(y) ∧ loc(y)) move this

  • ver

there through that piece gate move(p,../) ∧ loc(x) loc(z) gate(y) ∧ loc(y) pragmatic plan (conceptualization) syntactic plan/pattern formulation phonemisation motor planning (HSMM parameters) articulation (vocoding) m v u: ð ɪ s p

  • I

U s a r e i n t e r c

  • n

n e c t e d w i t h r e l a t e d I U s ( t h

  • s

e t h a t a r e a b

  • v

e / b e l

  • w

,

  • r
  • n

t h e s a m e l e v e l )

– I

U s f

  • r

m a n e t w

  • r

k t h a t r e f l e c t s t h e s y s t e m s t a t e

slide-9
SLIDE 9

I n c r e m e n t a l P r

  • c

e s s i n g i n t h e I U F r a m e w

  • r

k

say(move(p, [x, y, z]) ∧ loc(x) ∧ loc(z) ∧ gate(y) ∧ loc(y)) move this

  • ver

there through that piece gate move(p,../) ∧ loc(x) loc(z) gate(y) ∧ loc(y) pragmatic plan (conceptualization) syntactic plan/pattern formulation phonemisation motor planning (HSMM parameters) articulation (vocoding) m v u: ð ɪ s p

  • t

h e s y s t e m s t a t e i s c h a n g e d b y a d d i n g / r e m

  • v

i n g I U s

  • I

U s c a n b e m a n a g e d b y p r

  • c

e s s

  • r

s t h a t r e a c t t

  • n

e t w

  • r

k u p d a t e s ( a d d / r e v

  • k

e / u p d a t e )

slide-10
SLIDE 10

I n c r e m e n t a l P r

  • c

e s s i n g i n t h e I U F r a m e w

  • r

k

say(move(p, [x, y, z]) ∧ loc(x) ∧ loc(z) ∧ gate(y) ∧ loc(y)) move this

  • ver

there through that piece gate move(p,../) ∧ loc(x) loc(z) gate(y) ∧ loc(y) pragmatic plan (conceptualization) syntactic plan/pattern formulation phonemisation motor planning (HSMM parameters) articulation (vocoding) m v u: ð ɪ s p

  • t

h e s y s t e m s t a t e i s c h a n g e d b y a d d i n g / r e m

  • v

i n g I U s

  • I

U s c a n b e m a n a g e d b y p r

  • c

e s s

  • r

s t h a t r e a c t t

  • n

e t w

  • r

k u p d a t e s ( a d d / r e v

  • k

e / u p d a t e )

../ just enough lookahead to model co-articulation ../ just enough to keep sound-card bufgers full

slide-11
SLIDE 11

C

  • r

d i n a t i n g S p e e c h D e l i v e r y t

  • G

e s t u r e P r

  • g

r e s s

→ r e l a t i v e l y e a s y t

  • p

e r f

  • r

m i n t h e I U F r a m e w

  • r

k → r e q u i r e m e n t s f

  • r

c

  • r

d i n a t i

  • n

/ c

  • u

p l i n g :

– p

r

  • v

i d e s y n c h r

  • n

i z a t i

  • n

p

  • i

n t s b e t w e e n m

  • t

i

  • n

& s p e e c h

– s

y n c h r

  • n

i z a t i

  • n

s h

  • u

l d b e a v a i l a b l e

  • n

v a r i

  • u

s l i n g u i s t i c l e v e l s ( p h r a s e s , w

  • r

d s , s y l l a b l e s ) a n d

– s

y n c h r

  • n

i z a t i

  • n

s h

  • u

l d i n t e g r a t e w i t h p r

  • d

u c t i

  • n

c a p a b i l i t i e s ( N L G , p r

  • s
  • d

y , a r t i c u l a t i

  • n

, s y n t h e s i s )

slide-12
SLIDE 12

I n t e r f a c i n g w i t h E x e c u t i

  • n

C

  • n

t r

  • l
  • a

n c h

  • r

p

  • i

n t s b e t w e e n g e s t u r e & s p e e c h t h a t a r e c

  • p

l a n n e d i n a d v a n c e

– e

. g . s t a r t / c e n t e r / e n d

  • f

s

  • m

e I U

  • r
  • b
  • t

c

  • n

t r

  • l

m

  • n

i t

  • r

s g e s t u r e d e l i v e r y a n d n

  • t

i f i e s

  • f

d e v i a t i

  • n

s ( e . g . i n c l u d i n g t

e x p e c t e d

a n d e x p e c t a t i

  • n

e r r

  • r

)

– n

  • t

i f y t h e I U , i t w i l l a u t

  • m

a t i c a l l y d e t e r m i n e t h e r e l e v a n t p r

  • c

e s s i n g s t e p s

– e

x p e c t a t i

  • n

e r r

  • r

c

  • u

l d b e u s e d e . g . t

  • d

e t e r m i n e w h e t h e r s t r e t c h i n g

  • r

h e s i t a t i n g s h

  • u

l d b e p e r f

  • r

m e d

  • s

p e e c h d e l i v e r y r e t u r n s h

  • w

w e l l i t i s a b l e t

  • m

e e t t h e n e w g

  • a

l

slide-13
SLIDE 13

A n e x a m p l e

  • s

y n c h r

  • n

i z a t i

  • n
  • n

m

  • r

e a b s t r a c t l e v e l l e a v e s m

  • r

e f r e e d

  • m

( a n d r e s p

  • n

s i b i l i t y ) t

  • s

p e e c h d e l i v e r y :

– o

n w

  • r

d l e v e l :

– o

n c

  • n

c e p t l e v e l :

  • l

e t s p e e c h d e l i v e r y d e c i d e

  • n

t h e b e s t

  • p

t i

  • n

g i v e n t h e t i m i n g c

  • n

s t r a i n t s

through that gate through that gate through that gate gate(y) ∧ loc(y) through the here gate(y) ∧ loc(y) gate

  • ver

sync point sync point

slide-14
SLIDE 14

S u m m a r y

C

  • r

d i n a t e d S p e e c h D e l i v e r y i s w

  • r

k

  • i

n

  • p

r

  • g

r e s s

  • t

e m p

  • c

h a n g e s n

  • t

y e t a r t i c u l a t

  • r

i l l y p l a u s i b l e

– n

e e d t

  • d

e t e r m i n e s t r e t c h a b i l i t y i n g i v e n c

  • n

t e x t s

  • h

e s i t a t i

  • n

s a r e a v a i l a b l e ( b u t d

  • n
  • t

s

  • u

n d g r e a t )

  • s

i m p l e r e

  • g

e n e r a t i

  • n

w

  • u

l d b e e a s y , b u t i n f l e x i b l e

– t

h

  • r
  • u

g h r e

  • g

e n e r a t i

  • n

i s s t i l l a n

  • p

e n q u e s t i

  • n
  • g

e n e r i c i n t e r f a c e w i t h g e s t u r e i s y e t t

  • b

e d e t e r m i n e d

– t

h a t ' s w h y I c a m e t

  • t

h e w

  • r

k s h

  • p
  • I

d

  • n

' t h a v e a r

  • b
  • t
slide-15
SLIDE 15

T h a n k y

  • u

. b a u m a n n @ i n f

  • r

m a t i k . u n i

  • h

a m b u r g . d e , g e t t h e c

  • d

e a t i n p r

  • t

k . s f . n e t .

F u n d e d b y a D a i m l e r a n d B e n z F

  • u

n d a t i

  • n

P

  • s

t D

  • c

g r a n t .

slide-16
SLIDE 16

R a u m f ü r N

  • t

i z e n