[PDF] - Intelligent Multimedia Presentation Systems: A proposal of reference PDF Document

SLIDE 1

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/2304850

Intelligent Multimedia Presentation Systems: A proposal of reference model

Article · October 1999

Source: CiteSeer

CITATION

1

READS

19 6 authors, including: Some of the authors of this publication are also working on these related projects: Designing Multisensory Product Experience Through Virtual Reality Technologies View project Olfaction and olfactory display View project Monica Bordegoni Politecnico di Milano

320 PUBLICATIONS 1,887 CITATIONS

SEE PROFILE

Thomas Rist University of Applied Sciences Augsburg

183 PUBLICATIONS 3,686 CITATIONS

SEE PROFILE

Panos E. Trahanias Foundation for Research and Technology - Hellas

212 PUBLICATIONS 3,660 CITATIONS

SEE PROFILE

All content following this page was uploaded by Panos E. Trahanias on 17 August 2014.

The user has requested enhancement of the downloaded file.

SLIDE 2 INTELLIGENT MUL TIMEDIA PRESENT A TION SYSTEMS A PR OPOSAL F OR A REFERENCE MODEL M BORDEGONI G F A CONTI T RIST CNR ITIA CNR CNUCE DFKI Milano Italy Pisa Italy Saarbr ucken Germany S R UGGIERI P

TRAHANIAS

M WILSON Universit a di F OR THICS CRLCRAL Pisa Italy Her aklion Cr ete Gr e e c e Chilton Didc

t

UK The dev elopmen t

f

the so called intel ligent multime dia pr esentation systems IMMPSs is curren tly v ery activ ely addressed b y researc h groups w

rldwide

A common goal

f

the researc h comm unit y is to dev elop mec hanisms for the au tomated generation

f

m ultimedia presen tatio ns Up to no w some largesized protot yp es

f

IMMPSs ha v e b een built and user in terfaces

f

some applicatio ns already include automated comp

nen

ts for certain generation tasks suc h as text

r

graphics generation

Unfortunately
there

is no common agreemen t

n

a generic ar c hitecture for IMMPS with clear functional denitions

f

in v

lv

ed sub comp

n

en t s Moreo v er ev en the terminology used in the descriptions

f

existing IMMPSs v aries considerably across researc h teams With the prop

sal
f

a reference mo del for IMMPSs this pap er tries to ll a ma jor metho dolog ica l gap and th us ma y pro vide a sound basis for

ngoing

and future dev elopmen t s

f

IMMPSs In essence the prop

sed

reference mo del consists

f

sev eral la y ers referring to the particu lar subtasks whic h

ccur

in m ultimedia presen tation generation F

llo

wing the paradigm

f

kno wledgebased computing w e in tro duce a minim um set

f

explic itly enco ded kno wledge and assign it to some logically distinct kno wledge sources These sources allo w to share kno wledge among comp

nen

ts

f

dieren t la y ers in a clien tserv er fashion In

rder

to demonstra te the use

f

the reference mo del w e pro vide a comparison

f

t w

IMMPS

b y redescribing them in terms

f

the mo del

In

tro ducti

n

Observing an increasing n um b er

f

applications where the man ual author ing

f

presen tations is no longer feasible the dev elopmen t

f

mec hanisms for the automated generation

f

m ultim edia presen tations has b ecome a shared goal across man y disciplines Since the use

f

m ultiple media for con v eying information do es not guaran tee eectiv e and in telligible presen tations p er se these mec hanisms need to b e intel ligent in the sense that they are able to dra w appropriate design decisions Up to no w a lot

f

researc h and dev elop men t w

rk

has b een conducted addressing asp ects

f

automating m ultim edia presen tation generation and ev en some largesize protot yp es

f

IMMPSs ha v e b een dev elop ed eg MMI

WIP
MIPS

COMET

Ho

w ev er no generic mo del has emerged so far Eac h pro ject b egan from scratc h relying

nly
n

SLIDE 3 the past exp erience

f

the dev elop ers Th us it is not surprising that there is no agreemen t

n

the terminology to b e used

n

the functional denition

f

an IMMPS

r
n

a generic arc hitecture whic h reects the logical structure

f

pro cesses for the generation

f

m ultim edia presen tations With the prop

sal

for a reference mo del for IMMPSs this pap er tries to ll a ma jor metho dological gap and th us ma y pro vide a sound basis for

ngoing

and future dev elopmen ts

f

IMMPSs The agreemen t

n

a r efer enc e mo del in the eld will ha v e sev eral adv an tages First

f

all there is a general motiv ation for ha ving a reference mo del for an y class

f

related computing systems Among

ther

things a reference mo del will help to analyze and compare existing sys tems

f

a certain class

n

the basis

f

a common generic arc hitecture and b y means

f

a common terminology

Moreo

v er the generic arc hitecture b eing the core

f

a reference mo del will foster the mo dular dev elopmen t

f

future large size systems as eac h mo dule can b e assigned a w ell dened role in terms

f

the reference mo del with w ell dened in terfaces b et w een system comp

nen

ts and to

ther

systems Our reference mo del is targeted to w ards the class

f

systems

r

comp

nen

ts

f

sup erordinate systems whose task is to presen t information to the user in an ee ctive w a y

Hereb

y

the

attribute ee ctive means that the particular information needs

f

the individual users are b est met under a giv en set

f

presen tation constrain ts suc h as resource limitatio ns and the users kno wledge and st yle preferences Since in the v ast ma jorit y

f

nontrivial applications the information needs will v ary from user to user and from situation to situation a presen tation system should b e able to exibly generate v arious presen tations for

ne

and the same information con ten t to b e comm unicated Ha ving iden ti ed the class

f

systems to b e captured b y the mo del lets turn

ur

atten tion at the general guidelines that ha v e driv en

ur

design decisions

A

de quate mo dularisation T

facilitate

the dev elopmen t and compari son

f

practical largesize systems the reference mo del m ust comprise a mo dularisation

f

a generic pro cess for m ultimedia presen tation gener ation Idealy

this

mo dularization breaks do wn the generation pro cess in to logically distinct and computationally feasible subtasks

Appr
priate

de gr e e

f

abstr action The reference mo del should

n

the

ne

hand reect the p eculiarities

f

m ultim edia generation On the

ther

hand it should b e general enough to capture the whole class

f

IMMPSs Certainly

the

generic arc hitecture should abstract from concrete imple men tations as it is alw a ys p

ssible

to rely

n

dieren t mec hanisms to accomplish a single generation subtask and dieren t formats for repre sen ting kno wledge are alw a ys a c hoice

SLIDE 4

Identic

ation and classic ation

f

know le dge sour c es As in telligen t pre sen tation design is a kno wledge in tensiv e task the reference mo del should exhibit the basic set

f

logically distinct know le dge sour c es whic h are required for m ultimedia presen tation generation The reference mo del should also mak e clear ho w pro cesses and kno wledge sources are related to eac h

ther

In particular private kno wledge sources ie sources for whic h a single

wner

comp

nen

t can b e lo cated should b e distinguished from those sources whic h are shar e d among sev eral comp

nen

ts

Mo

deling

f

shar e d sour c es in the clientserver p ar adigm T

facilitate

sharing

f

kno wledge sources in the generic arc hitecture as w ell as in concrete system implem en tations the latter should b e mo delled follo wing the clien tserv er paradigm Suc h sources will b e referred to as exp ert mo dules

they

are deemed to serv e requests from clien t mo dules p

ssibly

b elonging to

ther

systems

Op

eness to

ther

standar ds Multimedia generation comprises subtasks whic h ha v e b een dealt with in

ther

disciplines Therefore the mo del should b e

p

en to com bine with existing

r

p

ten

tial standards in these disciplines F

r

example the Computer Gr aphics R efer enc e Mo del

ma

y b e used to instan tiate the sub comp

nen

t for graphics generation in the generic arc hitecture

f
ur

mo del Existing reference mo dels for h yp er media presen tations suc h as the Dexter mo del

r

the AHM Amster dam Hyp ermedia Mo del

ma

y b e used for the description

f

gener ated presen tation sp ecications Also

ne

ma y rely

n

a standardized language for the exc hange

f

kno wledge b et w een comp

nen

ts suc h as K QML Know le dge Query and Manipulation L anguage

Vice

v ersa the reference mo del can itself b ecome a comp

nen

t

f

a sup erordinate mo del suc h as the w ell kno wn Seeheim Mo del

whic

h has b een prop

sed

as a generic arc hitecture for user in terfaces The rest

f

the pap er is

rganized

as follo ws First some basic notions are in tro duced in

rder

to c haracterize the class

f

intel ligent multime dia pr esen tation systems

The

core

f

the mo del whic h is a generic arc hitecture

f

a generation pro cess for m ultim edia presen tations is giv en in section

T
demonstrate

the use

f

the mo del t w

existing

IMMPSs are redescrib ed with the prop

sed

terminology and arc hitecture in section

Basic

Notions Presen tation systems are designed for ac hieving goals b y means

f

presen tations that are p erceiv able and consequen tly

sub

ject to b e in terpreted b y

SLIDE 5 their in tended user who is alw a ys assumed to b e a h uman b eing Presen tation goals whic h ha v e to b e ac hiev ed b y the system p

ssibly

ac companied b y a set

f

pr esentational c

mmands
r

pr esentation c

nstr

aints aecting the presen tation pro cess constitute the primary input to the pre sen tation system Both goals and commands are assumed to b e form ulated

utside

the presen tation system ie b y the user

r

b y an external system

r

b y a sup erordinate comp

nen

t in case the presen tation system is a part

f

a larger system Goals and commands include a high lev el reference to collec tions

f

data together with the purp

se
r

the in ten tion for comm unicating information As an example a goal can b e form ulated as an enco ding

f

the fact that the system has to inform the user

f

the lo cation

f

a sp ecic switc h in a con trol panel Similarly

a

presen tational command can b e a represen ta tion

f

design constrain ts suc h as the minim um size

f

the switc h for b eing p erceiv able if graphics is c hosen as the presen tation media Application datakno wledge pro vides the seman tical grounding

f

eac h presen tation whic h ma y b e generated b y a system As with presen tation goals and commands it is assumed that application datakno wledge is part

f

the input to a presen tation system In

ther

w

rds

there m ust b e an external source eg an application system

r

a database that mak es a v ailable to the presen tation system the application data necessary for ac hieving p

sted

goals F

llo

wing the example ab

v

e the application can pro vide the switc h and the con trol panel geometries required to iden tify the switc h p

sition

Ho w ev er the request

f

informing the user

f

the switc h p

sition

could ha v e b een form ulated b y a dieren t system A m ultim edi a presen tatio n is a presen tation whic h comprises material in dieren t media suc h as text graphics sound video etc Ho w ev er it cannot b e denied that there is a lot

f

confusion ab

ut

the fundamen tal notion

f

me dia

One

reason

f

this misunderstanding is certainly due to the fact that the term media is used with dieren t meanings in dieren t con texts suc h as in semiotics psyc hology

telecomm

unications

r

computer science The closely related term mo dality is a further source

f

confusion Some authors seem to use b

th

terms as synon yms while

thers

tend to reserv e mo dalit y for input

nly
and

vice v ersa media for

utput

Again

thers

try to assign these terms to dieren t categories eg medium for the system used to con v ey a piece

f

in formation whereas mo dalit y denotes the w a y in whic h a presen tation is sensed Unfortunately

to

the b est

f
ur

kno wledge w e are not a w are

f

an y attempt so far that clearly distinguishes b

th

concepts from eac h

ther

without b ecom ing inconsisten t when generalizing the denition to co v er all p

ssible

media whic h ma y

ccur

in a presen tation T

v

ercome this problem w e

nly

emplo y

SLIDE 6 the term medium in

ur

mo del W e follo w

and

regard an y single mec hanism b y whic h to express information as a me dium

Consequen

tly

multime

dia is an adjectiv e referring to the use

f

m ultiple media A m ultimed ia presen tatio n system in that sense is a presen tation system that starts from a giv en presen tation goal as triggering input and generates a m ultim edia presen tation as

utput

In tellig en t m ultimed ia presen tati

n

systems are essen tially kno wledge based systems ie systems whic h rely

n

the notion

f

know le dge

as

the justied true b eliefs

f

an abstraction

f

data in

rder

to dra w appropriate design decisions The kno wledge presen t in an IMMPS ma y include a n um b er

f

know le dge sour c es dicult to explicitly quan tify

Ho

w ev er starting from the exp erience

f

previous systems the follo wing kno wledge sources seem to b e indisp ensable i kno wledge to reason ab

ut

application data suc h as term in terpretation and data c haracterization ii the discourse mo del and the con text referencies iii the users goals and plans capabilities attitudes kno wledge

r

b eliefs iv general kno wledge ab

ut

the design

f

m ultim edia presen tations in terms

f

design constrain ts cognitiv e theories media mo dels and device mo dels together with the design realization for a sp ecic media in a m ultim edia presen tation the systems c haracteristics relev an t to the presen tation pro cess and the kno wledge for iden tifying an eectiv e and co

rdinated

collection

f

media for the represen tation and presen tation

f

application data The kno wledge presen t in an IMMPS ma y include and

ften

this is the case

ther

explicitly enco ded kno wledge sources than those describ ed Ho w ev er as a w

rking

denition w e will classify a m ultimedia presen tation system as an intel ligent system if it exploits at le ast the ab

v

e men tioned kno wledge sources to ac hiev e presen tation goals

Outline
f

the Reference Mo del The core

f

the reference mo del is a generic arc hitecture for IMMPSs The conceptual design

f

this arc hitecture reects a mo dularisation

f

the design pro cess in to layers whic h are resp

nsible

for particular subtasks and a sep aration

f

shared kno wledge sources called exp erts from the la y ers Fig

in

tro duces the IMMPS reference arc hitecture It is comp

sed
f

four la y ers

SLIDE 7 namely c

ntr
l

c

ntent

layout and pr esentation

In

addition to their priv ate kno wledge these la y ers ma y exploit explicitly enco ded kno wledge pro vided b y a kno wledge serv er whic h is comp

sed
f

the applic ation

c
ntext
user

and de sign exp ert mo dules The system receiv es goals and presen tational commands from the go al formulation

designs

a presen tation asking the applic ation for application kno wledge and nally comm uni cates the presen tation to the user F urthermore a know le dge exchange format ie a language and a proto col is used for the exc hange

f

information among the system comp

nen

ts Ho w ev er this format is not explicitly dealt with here It is

nly

assumed that it underlies eac h arro w b et w een comm unicating comp

nen

ts

Goal Application User Control Content Layout Presentation Knowledge Server

Application Expert Context Expert User Expert

External servers

Design Expert

External clients formulation Layer Layer Layer Layer

Fig

Generic

IMMPS Arc hitecture

Goal Formulation User Notifications Presentation Layer Layout Layer Control Layer Content Layer Notifications Notifications Notifications Goals

Present. commands

Goals

Present. commands

Media comm. acts

Present. commands

Unrendered pres.

Present. commands

Presentations

Fig

Information

Flo w through La y ers Although the go al formulation and the applic ation pla y dieren t roles in most real systems they app ear as a single comp

nen

t usually called the applic a tion

Our

distinction emphasizes the dierence b et w een the availability

f

application kno wledgedata and the use

f

that kno wledgedata to satisfy presen tation goals Also it is necessary to mo del some in teractions

f

the system with external en tities This is the case for example when the user exp ert acquires kno wledge directly from the user

r

from an input subsystem In the gures in terac tions with unsp ecied systems called external serversclients

are

denoted b y dashed arro ws The mo deling could b e done b y means

f

data c aptur e metales as in the Computer Graphics Reference Mo del

SLIDE 8

L

ayers The four la y ers

f

the reference arc hitecture are collectiv ely resp

nsible

for re alizing the goal b eing requested Eac h la y er p erforms a transformation

n

its input and deliv ers the result to the next la y er in the hierarc h y as illustrated in Fig

The

input to the presen tation generation pro cess are goalscomm ands While b eing pro cessed through the la y ers goalscomma nds are ev en tually transformed in to m ultim edia presen tations Although the

w
f

information during this transformation is primary topdo wn it m ust b e stressed that man y in teractions can

ccur

b et w een comp

nen

ts This is indicated b y the notication arro ws in Fig

Eac

h la y er noties to the la y er ab

v

e the result

f

its pro cessing p

ssibly

together with some additional information suc h as causes

f

failure reco v ery strategies explanations etc The individual la y ers are describ ed in the follo wing paragraphs Con trol La y er The con trol la y er consisten tly co

rdinates

the presen tation pro cess in time Its task is to c ho

se

the next goal to b e ac hiev ed

r

the next presen tational command to b e executed The task

f

goal selection ma y

ccur

in m ultim edia generation for t w

dieren

t reasons First it migh t b e the case that the ex ternal goal form ulator p

ses

a set

f

goals to the presen tation system either all at

nce
r

in an undetermined piecemeal fashion In the latter case an already started generation pro cess ma y b e in terrupted in

rder

to ac hiev e a new incoming goal immediately

The

second reason is that presen tation goals giv en as input to the system ma y b e complex so that they ha v e to b e split in to sets

f

less complex goals While the decomp

sition
f

goals will b e done b y the go al r enement mo dule

f

the c

ntent

layer the con trol la y er

ugh

t still to decide in whic h

rder

the subgoals will actually b e pro cessed Deciding the next goal to b e ac hiev ed can b e a v ery simple task It could in v

lv

e

nly

p

pping

a goal from the discourse mo del in the con text exp ert On the

ther

hand some priv ate kno wledge ma y b e presen t in the con trol manager if the decision in v

lv

es more complex reasoning The role

f

the GF Interfac e mo d ule is to con v ert the messages to b e exc hanged b et w een goal form ulation and con trol manager in the appropriate format This task is not carried

ut

b y the underlying kno wledge exc hange format since the transformation dep ends

n

the sp ecic represen tation languages used in the IMMPS Con ten t La y er The con ten t la y er cf Fig

serv

es to determine a set

f

so called me dia c

m

munic ation acts These are comm unicatio n acts enric hed b y seman ticlogic

SLIDE 9 con ten t they are assigned to a particular medium whic h should b e used to con v ey that con ten t Comm uni cation acts are an extension

f

sp eec h acts

to

m ultimedia comm unicatio ns see also

The

task is accomplished b y means

f

co

rdinated

goal renemen t con ten t selection and media selection As presen tation goals ma y b e form ulated at a high lev el

f

abstraction they need to b e rened accordingly

The

term go al r enement is used to capture b

th

the de c

mp
sition
f

a goal in to a set

f

subgoals and the sp e cialization

f

a goal During goal renemen t the con ten t

f

the nal presen tation will b e determined The mo dule for c

ntent

sele ction assists in carrying

ut

this task In a concrete system this mo dule ma y app ear as a retriev al and lter comp

nen

t whic h comm unicates with the application exp ert The

utput
f

the goal renemen t pro cess is a set

f

c

mmunic

ation acts and a structural description

f

the relations that ma y hold b et w een these acts As so

n

as comm unicativ e acts ha v e b een w

rk

ed

ut

it m ust b e decided whic h media shall b e emplo y ed to con v ey them b est F

r

this task the reference arc hitecture includes a com p

nen

t for me dia sele ction

Since

there are man y dep endencies among c hoices the arc hitecture also foresees a c

r

dination mo dule Its mission is to merge comm unication acts passed b y the mo dules for con ten t and media selection In case

f

successful co

rdination

the acts no w called media comm unication acts are handed

v

er to to the la y

ut

la y er The co

rdination

pro cess can ho w ev er require negotiations among goal renemen t con ten tmedia selection and the subsequen t la y

ut

la y er Consider the switc h example where the min im um size sp ecied for the switc h to b e p erceiv ed migh t b e to

big

for the con trol b

ard

to t in the indicated windo w

Goal Formulation GF Interface Knowledge server Control Manager Content Layer

Fig

Con

trol La y er

Knowledge server Layout Layer Control layer Media Goal Selection Selection Content Coordination Refinement

Fig

Con

ten t La y er

SLIDE 10 La y

ut

La y er In this la y er the media comm uni cation acts are transformed in to a pr esentation layout The la y

ut

la y er is comp

sed
f

the layout manager

the

me dia design and r e alization and the c

r

dination cf Fig

The

task

f

the rst comp

nen

t is to design the general la y

ut

structure

f

the presen tation together with dispatc hing

f

the media comm uni cation acts to the relev an t media design and realization mo dules and notifying the con ten t la y er The result

f

single media pro cessing is co

rdinated

to the end

f

generating an unrendered presen tation to b e passed to the presen tation la y er F rom this the la y

ut

la y er receiv es notications Finally

it

sends notication to the con ten t la y er to rep

rt

the result

f

elab

rating

media comm unicativ e acts The determination

f

a presen tation la y

ut

can incur v arious negotiations among the in v

lv

ed media sp ecic generators As men tioned ab

v

e a ne gotiation pro cess can also

ccur

b et w een comp

nen

ts

f

dieren t la y ers F

r

example supp

se

for the purp

se
f

determining the con ten ts

f

a crossmedia reference expression the con ten t la y er has to query the la y

ut

manager at whic h part

f

the displa y space a certain picture will

ccur

It should b e noted that the unrendered presen tation cannot b e said to b e an in ternal represen tation

f

a presen tation but

nly
f

a piece

f

it since the layout layer has a restricted view

n

the presen tation pro cess and ma y b e ask ed to ac hiev e goals whic h are less complex than the

v

erall

ne

On the

ther

hand the

uput
f

this la y er en tails all the information required to run the presen tation prop erly

T
represen

t this information

ne

could rely

n

a sp ecication language as prop

sed

in

Content Layer

PresentationLayer ......... Design Layout Manager Knowledge Server Video Design Graphics Design Text Design Audio Design ........... Video Graphics Text Realization Realization Realization Realization Layout Coordination Realization Audio

Media Realization Media Design

Fig

La

y

ut

La y er

SLIDE 11 Presen tation La y er While the

utput
f

the la y

ut

la y er is a unrendered presen tation there m ust b e a further comp

nen

t whic h tak es this represen tation as input and con v erts it in to a presen tation p erceiv able b y the user This part

f

a presen tation sys tem is

ften

called presen tation displa y comp

nen

t

r

presen tation run time en vironmen t The pr esentation layer

f

the curren t reference mo del captures the functionalities

f

suc h comp

nen

ts A c

r

dination mo dule dispatc hes eac h part

f

the unrendered presen tation to the suitable device in terfaces The result presen ted to the user is the spatially and temp

rally

c

r

dinate d fusion

f

these

utputs
Know

le dge Server The Know le dge Server sho wn in Fig

pro

vides the la y ers

f

the reference ar c hitecture with sev eral t yp es

f

kno wledge It consists

f

four exp ert mo dules designed along the lines

f

kno wledge bases Eac h

f

them represen ts kno wl edge

n

a particular asp ect

f

the presen tation pro cess Applic ation

user
c
ntext

and design

They

all share the same general structure The exp erts are accessed b y

ther

comp

nen

ts in a clien tserv er fashion Requests will b e replied taking in to accoun t the user mo del and the con text Exp erts ma y re quire and vice v ersa pro vide services fromto external kno wledge sources suc h as the application and the user himself In this sense they are indep enden t mo dules easily in tegrable with

ther

systems whic h share the same kno wledge Exp ert mo dules The

v

erall structure

n

an exp ert mo dule is indep enden t from the particular class

f

systems w e are describing In principle they dier

nly

in the kno wl edge whic h they store and in the in terface

p

erations they pro vide Fig

sho

ws the

v

erall logical structure

f

a generic exp ert mo dule The core

f

the exp ert mo dule is the kno wledge it stores This kno wledge can consist

f

a n um b er

f

logically distinct kno wledge bases eac h for a logi cally dieren t asp ect to b e dealt with

r
nly

a single kno wledge base when suc h distinction is not relev an t The infer enc e engine pro vides a uniform and general view

f

the stored kno wledge and

f

that inferable from it Main te nance in v

lv

es incorp

rating

new kno wledge whic h could b e inconsisten t with the curren t state If this is the case inconsistency m ust b e resolv ed b y some triggered action p erformed b y an inte grity che cking submo dule The three comp

nen

ts know le dge b ase

infer

enc e engine and inte grity che cking

are

re ferred to as the Know le dge Base System KBS

f

the exp ert The exp erts pro vide services to the

ther

comp

nen

ts

f

the system through

SLIDE 12 the server manager Its clien ts are the la y ers

f

the presen tation system

ther

exp erts

r

external en tities namely the external sources

r

the application The serv er manager transforms the in terface

p

erations in to a collection

f
messages

to the inference engine collecting the answ ers and resp

nding

to the clien t Tw

in

terfaces c

ntext

exp ert and user exp ert

allo

w the inference engine to acquire kno wledge resp ectiv ely from the user exp ert and the con text exp ert Finally

the

ac quisition in terface supp

rts

access to

ther

undened serv ers This is necessary when the kno wledge

f

the exp ert dep ends

n
ther

factors in addition to user and con text As an example the application exp ert has to in teract with the application to request application data

Manager Inference Engine User Expert User Expert KB . . . . Acquisition Interface Integrity Checking KB Context Expert Clients External knowledge sources Interface

KBS

Context Expert Interface Server

Fig

General

structure

f

an exp ert mo dule It is

ften

useful and in tuitiv e to ha v e a logically structured view

f

a kno wl edge base This is the reason

f

the presence

f

sever al kno wledge bases Real systems are not required to satisfy this logical distinction in an implemen ta tion F

r

instance the implemen tation could com bine all kno wledge in a single kno wledge base

r

structure it under

ther

p

in

ts

f

view The arc hitecture

f

an exp ert mo dule is not directly constrained b y the class

f

IMMPSs This means that the notion

f

exp ert mo dule is generic and suitable for

ther

applications as w ell Although it will b e constrained to a certain ex ten t when instan tiating clien ts and external kno wledge sources it still remains general enough to allo w for in teractions with unsp ecied serv ersclien ts This is actually the role that the external serversclients pla y in

ur

arc hitecture They are supp

sed

to mo del in teractions with unsp ecied systems This al lo ws to merge the user exp ert with a functionally equiv alen t mo dule

f

another application

for

instance the input system The table b elo w summarizes

ur

instan tiation

f

the know le dge server

SLIDE 13 Modules

f

the Kno wledge Ser ver Exp ert Ext Know Sour c es Clients Know le dge Application Application La y ers Application kno wledge External serv ers External clien ts Con text External serv ers La y ers OtherExp erts Con text mo del External clien ts User External serv ers La y ers OtherExp erts User mo del External clien ts Design External serv ers La y ers Design kno wledge External clien ts

Using

the Reference Mo del In this section w e demonstrate ho w the reference mo del can b e used to re describ e existing systems in a common terminology

MMI
and

WIP ha v e b een c hosen b ecause t w

f

the authors w ere in v

lv

ed in the dev elopmen t

f

these systems

MMI
The

system MMI

A

MultiMo dal In terface for Man Mac hine In teraction with Kno wledge Based Systems w as dev elop ed with the purp

se
f

demonstrating the arc hitecture and dev elopmen t metho d required to pro duce large scale co

p

erativ e in terfaces to kno wledgebased systems

Within

the pro ject t w

demonstrators

ab

ut

lo cal and wide area net w

rk

design w ere pro duced

The

main concerns in the MMI

pro

ject w ere the arc hitecture notion

f

exp ert and the use

f

a common meaning represen tation CMR By exp ert it is mean t a mo dule p erforming sp ecic tasks with its

wn

priv ate data structures and whic h allo ws a sucien tly coheren t set

f

pro cesses to b e gathered in a single mo dule This corresp

nds

to

ur

notion

f

mo dule CMR is the common comm unication language among the comp

nen

ts

f

the system Comm unica tion b et w een the application and the application exp ert is in the language

f

the application In

ur

terminology

CMR

is a know le dge exchange format It is used to supp

rt

ssion and fusion

f

information b et w een media and to supply a common discourse con text through whic h to resolv e references made within and b et w een media Eac h CMR pac k et con tains

ne
r

more CMR acts along with the status mo de and time for those acts A media eld iden ties the me dia through whic h the pac k et w as receiv ed as user input

r

the

ne

for whic h it is destined as system

utput

MMI

w

as devised for coop erativ e dialogues This implies that its comp

nen

ts are designed for inputoutput

r

t w

w

a y

SLIDE 14 in teractions b et w een the user and the application Since w e are dealing

nly

with the

utput

generation w e will isolate and describ e this part

f

the system ev en though it is actually strongly merged with the input subsystem Goal form ulati

n
The

go al formulation manages input acquisition and ap plication data up dating In MMI

a

goal is not directly sp ecied b y the user The go al formulation pro cesses the m ultimedia

inputs

from the user building up a goal

r

a presen tational command as the result

f

the fusion

f

sev eral co

rdinated

inputs concerned with

utput

generation Using the terminology

f
nly

the attitudes User wants to know and User wants are passed to the presen tation system They corresp

nd

to goals and presen tational com mands resp ectiv ely

Con

trol La y er The c

ntr
l

manager called dialo gue c

ntr
l

ler

classies

the CMR passed b y the go al formulation in to goal

r

presen tational command A pair consisting

f

a CMR and its classication is called userdesir e

The

classi cation is done

n

the basis

f

the form

f

the CMR and the form

f

previous system presen tations The disc

urse

mo del stored in the con text exp ert is structured simply as a stac k The con trol manager p

ps

the top

f

the stac k to decide the next action ac hieving a goal executing a presen tational command

r

notifying to the goal form ulation A notication from the con ten t la y er can con tain some reco v ery strategies if the presen tation pro cess fails These strate gies em b

dy

subgoals to b e ac hiev ed and notications to the goal form ulation whic h are pushed in the disc

urse

mo del Con ten t La y er The role

f

the con ten t la y er is to con v ey to the user the systems in ten tions in suc h a w a y that the rules

f

co

p

erativit y in dialogue are follo w ed The goal renemen t mo dule p erforms rst an analysis called informal semantics

f

the goal in input exploiting some priv ate kno wledge in

rder

to pro vide pragmatic dialogue

rien

ted functionalities b ey

nd

the narro w range

f

selecting the data describ ed b y the goal called formal seman tics

The

main functionalities are repair iden tication

f

error conditions whic h prev en t straigh tforw ard handling

f

goals b y means

f

task plans ex planations the determination and pro vision

f

presen tations that are more in formativ e than formal seman tics alone w

uld

giv e and clarication assessing goals c hec king their v alidit y

etc

A submo dule called c

mmunic

ation plan ner pro duces comm unication acts whic h are then forw arded to the mo dules for con ten t selection and media selection resp ectiv ely

The

con ten t selection mo d ule in terprets the comm unication acts b y decomp

sing

them in terms

f

simple predicates directly in terpretable b y the application exp ert and then collecting and comp

sing

the answ ers from it Media selection relies

n

me dia sele ction rules whic h are based

n

cognitiv e studies see

for

an

v

erview Finally

a

co

rdination

mo dule collects the media comm unication acts and sends them

SLIDE 15 to the la y

ut

la y er La y

ut

La y er The la y

ut

manager called interfac e exp ert in MMI

main

tains a record

f

windo w p

sitions

and pro vides lo cations for new windo ws It passes media comm unicatio n acts receiv ed from the con ten t la y er to the media design media realization and co

rdination

The

utput

media in the MMI

are

English F renc h and Spanish language and graphics The graphics media design and realization is sup ervised b y a gr aphics manager whic h has at its disp

sal

a n um b er

f

graphics to

ls

including tables bar c harts pie c harts scatter plots and net w

rk

to

ls

Presen tation La y er The co

rdination

mo dule mainly consists

f

a windo w manager SunViewXwindo ws together with a comp

nen

t dispatc hing the unrendered presen tations to the suitable device in terface The

nly
utput

device is the displa y

User

Exp ert The user exp ert dynamic al ly acquires and stores kno wledge ab

ut

the users including The users general kno wledge

f

the domain

f

ap plication eg whic h domain

b

jects the user kno ws ab

ut

the users kno wl edge

f

the MMI

system

eg whic h MMI

commands

the user kno ws ab

ut

users preferences eg whic h currency the user w

uld

prefer a stereot yp e hi erarc h y

f

users in the domain whic h allo ws m ultiple inheritance from dieren t stereot yp es the w a y that h uman exp erts decide that an in terlo cutor b elongs to a particular stereot yp e Information ab

ut

the curren t user is deriv ed from the discourse stored in the con text exp ert

r

ev en b y the user himself eg c hanging his stereot yp e through a graphical in terface to sho w the inheritance net w

rk
f

user stereot yp es the kno wledge within a particular user mo del the predicates p ermitted in user mo dels and an y inconsistencies b et w een b eliefs in a user mo del

r

deriv ed from its paren ts The graphical in terface is mo delled as an external clien t

f

the user exp ert Application and Application Exp ert In MMI

the

application is an ex p ert system called NEST a Net w

rk

design Exp ert SysT em pro viding kno wl edge

n

net w

rk

design lo cation

f

mac hines graphics

b

jects etc In partic ular its kno wledge base con tains all the denitions

f

the needed

b

jects ie b

th

the v arious net w

rk

comp

nen

ts and the top

logical

information relativ e to the buildings The role

f

the application exp ert is to pro vide the data describ ed b y a part

f

a goal and the denotations

f

the sym b

ls

used in the CMR pac k ets b y exploiting the application kno wledge according to the user preferences eg currency

unit
f

measure preferred etc and the con text Con text Exp ert This mo dule pro vides con textual functionalities essen tially anaphora and ellipses resolution whic h are in v

lv

ed in the con textual pro cess ing

f

eac h mo v e and joins an MMI

standard

reied w

rld

with a reied w

rld

SLIDE 16

f

the application The MMI

standard

w

rld

con tains conceptual lab els repre sen ting the reied

b

jects

f

the la y

ut

the graphics

b

jects and the w

rd

sense represen tativ es that are relev an t to the comm unication for English F renc h and Spanish languages The w

rld
f

the application is represen ted b y means

f

the relev an t terms

f

the application and the w

rd

senses represen tativ es that are relev an t to the comm uni cation for English F renc h and Spanish languages Design Exp ert The design exp ert pro vides kno wledge ab

ut

design con strain ts cognitiv e impact

f

the used media device mo del and media c harac teristics The kno wledge is tailored to the user preferences and to the con text

WIP

The design

f

the WIP system

WIP

stands for the German abreviation

f

Know le dgeBase d Pr esentation

f

Information

w

as inuenced b y the

bser

v ation that comm unication is alw a ys situated ie dep ends

n

some con text This is tak en in to accoun t b y considering the users preferences and design constrain ts called gener ation p ar ameters

A

further basic assumption in the WIP system is that not

nly

the generation

f

text and dialog con tributions but also the design m ultim edia presen tations are planning tasks The cur ren t protot yp e

f

WIP generates m ultim edia explanations and instructions

n

assem bling using

r

main taining ph ysical devices The ma jor design goals

f

WIP are the generation

f

co

rdinated

presen tations from a common represen tation the adaptation

f

these presen tations to the in tended target audience and situation and the incremen talit y

f

all pro cesses constituting the design and realization

f

the m ultim edia

utput

In addition page la y

ut

is addressed as a rethorical force Goal F

rm

ulatio n In WIP go al formulation can b e done b y the user via a men uin terface This in terface allo ws the user to mo dify the generation pa rameters and to c ho

se

a goal to b e ac hiev ed in MMI

instead

the goal is built up b y the goal form ulation as a function

f

the users inputs A goal is expressed as a men tal state whic h the presen tation view er is to come ab

ut

Con trol La y er In WIP the selection

f

the next goals to b e accomplished is done b y a sub comp

nen

t

f

the presen tation planner Th us WIPs presen tation planner actually spans

v

er t w

la

y ers

f
ur

generic reference arc hitec ture In con trast to MMI

WIP
nly

generates nonin teractiv e presen tations Therefore no reco v ery strategy is pro vided b y the con ten t la y er when the user in terrupts a presen tation Con ten t La y er A t the heart

f

the presen tation system is a parallel top do wn planning mo dule Its task is to nd a presen tation strategy for the giv en

SLIDE 17 goal b y incremen tally generating a renemen tst yle plan in the form

f

a di rected acyclic graph D A G b y means

f

some presen tation strategies They reect general presen tation kno wledge

r

they em b

dy

more sp ecic kno wl edge pro vided b y the application exp ert

f

ho w to presen t a certain sub ject The lea v es

f

the planned D A G are sp ecications for elemen tary m ultim edia comm unication acts whic h are elab

rated

b y the media design and realization in the la y

ut

la y er WIPs presen tation planner instan tiates the c

ntent

layer mo dule in the reference mo del since it is resp

nsible

for go al r enement c

n

tent sele ction me dia sele ction and c

r

dination

Whereas

in the MMI

system

the goal renemen t media selection and co

rdination

tasks are p erformed se quential ly

in

the WIP system they are p erformed concurren tly

The

reason for using suc h an in tegrated approac h is that in terdep endencies b et w een these pro cesses can b e handled within a uniform pro cessing mec hanism and that the approac h also allo ws for incremen tal

utput

generation La y

ut

La y er The WIP la y

ut

manager stores a set

f

do cumen t t yp es to gether with some la y

ut

constrain ts for eac h In addition comm uni cativ e acts asking for reply eg is visibleob ject are answ ered b y the la y

ut

manager after dispatc hing the request to the appropriate mo dules for media design and realization In WIP

these

are mo dules for generating Dgraphics German natural language and English natural language In illustrated instructions for tec hnical equipmen t graphics are used in

rder

to accomplish presen tation tasks suc h as depicting a domain

b

ject in a certain state sho wing an

b

jects lo cation

r

visualizing the course

f

an action The dev elop ers

f

WIP

p

erationalized certain D and D illustration tec hniques frequen tly used b y h uman illustrators The formalization is based

n

a comp

sitional

seman tics

f

pictures Using graphical design strategies graphics design is in principle a goaldriv en planning pro cess Ho w ev er it do es not seem feasible to strictly separate a graphics design and realization phase as some realization

p

erators ha v e side eects whic h are computationally exp ensiv e to an ticipate A solution to this problem is to in terlea v e graphics design and realization and to allo w for feedbac k Notew

rth

y is also that WIP do es a negrained co

rdination
f

text and graphics generation F

r

example WIP is able to generate crossmedia

deictic

assertions lik e The

no

switc h is lo cated in the upp erleft part

f

the picture and referring expressions whic h are itself comp

sed
f

dieren t media suc h as sho wing a picture together with the assertion The switc h

n

the fron tside Presen tation La y er The presen tation la y er

f

the WIP system comprises a windo w manager and an in terface to a p

stscript

prin ter User Exp ert The WIP stereot yp e user mo del distinguishes novic e and ex p ert users Users goals preferences and kno wledge are stored in the kno wledge

SLIDE 18 bases The user mo del is up dated after a goal has b een ac hiev ed F rom that time

n

the user is supp

sed

to kno w the information con v ey ed b y the pre sen tation

f

the goal Application and Applicatio n Exp ert The application kno wledge is partly co died as prop

sitions

in a terminological logics

and

partly as geometric wireframes for the Dgraphics generation The prop

sitionally

represen ted kno wledge is used b

th

for the generation

f

text and graphics as the main source

f

kno wledge ab

ut

the domain Con text Exp ert The con text kno wledge consists

f

a do cument design plan

and

some predicates for managing referring expressions The former is incre men tally built b y WIPs presen tation planner The lea v es

f

the resulting do cumen t design plan are sp ecications for elemen tary me dia c

mmunic

ation acts suc h as sp eec h acts and pictorial acts whic h are to b e accomplished b y WIPs text and graphics generator resp ectiv ely

Design

Exp ert The design exp ert

f

WIP stores kno wledge

n

gener ation p ar ameters eg constrain ts

n

the do cumen t la y

ut

lik e shortlong presen ta tion etc giv en b y the user la y

ut

constrain ts dynamically inferred from the con text and the user preferences and nally

kno

wledge

n

device a v ailabili t y whic h is acquired from the

p

erating system mo delled as external serv erclien t

Conclusions

While signican t results and exp ertise ha v e b een gained from building the rst generation

f

IMMPSs there ha v e b een no promising attempts to bring together the dieren t lines

f

exp ertise from across disciplines and viewp

in

ts and assem ble them in to a sound corpus

f

scien tic theory

The

purp

se
f

this pap er w as to

utline

a preliminary v ersion

f

a reference mo del for IMMPSs The prop

sed

mo del ma y b e summarized b y the equation IMMPS

La

y ers

Exp

erts The design

f

the mo del reects b

th

a decomp

sition
f

m ultimedia genera tion in to logically distinct subtasks represen ted as La y ers and a separation

f

these tasks from the kno wledge sources called Exp erts whic h migh t b e exploited to accomplish them The mo del is mean t as a rst step to w ards a broader agreemen t

f

the scien tic and industrial comm unit y

n

the topic F urther w

rk

and discussions are necessary to

btain

a rened reference mo del whic h ev en tually can b e forw arded to the relev an t b

dies

dev

ted

to standard ization activities Ho w ev er the curren t mo del can already b e used to analyze and compare r e al systems and ma y pro vide guidance for the dev elopmen t

f

new IMMPSs

SLIDE 19 References

S

K F einer and KR McKeo wn Automating the Generation

f

Co

rdinate

Multimedia Explanations In Ma ybury

Computer

Graphics Reference Mo del In ternational Standard Organization ISOIEC IS

F

Halasz and M Sc h w artz The dexter h yp ertext reference mo del A CM Communic ation V

l
No
L

Hardman D Bulterman and G v an Rossum The amsterdam h yp ermedia mo del Adding time and con text to the dexter mo del A CM Communic ation V

l
No
K

QML Advisory Group An Ov erview

f

K QML A Kno wledge Query and Manipulation Language h ttpretriev ercsum b cedukqml

G

E Pfa editor User Interfac e Management Systems Pr

c

e e dings

f

the Se eheim Workshop Springer V erlag

F

Roth and E Heey

In

telligen t Multimedia Presen tation Systems Researc h and Principles

In

Ma ybury

JR

Searle What is a sp eec h act In M Blac k editor Phylosophy in A meric a pages

E

Andr

e

and T Rist T

w

ards a PlanBased Syn thesis

f

Illustrated Do cu men ts In Pr

c

e e dings

f

ECAI

Sto

c kholm

M

T Ma ybury

Planning

Multimedia Explanations Using Comm uncativ e Acts In Intel ligent Multime dia Interfac es

MD

Wilson Enhancing m ultimedia in terfaces with in telligen ce Multime dia systems and applic ations

The

MMI

Demonstrator

Systems A MultiMo dal In terface for Man Mac hine In teraction with Kno wledge Based Systems T ec hnical Rep

rt

RAL Rutherford Appleton Lab

ratory
UK
D

Sedlo c k G Do e M Wilson and D T rotzig F

rmal

and informal in terpre tation for coop erativ e dialogue

HR

Chapp el M D Wilson and B Cahour Engineering User Mo dels to Enhance Multimo dal Dialogue In JA Larson and C Unger editors En gine ering for HumanComputer Inter action pages

Elsevier

Science Publishers Amsterdam

E

Andr

e

W Finkler W Graf T Rist A Sc hauder and W W ahlster WIP The Automatic Syn thesis

f

Multimo dal Presen tations In Ma ybury

E

Andr

e

and T Rist Referring to World Ob jects with Text and Pictures In Pr

c

e e dings

f

Coling

Osak

a

J

Heinsohn D Kudenk

B

Neb el and H J Protlic h RA T

represen

tation

f

actions using terminological logics T ec hnical rep

rt

DFKI Saarbr

uc

k en German y

M

Ma ybury

editor

Intel ligent Multime dia Interfac es AAAIThe Mit Press

View publication stats

View publication stats