[PPT] - ' $ Algebraic Semiotics and User In terface Design Joseph PowerPoint Presentation

SLIDE 1 ' & $ % Algebraic Semiotics and User In terface Design Joseph A. Goguen Departmen t

f

Computer Science & Engineering Univ ersit y

f

California at San Diego

SLIDE 2 ' & $ % ABSTRA CT HCI lac ks scien tic theories for design; so new media, new metaphors (b ey

nd

the desktop), new hardw are, non-standard users (e.g., with disabilities) can b e c hallenging. Semiotics seems natural, but (1) lac ks mathematical basis, (2) considers single signs (no v els, lms, etc.), not represen tations; (3) do esn't address dynamic signs,

r

(4) so cial issues, e.g., for co

p

erativ e w

rk.

Algebraic semiotics denes sign system & represen tation, giv es calculus

f

represen tation & represen tation qualit y . Case studies

n

bro wsable pro

f

displa ys, scien tic visualization, natural language metaphor, blending, h umor. So cial foundation uses ideas from ethnometho dology .

SLIDE 3 ' & $ % Outline 1. Motiv ation: Some Problems 2. Algebraic Semiotics 3. Calculus

f

Represen tation 4. Case Studies 5. Summary & F uture Researc h

SLIDE 4 4 ' & $ % 1. Motiv ation: Some Problems Most HCI results are:

sp

ecialized & precise (e.g., Fitt's la w),

r

else

general

but

f

uncertain reliabilit y & generalit y (e.g., proto col analysis, questionnaires, case studies, usabilit y studies). What w e need are scien tic theories to guide design, e.g., for

new

media,

new

metaphors (b ey

nd

the desktop),

new

hardw are,

non-standard

users (e.g., with disabilities).

SLIDE 5 5 ' & $ % Semiotics, the general theory

f

signs, seems natural for a general HCI framew

rk.

But it 1. do es not ha v e mathematical st yle & so do es not supp

rt

engineering applications; 2.

nly

considers single signs

r

sign systems (e.g., no v el, lm), not represen ting signs in

ne

system b y signs in another, as needed for in terfaces; 3. has not addressed dynamic signs, as needed for user in teraction; 4. has not considered so cial issues, as arise in co

p

erativ e w

rk;

5. ignores the situated, em b

died

asp ect

f

sign use.

SLIDE 6 6 ' & $ % 2. Algebraic Semiotics Algebraic Semiotics pro vides:

precise

algebraic denitions for sign system & represen tation;

calculus
f

represen tation, with la ws ab

ut
p

erations for com bining represen tations;

precise

w a ys to compare qualit y

f

represen tations. Ha v e case studies

n

bro wsable pro

f

displa ys, scien tic visualization, natural language metaphor, blending, & h umor. So cial foundations grounded in ideas from ethnometho dology: semiosis, the creation

f

meaning, is situated, em b

died,

etc.

SLIDE 7 7 ' & $ % 2.1 Signs and Sign Systems

Signs

should not b e studied in isolation, but rather

as

elemen ts

f

systems

f

related signs, e.g., v

w

el systems, trac signs, alphab ets, n umerals, n um b ers.

Signs

ma y ha v e parts, subparts, etc.,

f

dieren t sorts.

Sign

parts ma y ha v e dieren t saliency, determined b y ho w constructed. Signs b ecome what they are b y ha ving dieren t attributes than

ther

signs { clear from mac hine learning

f

patterns. Same sign in dieren t system has dieren t meaning { e.g., alphab ets. Com bines ideas

f

P eirce (sign), Saussure (structure), Goguen (ADTs).

SLIDE 8 8 ' & $ % F

rmalize

sign system as algebraic theory with data, plus 2 sp ecic semiotic items:

signature

for sorts, subsorts &

p

erations (constructors & selectors);

axioms

(e.g. equations) as constrain ts;

data

sorts & functions;

lev

els for sorts;

priorit

y

rdering
n

constructors. Sorts classify signs,

p

erations construct signs, data sorts pro vide v alues for attributes

f

signs, lev els & priorities indicate saliency . This is not the formal v ersion; also not necessarily nal. Diers from approac hes

f

Gen tner, Carroll, etc.

axiomatic

with lo

se

seman tics, not set-based; giv es a language, not a mo del; this allo ws partial mo dels,

p

en structure, etc.

SLIDE 9 9 ' & $ % 2.2 Represen tation User in terface design means designing go

d

represen tations. E.g., GUIs represen t functionalit y with icons, men us, etc. Basic insigh t: represen tations are maps M : S 1 ! S 2

f

sign systems, called semiotic morphisms, preserving as m uc h as reasonable:

sorts

& subsorts,

ps,

preserving source & target sorts,

axioms

to consequences

f

axioms,

data

& functions,

lev

els

f

sorts,

priorit

y

f

constructors. \Reasonable" qualication due to need for tradeos.

SLIDE 10 10 ' & $ % 2.3 Simple Examples 1. S E { English sen tences. 2. S T { parse trees for English sen tences. 3. S P { prin ted page format. 4. P : S E ! S T { parsing. 5. H : S T ! S P { phrase structure represen tation. Time ies lik e an arro w. [[time] N [[f l ies] V [[l ik e] P [[an] Det [ar r

w

] N ] NP ] PP ] VP ] S . Can't alw a ys preserv e ev erything

resulting

displa y ma y b e to

complex

for h umans. And sometimes just w an t to summarize some data set.

SLIDE 11 11 ' & $ % 2.4 Qualit y

f

Represen tation Con ten t means v alues

f

selector

ps,

e.g., size, color.

Easy

to dene sort preserving, constructor preserving, lev el preserving, con ten t preserving, etc.

But

not v ery useful since

ften

are not preserv ed.

Instead,

dene more sort preserving, more lev el preserving, more constructor preserving, more con ten t preserving, etc.

These

comparativ e notions dene

rderings
n

morphisms.

Can

com bine

rderings

to get righ t

ne

for giv en application.

Giv

en S; S ,

ne

ma y preserv e more lev els,

ther

more con ten t.

More

imp

rtan

t to preserv e structure than con ten t.

More

imp

rtan

t to preserv e lev els than priorit y .

Also

it's easier to describ e structure.

SLIDE 12 12 ' & $ % 3. Calculus

f

Represen tation Can comp

se

morphisms & so study comp

sed

represen tations, as arise in iterativ e design. Ha v e iden tit y & asso ciativ e la ws: A ; 1 S = A 1 S ; B = B A ; (B ; C ) = (A ; B ) ; C Therefore ha v e a category. This giv es

ther

simple la ws, plus notions: isomorphism

f

sign systems, sum & pro duct

f

sign systems & represen tations, plus m uc h more (see follo wing).

SLIDE 13 13 ' & $ % 3.1 Blending F auconnier & T urner studied blending metaphors, using conceptual spaces { sign systems with

nly

constan ts & relations. Conceptual blend

f

maps with same source, the generic space, & targets called input spaces, com bining their features in blend space.

@

@ @ I

@

@ @ I 6 I 1 I 2 G B W e generalize to arbitrary sign systems, morphisms, & diagrams.

SLIDE 14 14 ' & $ % Examples: house b

at;

road kill; computer virus; articial life; jazz piano; conceptual space; blend diagram; ... Blend diagram suggests categorical pushout { but do esn't w

rk,

since blends not unique. Example: \house

b
at"

has 4 dieren t maximal blends: 1. houseb

at;

2. b

athouse;

3. amphibious R V; 4. b

at

for mo ving houses (!). But since

rdered

category, use \lax" pushout:

has

non-unique result; and

can

actually calculate the 4 blends ab

v

e! Order b y f

g

i g preserv es as m uc h con ten t as f , as man y axioms as f , and is as inclusiv e as f .

SLIDE 15 15 ' & $ % 3.2 Some La ws A

1
=

A 1

A
=

A A

B
=

B

A

A

(B
C

)

=

(A

B

)

C

a

b
=

b

a

a

(b
c)
=

(b; a)

c

(a

b)
c
=

a

(b;

c) A; B ; C can b e either sign systems

r

semiotic morphisms. Pro duct is sp ecial blend with common space empt y; sum

f

theories giv es mo del pro duct. So pro duct la ws are sp ecial blends la ws.

SLIDE 16 16 ' & $ % 4. Case Studies 1. Blending (already discussed). 2. Metaphor (similar to F auconnier & T urner). 3. Scien tic visualization. 4. Pro

f

presen tation. 5. Humor. So w e will do items 3, 4, 5.

SLIDE 17 17 ' & $ % 4.1 Scien tic Visualization Visualizations

f

complex data help scien tists disco v er, v erify & predict patterns. Dicult to construct \appropriate" visualizations. But visualizations ar e represen tations &

ur

qualit y measures apply; b est to use in semi-formal st yle: 1. use ideas & results to guide examination; 2. use formalism

nly

if needed for dicult design decision. Tw

examples

illustrate tec hniques: 1. co de visualizer. 2. mo vie visualizer. Able to suggest impro v emen ts in b

th

cases.

SLIDE 18 18 ' & $ % 4.2 Pro

f

Presen tation

Understanding

pro

fs

is notoriously dicult. Wh y?

T

atami pro ject views pro

fs

as represen ting underlying math.

Then

can apply algebraic semiotics, qualit y measures, etc.

But

what is the underlying math?

Imp
rtan

t ingredien ts include: 1. narrativ e (Lab

v

& Linde). 2. drama { Aristotle said \drama is conict." 3. image sc hemas (Lak

&

Nunez).

Pro
fw

eb data structure includes narrativ e & conict, as w ell as formal sen tences & inference rules. See www.cs.ucsd.edu/groups/tatami/kumo/exs/.

SLIDE 19 19 ' & $ % 4.3 Humor Studied corpus

f
v

er 50 h umorous

xymorons

| \military in telligence," \go

d

grief," \almost exactly ," ... \Oxymoron" is phrase with con tradictory (or incongruous) terms. Humorous

xymorons:

con v en tional & con tradictory meaning. i.e., 2 dieren t blends,

ne

with conicting elemen ts. Studied

v

er 40 newspap er carto

ns

{ ab

ut

3/4 ha v e same pattern. So this seems a general facet

f

h umor. Note that h umor is used in man y in terfaces,

ften

badly .

SLIDE 20 20 ' & $ % 5. Summary & F uture Researc h Algebraic semiotics seems promising for user in terface design & can handle metaphors, blends, h umor. But m uc h more w

rk

is needed:

More

case studies, more carefully done.

Dynamic

signs for user in teraction { use hidden algebra.

Com

bine Gibsonian aordances with algebraic semiotics.

More
n

narrativ e structure.

More
n

so cial foundations, semiosis.

Ho

w to c ho

se
rderings
n

represen tations?