Human Capabilities SE3830 - Jay Urbain, PhD Credits: Card, Moran, - - PowerPoint PPT Presentation

Human Capabilities

SE3830 - Jay Urbain, PhD

Credits: Card, Moran, Newell, The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, 1983.

• Prof. Robert Miller, MIT Department of Electrical Engineering and Computer Science.

Donald Norman, Design of Everyday Things

User interface hall of shame?

• Looking for contributions!

Eye Tracking!

• http://online.wsj.com/article/SB100014241278873241052045783823

53581452288.html?mod=WSJ_hp_EditorsPicks

• http://www.tobii.com/group/news-and-events/tobii-in-media/tobii-

presents-eye-controlled-laptop/

• http://www.nytimes.com/2011/03/27/business/27novel.html?ref=busi

ness

• Andrew T. Duchowski, a professor of computer science at Clemson

University, and author of the book “Eye Tracking Methodology.”

• Kinect - Ontology surgeons bring video game technology to the OR

http://www.ctv.ca/CTVNews/Health/20110318/surgeons-kinect- 110318/

Introduction

• This course is about building effective human-computer
• interactions. We've talked about:
• Usability.
• Guidelines, principles, and theories.
• Evaluating interface designs.
• What about the properties of the system we are designing

for?

Introduction

• We should understand the properties of the system we are

designing for. – Speed, memory size, hard disk, operating system, and the interaction between these in the computer system. – Processors, memories, and properties of the human cognitive apparatus we are designing an interaction for.

Topics

• Human information processing
• Perception
• Motor skills
• Memory
• Decision making
• Attention
• Vision

Hierarchical temporal memory (HTM)

• Machine learning model (Jeff Hawkins and Dileep George)

that models some of the structure and algorithmic properties

• f the neocortex.
• HTM is a biomimetic model based on the memory-

prediction theory.

• HTM combines and extends approaches used in Bayesian

networks, spatial and temporal clustering algorithms, while using a tree-shaped hierarchy of nodes that is common in neural networks.

Memory-prediction Framework

• The central concept of the framework is that bottom-up

inputs are matched in a hierarchy of recognition.

• Evoke a series of top-down expectations encoded as

potentiations.

• These expectations interact with the bottom-up signals to

both analyze those inputs and generate predictions of subsequent expected inputs.

Model Human Processor

Developed by Card, Moran, and Newell as a way to summarize decades of psychology research in an engineering model. (The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, 1983).

Model Human Processor (MHP)

• This model is an abstraction, but provides us with

numerical parameters describing how we behave.

• Just as a computer has memory and a processor, so

does our model of a human.

• The model has several different kinds of memory, and

several different processors.

MHP - Memory

• Input from the eyes and ears is first stored in the

short-term sensory store.

• The perceptual processor takes the stored sensory

input and attempts to recognize symbols in it: letters, words, phonemes, icons.

• It is aided in this recognition by the long-term

memory, which stores the symbols you know how to recognize.

MHP - Cognitive Processor

• The cognitive processor takes the symbols

recognized by the perceptual processor and makes comparisons and decisions.

• It might also store and fetch symbols in working

memory (RAM).

• The cognitive processor does most of the work that we

think of as “thinking”.

MHP - Motor Processor

• The motor processor receives an action from the

cognitive processor and instructs the muscles to execute it.

• There’s an implicit feedback loop here:

– the effect of the action can be observed by your senses, and used to correct the motion in a continuous process.

• Finally, there is a component corresponding to your

attention, which might be thought of like a thread of control in a computer system.

Memories

Each memory has properties

• Encoding: type of things stored
• Size: number of things stored
• Decay time: how long memory lasts

Short Term Sensory Store - VIS

• Visual information store (VIS)

– encoded as physical image – size ~ 17 [7-17] letters – decay ~ 200 ms [70-1000 ms] – Basically a framebuffer for the eyes. – Instead of being encoded as pixels, encoded as physical features of the image: curvature, length, edges. – Measure in letters since psych studies typically use letters to measure VIS properties. – VIS memory is fleeting, decaying in a few hundreds msecs.

Short Term Sensory Store - AIS

• Auditory information store (AIS)

– Encoded as physical sound – Size ~ 5 [4.4-6.2] letters – decay ~ 1500 ms [900-3500 ms] – Buffer for physical sound. – Smaller than the VIS (in terms of letters), but lasts longer – seconds, rather than tenths of a second.

Short Term Sensory Store

• VIS and AIS are pre-attentional; i.e., they don’t need the

spotlight of attention to focus on them in order to be collected and stored.

• Attention can be focused on the visual or auditory

stimulus after the fact.

• That accounts for phenomena like “What did you say?

Oh yeah.”

Processors

• Processors have a cycle time (time to accept 1 input,

process, & generate 1 output) – Tp ~ 100ms [50-200 ms] (Slowman, Middleman, Fastman) – Tc ~ 70ms [30-100 ms] – Tm ~ 70ms [25-170 ms]

• Fastman may be up to 10x faster than Slowman
• Cycle times derived from psych studies

Processors

• Variation due to humans, stimuli, & conditions
• Cognitive processor actually works faster under load!
• Playing a video game vs. watching TV.
• Cognitive processor is also faster on practiced tasks.

Perceptual Fusion

• Two stimuli within the same PP (Perceptual Processor)

cycle (Tp ~ 100ms) appear fused.

– Like snapping a picture every Tp secs. – If 2 events happen within Tp secs, they will both appear in the picture. – If you move within Tp, you will be fused into a single in-motion element (frame).

• Consequences

– 1/ Tp frames/sec is enough to perceive a moving picture (10-15 fps OK, 20 fps smooth, 24 fps for movies). – Computer response < Tp feels instantaneous. – Causality is strongly influenced by fusion.

Bottom-up vs. Top-Down Perception

• Perception not an isolated process
• Bottom-up combines features of stimulus
• Top-down uses context
• Temporal (audio), spatial (visual)
• Draws on long-term memory

Chunking

• “Chunk”: unit of perception or memory
• Chunks are defined by symbols that represent the

activation of past experience.

• Depends on presentation and what you already know

B M W R C A A O L I B M F B I MWR CAA OLI BMF BIB BMW RCA AOL IBM FBI

• 3-4 digit chunking is ideal for encoding unrelated digits

Chunking

• Ability to form chunks in working memory depends on

how the information is presented –

– a sequence of individual letters tend to be chunked as letters – a sequence of three-letter groups tend to be chunked as groups.

• It also depends on what we already know.
• If the three letter groups are well-known TLAs (three-

letter acronyms) we are better able to retain them in working memory.

Chunking

Famous study of chess players:

• Novices and chess masters were asked to study

chess board configurations and recreate them from memory.

• The novices could only remember the positions of a

few pieces.

• Masters could remember entire boards, but only when

the pieces were arranged in legal configurations.

• When the pieces were arranged randomly, masters

were no better than novices.

• The ability of a master to remember board

configurations derives from their ability to chunk the board, i.e., recognizing patterns from their past experience of playing and studying games.

Attention and Perception

• Spotlight metaphor

– Used for how attention behaves in perception. – You can focus your attention (PP) on one input channel in your environment at a time. – Channel could be visual or audio field (sound or loc.)

• Once you’ve focused your attention on a particular

channel, all the stimuli within the area of the “spotlight” are then processed in parallel, whether you mean to or not.

• This can cause interference.
• Visual dominance: easier to attend to visual channels than

auditory channels

Attention and Perception

• Say the colors of these words aloud and time

yourself: – Book – Pencil – Slide – Window – Car – Hat

Attention and Perception

• Do it again:

– Green – Orange – Red – Black – Pink – Blue

Attention and Perception

• Why is this more difficult the second time?

Attention and Perception

Why?

• The word which names a different color, interferes with

the color we’re trying to say.

• This is called the Stroop effect.

– Demonstration of interference in the reaction time of a task. – http://en.wikipedia.org/wiki/Stroop_effect

• Take away:

– Choose the secondary characteristics of our displays – like the multiple dimensions of stimulus, or the context around the stimulus – to reinforce the message of the display, not interfere with it.

Cognitive Processing

• Cognitive processor

– compares stimuli – selects a response

• Types of decision making

– Skill-based – Rule-based – Knowledge-based

Hick-Hyman Law of Choice Reaction Time

• Simple reaction time – responding to a single stimulus with

a single response – takes just one cycle of the human information processor. RT = Tp+Tc+Tm

• But if the user must make a choice –

– Choosing a different response for each stimulus - cognitive processor has to do more work.

The Hick-Hyman Law of Reaction Time

• assesses cognitive information capacity in choice reaction experiments
• shows that the number of cycles required by the cognitive processor is

proportional to amount of information in the stimulus.

Hick-Hyman Law of Choice Reaction Time

• Given N equally probable stimuli, each requiring a different

response – how much time is required to respond?

– Cognitive processor needs log N cycles to decide which stimulus was actually seen and respond appropriately. – Given N equally probable choices, the average reaction time RT required to choose among them is approximately

For N equally probable choices: RT = c + d log2 (N + 1) For N choices with unequal probabilities: RT = c + d * sum(pi*log2 (1/pi + 1))

Note: Works for skill-based decisions only

• c – offset constant not always

used

• d – proportionality (scaling)

constant determined empirically

Hick-Hyman Law of Choice Reaction Time

• Intuitively, one can reason that Hick's Law has a logarithmic form

because people subdivide the total collection of choices into categories, eliminating about half of the remaining choices at each step, rather than considering each and every choice one-by-one, requiring linear time.

Speed-Accuracy Tradeoff

• Accuracy varies with reaction time

– Can choose any point on curve – Can move curve with practice

Hick-Hyman Law of Choice Reaction Time

Original work:

• Hick, William E.; On the rate of gain of information. Quarterly Journal of

Experimental Psychology, 4:11-26, 1952.

• Hyman, Ray; Stimulus information as a determinant of reaction time.

Journal of Experimental Psychology, 45:188-196, 1953.

End part 1

Divided Attention (Multitasking)

• Resource metaphor

– Attention is a resource that can be divided among different tasks simultaneously

• Multitasking performance depends on:

– Task structure

• Tasks with different characteristics are easier to share; tasks

with similar characteristics tend to interfere.

• Modality: visual vs. auditory
• Encoding: spatial vs. verbal
• Component: perceptual/cognitive vs. motor vs. working

memory – Difficulty

• Easy or well-practiced tasks are easier to share

Stanford Study

Ophir, E., Nass, C., & Wagner, A. (2009). Cognitive control in media

• multitaskers. In Proceedings of the National Academy of Sciences,

106(33), 15583-15587. Download “Media multitaskers pay mental price,” Adam Gorick, Stanford Report, August 24, 2009.

http://news.stanford.edu/news/2009/august24/multitask-research-study-082409.html

“Why Multitaskers Stink at Multitasking,” WSJ, August 26, 2009.

http://blogs.wsj.com/digits/2009/08/26/why-multitaskers-stink-at-multitasking/tab/article/

Stanford Study

• “People who are regularly bombarded with several streams of electronic

information do not pay attention, control their memory, or switch from one job to another as well as those who prefer to complete one task at a time.”

• "They're suckers for irrelevancy," said communication Professor Clifford Nass,
• ne of the researchers whose findings are published in the Aug. 24 edition of the

Proceedings of the National Academy of Sciences. "Everything distracts them."

Stanford Multitasking Study

Stanford Study:

• Social scientists have long assumed that it's impossible to process more

than one string of information at a time. The brain just can't do it.

• But many researchers have guessed that people who appear to

multitask must have superb control over what they think about and what they pay attention to.

• Set out to learn what gives multitaskers their edge. What is their gift?
• "We kept looking for what they're better at, and we didn't find it," said

Ophir, the study's lead author.

Divided Attention (Multitasking)

Stanford Study:

• Two groups: those that do a lot of multimedia multitasking and those

that don’t.

• In one experiment, the groups were shown sets of two red

rectangles alone or surrounded by two, four or six blue rectangles.

• Each configuration was flashed twice, and the participants had to

determine whether the two red rectangles in the second frame were in a different position than in the first frame.

• They were told to ignore the blue rectangles, and the low

multitaskers had no problem doing that. But the high multitaskers were constantly distracted by the irrelevant blue images. Their performance was horrible.

• HMM – heavy multimedia

multitaskers

• LMM – light multimedia

multitaskers

• SEM – Standard Error of

the Mean (SEM) is the standard deviation of the sample-mean's estimate of a population mean.

Divided Attention (Multitasking)

Stanford Study:

• Because the high multitaskers showed they couldn't ignore

things, the researchers figured they were better at storing and

• rganizing information. Maybe they had better memories?
• A second test proved that theory wrong. After being shown

sequences of alphabetical letters, the high multitaskers did a lousy job at remembering when a letter was making a repeat appearance.

• "The low multitaskers did great," Ophir said. "The high multitaskers

were doing worse and worse the further they went along because they kept seeing more letters and had difficulty keeping them sorted in their brains."

Divided Attention (Multitasking)

Stanford Study:

• Puzzled why the heavy multitaskers weren't performing well, the

researchers conducted a third test.

• If the heavy multitaskers couldn't filter out irrelevant information or
• rganize their memories, perhaps they excelled at switching from
• ne thing to another faster and better than anyone else?
• Wrong again, the study found.
• The test subjects were shown images of letters and numbers at

the same time and instructed what to focus on. When they were told to pay attention to numbers, they had to determine if the digits were even or odd. When told to concentrate on letters, they had to say whether they were vowels or consonants.

• Again, the heavy multitaskers underperformed the light multitaskers.

Divided Attention (Multitasking)

Stanford Study:

• "They couldn't help thinking about the task they weren't doing," Ophir
• said. "The high multitaskers are always drawing from all the

information in front of them. They can't keep things separate in their minds.“

• Bottom line: people who heavily multitask have trouble focusing.

Motor Processing

• Open-loop control

– Motor processor runs a program by itself – cycle time is Tm ~ 70 ms

• Closed-loop control

– Muscle movements (or their effect on the world) are perceived and compared with desired result – cycle time is Tp + Tc + Tm ~ 240 ms

Motor Processing

Experiment:

• Take a sheet of lined paper and scribble a sawtooth wave back and

forth between two lines, going as fast as you can, but trying to hit the lines exactly on every peak and trough. Do it for 5 seconds.

• The frequency of the sawtooth carrier wave is dictated by open-loop

control, so you can use it to derive your Tm.

• The frequency of the wave’s envelope, the corrections you had to

make to get your scribble back to the lines, is closed-loop control. You can use that to derive your value of Tp + Tc.

Fitts’s Law

• Specifies how fast you can move your hand to a target of

a certain size at a certain distance away.

• Fundamental law of the human sensory-motor system,

which has been replicated by numerous studies.

• Time T to move your hand to a target of size S at

distance D away is: T = RT + MT = a + b log (2D/S)

• Depends only on index of difficulty log(2D/S)

Explanation of Fitts’s Law

• Moving your hand to a target is closed-loop control
• In each cycle, your motor system instructs your hand to

move the entire remaining distance D.

• The accuracy of that motion is proportional to the

distance moved, so your hand gets within some error D

• f the target.
• Each cycle covers remaining distance D with error D

Implications of Fitts’s Law

• Targets at screen edge are easy to hit

– Edge of the screen stops the mouse pointer, so you don’t need more than one correcting cycle to hit it. – Mac menubar beats (old) Windows menubar – Unclickable margins are foolish

• Hierarchical menus are hard to hit

– Due to correction cycles the user is forced to spend time getting the mouse pointer carefully over the submenu. – Windows: 0.5s timeout destroys causality (> slowman) – Mac does a better job with a triangular zone. – Linear popup menus (large D, small S) vs. pie menus (small D, large S).

Power Law of Practice

• Time Tn to do a task the nth time is:

Tn = T1 n–

is typically 0.2-0.6

• Time to do a task decreases with practice.
• Linear curve on a log-log scale of time and number
• f trials.
• Novices get rapidly better at a task with practice, but

then their performance levels off to nearly flat (although still slowly improving).

Working Memory (WM)

• Small capacity: 7 ± 2 “chunks” (George Miller)
• Fast decay (7 [5-226] sec)
• Maintenance rehearsal fends off decay
• Interference causes faster decay
• Where you do your conscious thinking (cognitive

processor). Note: The currently favored model in cognitive science holds that working memory is not actually a separate place in the brain, but rather a pattern of activation of elements in the long-term memory.

Long-term Memory (LTM)

• Huge capacity
• Little decay
• Elaborative rehearsal moves chunks from WM to LTM

by making connections with other chunks (relations).

• Long-term memory is probably the least understood part
• f human cognition.

The Eye

Vision is the primary way that a graphical user interface communicates to the user.

The Eye

• Cornea is the transparent, curved membrane on the front of the eye.
• Aqueous humor fills the cavity between the cornea and the lens, and

provides most of the optical power of the eye because of the large difference between its refractive index and the refractive index of the air

• utside the cornea.
• Iris is the colored part of the eye, which covers the lens. It is an opaque

muscle, with a hole in the center called the pupil that lets light through to fall on the lens. The iris opens and closes the pupil depending on the intensity of light; it opens in dim light, and closes in bright light.

• The lens focuses light. Under muscle control, it can move forward and

backward, and also get thinner or fatter to change its focal length.

• The retina is the surface of the inside of the eye, which is covered with

light-sensitive receptor cells.

• The fovea is the spot where the optical axis (center of the lens) impinges on

the retina. The highest density of photoreceptors can be found in the fovea; the fovea is the center of your visual field.

Photoreceptors

• Rods (non-color vision)

– Only one kind (peak response in green wavelengths) – Sensitive to low light (“scotopic vision”)

• Multiple nearby rods aggregated into a single nerve signal

– Saturated at moderate light intensity

• Cones

– Operate in brighter light

• Cones do most of the vision under photopic conditions

– Three kinds: S(hort), M(edium), L(ong) – S cones are very weak, centered in blue wavelengths – M and L cones are more powerful (2x), overlapping – M centered in green, L in yellow (but called “red”)

Signals from Photoreceptors

• Rods and cones do not send their signals directly to

the visual cortex; instead, the signals are recombined into three channels. 1. Brightness – M + L + rods 2. Red-green difference – L - M 3. Blue-yellow difference – weighted sum of S, M, L

Signals from Photoreceptors

• Difference channels drive the theory of opponent

colors: red and green are good contrasting colors because they drive the red-green channel to opposite extremes.

• Similarly, black/white and blue/yellow are good

contrasting pairs.

Color Blindness

• Red-green color blindness (protonopia & deuteranopia)

– 8% of males (can be significantly higher in some cohorts) – 0.4% of females

• Blue-yellow color blindness (tritanopia)

– Far more rare

• Guideline: don’t depend solely on color distinctions

– Use redundant signals: brightness, location, shape How do red-green color-blind people know whether the light is green or red?

• http://science.howstuffworks.com/life/human-

biology/colorblindness2.htm

Chromatic Aberration

• Different wavelengths focus differently

– Highly separated wavelengths (red & blue) can’t be focused simultaneously

• Guideline: don’t use red-on-blue text

– It looks fuzzy and hurts to read

Blue Details Are Hard to Resolve

• Fovea has no S cones

– Can’t resolve small blue features (unless they have high contrast with background)

• Lens and aqueous humor turn yellow with age

– Blue wavelengths are filtered out

• Lens weakens with age

– Blue is harder to focus

• Guideline: don’t use blue against dark backgrounds

where small details matter (text!)

Fovea Has No Rods

• Rods are more sensitive to dim light
• In scotopic conditions, peripheral vision (rod-rich) is

better than foveal vision

• Easier to see a dim star if you don’t look directly at it

Summary

• Understanding human as model processor helps us

design interactions that capitalize on human strengths and support human weaknesses.

Summary

• Perceptual Fusion (Tp)
• Bottom-up/Top-down processing
• Chunking (recognize patterns from past experience)
• Spotlight metaphor (attention and perception), visual

dominance

• Stroop affect (choose secondary characteristics to

reinforce)

• Cognitive processing time increases with number of

stimuli, decisions.

Summary

• Accuracy varies with reaction time
• Speed vs. accuracy tradeoff
• Multitasking (task structure, difficulty)
• Fitt's Law
• Power Law of Practice
• Working memory 7 +/- 2
• Long Term memory (rehersal WM->LTM)