Evaluating Haptic and Auditory Guidance to Assist Blind People in - - PowerPoint PPT Presentation

▶
evaluating haptic and auditory guidance to assist blind
SMART_READER_LITE
LIVE PREVIEW

Evaluating Haptic and Auditory Guidance to Assist Blind People in - - PowerPoint PPT Presentation

Dec 25, 2022 40 likes •876 views

Evaluating Haptic and Auditory Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras TACCES ESS | ASSET ETS 2016 Lee Stearns 1 , Ruofei Du 1 , Uran Oh 1 , Catherine Jou 1 , Leah Findlater 2 , David A. Ross 3 , Jon

slide-1
SLIDE 1

Lee Stearns1, Ruofei Du1, Uran Oh1, Catherine Jou1, Leah Findlater2, David A. Ross3, Jon E. Froehlich1

University of Maryland: Computer Science1, Information Studies2, Atlanta VA R&D Center for Visual & Neurocognitive Rehabilitation3

Evaluating Haptic and Auditory Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

TACCES ESS | ASSET ETS 2016

slide-2
SLIDE 2

What if printed text could be accessed

through touch in the same way as braille?

*Video Credit: YouTube—Ginny Owens—How I See It (Reading Braille)

slide-3
SLIDE 3

What if printed text could be accessed

through touch in the same way as braille?

slide-4
SLIDE 4

What if printed text could be accessed

through touch in the same way as braille? Reading printed materials is still an important but

challenging task for people with visual impairments

slide-5
SLIDE 5

POPULAR READING DEVICES

slide-6
SLIDE 6

POPULAR READING DEVICES

Scanner | OCR | Screen Reader

slide-7
SLIDE 7

POPULAR READING DEVICES

Dedicated devices (e.g., video magnifiers)

slide-8
SLIDE 8

POPULAR READING DEVICES

Smartphone apps (e.g., KNFB Reader iOS)

slide-9
SLIDE 9

POPULAR READING DEVICES

Wearable Cameras (e.g., OrCam)

slide-10
SLIDE 10

Scanner | OCR | Screen Reader

Dedicated Devices (e.g., video magifiers) Smartphone Apps (e.g., KNFB Reader iOS) Wearable Cameras (e.g., OrCam)

POPULAR READING DEVICES

slide-11
SLIDE 11

Open Questions (Existing Devices)

  • 1. How to assist with aiming the camera

to capture desired content?

slide-12
SLIDE 12

Open Questions (Existing Devices)

  • 1. How to assist with aiming the camera

to capture desired content?

  • 2. How to handle complex documents

and convey layout information?

slide-13
SLIDE 13

HANDSIGHT

A vision-augmented touch system

slide-14
SLIDE 14

HANDSIGHT

A vision-augmented touch system Tiny CMOS cameras,

slide-15
SLIDE 15

HANDSIGHT

Tiny CMOS cameras, haptic actuators mounted

  • n one or more fingers

A vision-augmented touch system

slide-16
SLIDE 16

HANDSIGHT

Tiny CMOS cameras, haptic actuators mounted

  • n one or more fingers

Smartwatch for power, processing, speech and audio output A vision-augmented touch system

slide-17
SLIDE 17

HANDSIGHT

Tiny CMOS cameras, haptic actuators mounted

  • n one or more fingers

Smartwatch for power, processing, speech and audio output A vision-augmented touch system

* Originally proposed in Stearns et al. 2014

slide-18
SLIDE 18

AUGMENTING THE USER’S FINGER

Survey: Digital Digits (Shilkrot et al. 2015)

slide-19
SLIDE 19

Camera & Optical Mouse Sensor

AUGMENTING THE USER’S FINGER

Magic Finger (Yang et al. 2012)

slide-20
SLIDE 20

Camera and Vibration Motor

AUGMENTING THE USER’S FINGER

FingerReader (Shilkrot et al. 2014, 2015)

slide-21
SLIDE 21

Camera Vibration Motors

AUGMENTING THE USER’S FINGER

HandSight (Stearns et al. 2014)

Processing+Power

slide-22
SLIDE 22

Advantages of Finger-Based Reading

  • 1. Does not require framing an overhead camera
slide-23
SLIDE 23

Advantages of Finger-Based Reading

  • 1. Does not require framing an overhead camera
  • 2. Allows direct access to spatial information
slide-24
SLIDE 24

Advantages of Finger-Based Reading

  • 1. Does not require framing an overhead camera
  • 2. Allows direct access to spatial information
  • 3. Provides better control over pace and rereading
slide-25
SLIDE 25

Advantages of Finger-Based Reading

  • 1. Does not require framing an overhead camera
  • 2. Allows direct access to spatial information
  • 3. Provides better control over pace and rereading
slide-26
SLIDE 26

Advantages of Finger-Based Reading

  • 1. Does not require framing an overhead camera
  • 2. Allows direct access to spatial information
  • 3. Provides better control over pace and rereading

New Challenges

  • 1. How to precisely trace a line of text?
  • 2. How to support physical navigation?
slide-27
SLIDE 27

COMPARING TWO TYPES OF

DIRECTIONAL FINGER GUIDANCE

  • 2. Audio via built-in
  • r external speakers
  • 1. Finger-mounted

vibration motors

slide-28
SLIDE 28
  • 1. Finger-mounted

vibration motors

COMPARING TWO TYPES OF

DIRECTIONAL FINGER GUIDANCE

slide-29
SLIDE 29

Move up

  • 1. Finger-mounted

vibration motors

COMPARING TWO TYPES OF

DIRECTIONAL FINGER GUIDANCE

slide-30
SLIDE 30

Move down

  • 1. Finger-mounted

vibration motors

COMPARING TWO TYPES OF

DIRECTIONAL FINGER GUIDANCE

slide-31
SLIDE 31
  • 2. Audio via built-in
  • r external speakers

COMPARING TWO TYPES OF

DIRECTIONAL FINGER GUIDANCE

slide-32
SLIDE 32

Higher pitch: move up

  • 2. Audio via built-in
  • r external speakers

COMPARING TWO TYPES OF

DIRECTIONAL FINGER GUIDANCE

slide-33
SLIDE 33

Lower pitch: move down

  • 2. Audio via built-in
  • r external speakers

COMPARING TWO TYPES OF

DIRECTIONAL FINGER GUIDANCE

slide-34
SLIDE 34

Research Questions

  • 1. T
  • what extent are finger-based cameras a viable

accessibility solution for reading printed text?

  • 2. What design choices can improve this viability?
slide-35
SLIDE 35

Study Overview

Study I: initial iPad study (19 participants)

Study II: physical prototype study (4 participants)

slide-36
SLIDE 36

Study I: initial iPad study (19 participants)

Study II: physical prototype study (4 participants)

Study Overview

slide-37
SLIDE 37

Study I: initial iPad study (19 participants)

Study Overview

Study I: initial iPad study (19 participants) Goals: Compare audio/haptic Explore & interpret spatial layouts Assess reading and comprehension

Study Overview

slide-38
SLIDE 38

Study I Method

Used an iPad to focus on user experience, gather finger trace data

slide-39
SLIDE 39

Study I Method

Used an iPad to focus on user experience, gather finger trace data

19 participants

Median Age 48 (SD=12, Range=26-67) Gender 11 Male, 8 Female Vision Level 10 T

  • tally Blind, 9 Light Sensitive
slide-40
SLIDE 40

Study I Method

Used an iPad to focus on user experience, gather finger trace data 19 participants

Within-subjects, two guidance

conditions: audio and haptic

Haptic vibrations Audio pitch

slide-41
SLIDE 41

Study I Method

Used an iPad to focus on user experience, gather finger trace data 19 participants Within-subjects, two guidance conditions: audio and haptic

Participants read two documents for each condition

plain magazine

slide-42
SLIDE 42

Study I Method

Used an iPad to focus on user experience, gather finger trace data 19 participants Within-subject, two guidance conditions: audio and haptic Participants read two documents for each condition

Analysis: reading speed and accuracy, comprehension, subjective feedback

audio haptic

slide-43
SLIDE 43

Exploration Mode Reading Mode

System Design: Exploration and Reading Modes

slide-44
SLIDE 44

Continuous audio feedback to identify content beneath finger Flute sound: text Cello sound: picture Silence: empty space Same across both conditions

System Design: Exploration Mode

slide-45
SLIDE 45

Flute sound: text Silence: empty space Cello sound: picture

slide-46
SLIDE 46

Bimanual: right index finger to read, left to anchor start of line Directional guidance: audio or haptic depending on condition Used to stay on the line or find the start of the next line Audio: pitch of continuous audio Haptic: strength and position of vibration Additional audio cues (same for both conditions) Start/end of line or paragraph Synthesized speech

System Design: Reading Mode

slide-47
SLIDE 47

Above the line: downward guidance (low pitch or lower vibration motor) Below the line: upward guidance (high pitch or upper vibration motor) Start/end of line or paragraph (short but distinctive audio cues)

slide-48
SLIDE 48

Study I Findings

Haptic vs. Audio: Quantitative Performance

slide-49
SLIDE 49

Study I Findings

Haptic vs. Audio: Quantitative Performance

Line tracing / magazine documents: Audio significantly more accurate (p = 0.018)

slide-50
SLIDE 50

Study I Findings

Haptic vs. Audio: Quantitative Performance

Line tracing / magazine documents: Audio significantly more accurate (p = 0.018)

audio haptic

Example finger traces—Dashed red lines mark drift off of the line

slide-51
SLIDE 51

Study I Findings

Haptic vs. Audio: Quantitative Performance

Line tracing / magazine documents: Audio significantly more accurate (p = 0.018)

Comprehension high, no significant differences between conditions

audio haptic

Example finger traces—Dashed red lines mark drift off of the line

slide-52
SLIDE 52

Study I Findings

Haptic vs. Audio: Subjective Preference

Preferences split (11 haptic, 7 audio, 1 equal preference)

slide-53
SLIDE 53

Study I Findings

Haptic vs. Audio: Subjective Preference

Preferences split (11 haptic, 7 audio, 1 equal preference)

Preferred Haptic

More intuitive Easier to use Faster Less distracting

slide-54
SLIDE 54

Study I Findings

Haptic vs. Audio: Subjective Preference

Preferences split (11 haptic, 7 audio, 1 equal preference)

Preferred Haptic Preferred Audio

More intuitive Easier to use Faster Less distracting Less confusing More comfortable No desensitization

slide-55
SLIDE 55

Study I Findings

Haptic vs. Audio: Subjective Preference

Preferences split (11 haptic, 7 audio, 1 equal preference)

Preferred Haptic Preferred Audio

More intuitive Easier to use Faster Less distracting Less confusing More comfortable No desensitization

Reflects contradictory findings in Stearns et al. 2014, Shilkrot et al. 2014, 2015

slide-56
SLIDE 56

Study I Findings

Overall Reading Experience

Pros

Low learning curve Flexible Direct control over speed

slide-57
SLIDE 57

Study I Findings

Overall Reading Experience

Pros Cons

Low learning curve Flexible Direct control over speed Hard to use for reading High cognitive load may affect comprehension

slide-58
SLIDE 58

Study I Findings

Exploration Mode

Participants appreciated direct access to spatial information, and

nearly all able to locate images and count the number of columns.

slide-59
SLIDE 59

Study I: initial iPad study (19 participants)

Study II: physical prototype study (4 participants)

Study Overview

slide-60
SLIDE 60

Study I: initial iPad study (19 participants)

Study II: physical prototype study (4 participants)

Study Overview

slide-61
SLIDE 61

Goals:

Evaluate HandSight prototype

Gather subjective feedback

Compare with KNFB Reader iOS

Study II: physical prototype study (4 participants)

Study Overview

slide-62
SLIDE 62

Finger-mounted camera to read physical documents

Study II: HandSight Prototype System

slide-63
SLIDE 63

Study II Method

HandSight:

Each participant used their preferred guidance from Study I to explore and read two documents

slide-64
SLIDE 64

Study II Method

KNFB Reader iOS:

Photograph and read 3 physical documents

slide-65
SLIDE 65

Study II Findings

HandSight: Overall Experience

Average reading speed: 45 wpm (SD=19, Range=18-60) Rated somewhat easy to use, but slow and required concentration

slide-66
SLIDE 66

Study II Findings

HandSight: Overall Experience

Average reading speed: 45 wpm (SD=19, Range=18-60) Rated somewhat easy to use, but slow and required concentration

Participant Quotes: “I’m very pleased and excited about the system. I think it could make a great difference in my life.” (P19) “It seems like a lot of effort for reading text.” (P12)

slide-67
SLIDE 67

Study II Findings

HandSight vs. KNFB Reader iOS

Participants unanimously preferred KNFB Reader iOS

HandSight KNFB Reader iOS

slide-68
SLIDE 68

Study II Findings

HandSight vs. KNFB Reader iOS

Participants unanimously preferred KNFB Reader iOS

Faster, easier to concentrate on the content of the text

HandSight KNFB Reader iOS

slide-69
SLIDE 69

Implications

Pros

Feasibility of a Finger-Based Reading Approach

slide-70
SLIDE 70

Implications

Pros

Spatial layout information

Feasibility of a Finger-Based Reading Approach

slide-71
SLIDE 71

Implications

Pros

Spatial layout information Direct control over reading

Feasibility of a Finger-Based Reading Approach

slide-72
SLIDE 72

Implications

Pros

Spatial layout information Direct control over reading Reduced camera framing issues

Feasibility of a Finger-Based Reading Approach

slide-73
SLIDE 73

Implications

Pros

Spatial layout information Direct control over reading Reduced camera framing issues

Efficient text detection and recognition

Feasibility of a Finger-Based Reading Approach

slide-74
SLIDE 74

Implications

Pros

Spatial layout information Direct control over reading Reduced camera framing issues

Efficient text detection and recognition

* W * We o

  • bserved t

these i in o

  • ur s

studies

Feasibility of a Finger-Based Reading Approach

slide-75
SLIDE 75

Implications

Feasibility of a Finger-Based Reading Approach

Pros Cons

Spatial layout information Direct control over reading Reduced camera framing issues Efficient text detection and recognition

* W * We o

  • bserved t

these i in o

  • ur s

studies

Slower, requires increased concentration and physical dexterity

slide-76
SLIDE 76

Implications

Feasibility of a Finger-Based Reading Approach

Pros Cons

Spatial layout information Direct control over reading Reduced camera framing issues Efficient text detection and recognition

* W * We o

  • bserved t

these i in o

  • ur s

studies

Slower, requires increased concentration and physical dexterity

* Co * Consistent wi with Sh Shilk ilkrot et et al. 2014, 2014, 2015 2015

slide-77
SLIDE 77

Implications

Feasibility of a Finger-Based Reading Approach

Pros Cons

Spatial layout information Direct control over reading Reduced camera framing issues Efficient text detection and recognition

* W * We o

  • bserved t

these i in o

  • ur s

studies

Slower, requires increased concentration and physical dexterity

* Co * Consistent wi with Sh Shilk ilkrot et et al. 2014, 2014, 2015 2015

Importance of spatial layout information is unclear

slide-78
SLIDE 78

Implications

Feasibility of a Finger-Based Reading Approach

Pros Cons

Spatial layout information Direct control over reading Reduced camera framing issues Efficient text detection and recognition

* W * We o

  • bserved t

these i in o

  • ur s

studies

Slower, requires increased concentration and physical dexterity

* Co * Consistent wi with Sh Shilk ilkrot et et al. 2014, 2014, 2015 2015

Importance of spatial layout information is unclear

* Ha * Has y yet t to b be i investigated i in t this c context

slide-79
SLIDE 79

Future Work

Study utility of spatial layout information in everyday use (e.g., newspapers, menus, maps, graphs)

slide-80
SLIDE 80

Future Work

Study utility of spatial layout information Explore possibilities for camera placement

slide-81
SLIDE 81

HANDSIGHT

a a vision au augm gmented d touch system

slide-82
SLIDE 82

Questions?

Contact: lstearns@umd.edu

Thank you to our participants and the Maryland State Library for the Blind and Physically Handicapped. This research was funded by the Department of Defense.

Lee Stearns1, Ruofei Du1, Uran Oh1, Catherine Jou1, Leah Findlater2, David A. Ross3, Jon E. Froehlich1

University of Maryland: Computer Science1, Information Studies2, Atlanta VA R&D Center for Visual & Neurocognitive Rehabilitation3

Evaluating Haptic and Auditory Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

TACCESS | ASSETS 2016

slide-83
SLIDE 83

Limitations of previous studies*

  • 1. Small sample size (3-4 participants)
  • 2. No quantitative performance metrics
  • 3. Contradictory participant preferences

* Stearns et al. 2014, Shilkrot et al. 2014, 2015