Italicizing Optional Instructions on Mobile Online Surveys Improves - PowerPoint PPT Presentation

Italicizing Optional Instructions on Mobile Online Surveys Improves Visual Filtering of Survey Content: An Eye Tracking Study Brian Falcone, Lawrence Malakhoff, Lin Wang Presenter: Alda Rivas U.S. Census Bureau AAPOR 73rd Annual Conference May 17 2018 Denver, Colorado This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau. 1

Two studies: First study: Effect of typographic cues on efficiency and accuracy with which respondents complete a survey on a mobile device Follow-up study: Incorporation of eye tracking to explore the mechanism through which typographic cues affect efficiency. 2

What is Typographic Cueing? • Reveals text content structure through changes in weight, size, case, typeface, etc. (Keyes, 1993) Bold Question stem • Differentiates information categories Italicized instructions  Use dark print for questions and light print for response choices (Dilman, 2007)  Separate optional or occasionally needed instructions from the question statement by font or symbol variations (Dilman, 2007) 3

Motivation Gap of empirical evidence – Support of theory. – Generalization of benefits of typographic cues to mobile interface. Main research question: – Does italicizing optional instructions make them more distinguishable? • Participants who are presented with optional instructions will have shorter completion time on task because they are filtering out this information. 4

Methodology Question stem Bolded Not Bolded (plain) • 2 x 2 Between-subjects design Italicized Bolded stem + Plain stem + Italicized • 30 participants Instructions Italicized instructions instructions ( n = 7) ( n = 8) • Experimental task: Bolded + Plain Plain stem + Plain Not – 5 Question survey instructions ( n = 8) instructions Italicized • 4 different conditions ( n = 7) (plain) – Survey completion time A: Bold + Plain B: Bold + Italics C: Plain + Italics D: Plain + Plain • Word recognition task: – 10 words seen in survey – 10 words NOT seen in survey – Recognition score 5

Survey Completion Time Italicized instructions result in faster survey completion times compared to plain (regular) text. Italicized Instructions 100 Plain Text Instructions Average Survey Completion Time 90 80 Bold – F( 3,26)=.04, p = ns 70 • Ital – F( 3,26)=4.94, p < .05 • (Seconds) 60 Bold x Ital – F (3,26)= .00 , p = ns 62.20 • 61.15 50 40 45.56 43.51 30 20 10 0 Bolded Q-Stem Plain Text Q-Stem 6

Word Recognition Neither bold q-stem nor italicized instructions significantly affected the amount of words recognized. Italicized Instructions 70% Average Corrected Recognition Score (HR - FA) Plain Text 60% Instructions Bold - F (3,26)= .26, p = n.s. • Italics – F (3,26)= .97, p = n.s. • 50% Bold x Ital – F(3,26)= .01, p = n.s. • 40.24% 40% 34.89% 30.85% 30% 27.48% 20% 10% 0% Bolded Q-Stem Plain Text Q-Stem 7

Conclusions and Limitations – Implementing a typographic cue (italicizing) for optional instructions does result in shorter completion time. – Utilized an indirect measure of visual attention (word recognition list). – Could not determine WHY individuals who used a survey designed with italicized instructions were faster. 8

Follow-up Experiment Research question and hypothesis – WHY did individuals who used a survey designed with italicized instructions complete surveys faster? • Italicized optional instructions results in lower visual attention and faster survey completion times – Use a direct measure of visual attention by incorporating eye-tracking methodology 9

Methodology • Participants: 16 Census Bureau Employees or Contractors – Setting: In-person tests at Census Bureau Headquarters in Suitland, Maryland • Task: Simulated Mobile Survey on PC using Interactive PowerPoint – Same 5 survey questions from previous behavioral study • Between-subjects design: 2 conditions – Italicized Instructions VS Plain Text Instructions 10

Eye Tracking Metrics and Analysis • Metrics: – Total Fixation Duration – Fixation Count 11

Eye Tracking Metrics and Analysis • Analysis: – Area of Interest (AOI) Analysis: • Defined visual area for instructional text for each survey question. Captures metrics within this AOI – Summed total fixation duration and fixation count across all five survey questions for each participant 12

Preliminary data: Fixation Count and Total Fixation Duration Average Fixation Count Between Conditions • Results: 42.14285714 45 40 Average Fixation Count – Average Fixation Count was 35 30 nearly double when the 25 20.96 20 instructions were plain text 15 10 5 0 Italicized Instructions Plain Instructions – Average Total Fixation Condition Duration was nearly double Average Total Fixation Duration Between when the instructions were Conditions plain text 12 Average Fixation Duration (Sec) 9.79 10 8 6 4.332 4 2 0 Italicized Instructions Plain Instructions Condition 13

Gaze Plots • Gazeplots: – Gaze path and fixation duration for 3 participants from both conditions on the final survey question – Much more time was spent reading the instructions in the plain text group Italicized Instructions (n=3) Plain Instructions (n=3) 14

Heatmaps • Fixation Count Heatmaps: – There is a clearly larger area of the instructional text being attended to by participants who saw plain text instructions Italicized Instructions (n=3) Plain Instructions (n=3) 15

Heatmaps • Fixation Duration Heatmaps: – There seems to be more time being spent fixating on the area of the instructional text being attended to by participants who saw plain text instructions vs italicized instructions Italicized Instructions (n=3) Plain Instructions (n=3) 16

Conclusions from preliminary data Theoretical implications: • More time is spent reading the instructional text in the plain text condition – This supports the theory that information deemed unneeded by the respondent is being visually filtered or skipped when text is italicized. Practical implications: – Evidence that typographic principles are applicable to mobile design. Incorporating eye tracking to the design allowed us to directly observe the source of a cognitive benefit afforded by a design using typographic cues. 17

Acknowledgement • Chistopher Antoun – for major contributions to the design and analysis of the initial behavioral study that led to this follow-up research and helping with data collection • Elizabeth Nichols and Erica Olmsted-Hawala – for contributing their knowledge and expertise towards the design of the initial behavioral study. Also, a special thanks to Elizabeth for securing a booth for the Human Factors and Usability Research (HFUR) group at the Census Innovation Fair where a bulk of the ET data was collected. • All Census Employees who volunteered to participate • MetroStar for software development • Sabin Lakhe for technical support 18