THE STATE OF SPEECH IN HCI: TRENDS, THEMES & CHALLENGES - PowerPoint PPT Presentation

THE STATE OF SPEECH IN HCI: TRENDS, THEMES & CHALLENGES @lmhclark @cogsis 
 LEIGH CLARK @hci_ucd UNIVERSITY COLLEGE DUBLIN The CogSIS Project

UPCOMING EVENTS Measuring and designing trust in Human-Agent Interaction workshop Before HAI 2018 Conference: 15th December Southampton, UK https://sites.google.com/view/mdt-hai2018/ Conversational user interface (CUI) conference August 2019 Dublin, Ireland Details TBC

Ben Cowan Philip Doyle Diego Garaialde

Emer Gilmartin Stephan Schlögl Jens Edlund Trinity College Dublin MCI Centre Innsbruck KTH Stockholm Cosmin Munteanu Matthew Aylett João Cabral University of Toronto CereProc Ltd Trinity College Dublin Mississauga

https://www.amazon.com/AmazonBasics-Microwave-Compact-Works-Alexa/ dp/B07894S727

RESEARCH AIMS MAP OUT: PUBLICATION TRENDS RESEARCH METHODS RESEARCH THEMES

SEARCH TERMS & DATABASES ABSTRACT, TITLE & KEYWORD SEARCH speech interface; voice user interface; voice system; human computer dialog*; human machine dialog*; natural language dialog* system; natural language interface; conversational interface; conversational agent; conversational system; conversational dialog* system; automated dialog* system; interactive voice response system; spoken dialog* system; spoken human machine interaction; human system dialog*; intelligent personal assistant +

INCLUSION/EXCLUSION CRITERIA 1181 INCLUDE EXCLUDE Speech focused Embodiment Full No interaction conference / evaluation journal papers Non-full / non- English peer reviewed 68

RESEARCH METHODS

DIRECTION OF COMMUNICATION User-system User input only System output dialogue (44) (16) only (12)

CONCEPTS MEASURED User attitudes 36 Task performance 33 Lexis & syntax 20 Perceived usability 18 System usage 15 User recall 7 Physiological data 3 Other 11

RESEARCH THEMES

SYSTEM SPEECH PRODUCTION Synthesis 8 Content 7

MODALITY COMPARISON Keyboard and/or 10 mouse Digital pen 3

USER SPEECH PRODUCTION General production 3 Addressee 2 identification Alignment 1

ASSISTIVE TECHNOLOGY & ACCESSIBILITY Tabletop designs - physicians, deaf patients & interpreters Mobile interface - limited hand dexterity Voiced-based browser plugin - blind users https://www.nationaldeafcenter.org/topics/assistive-technology

DESIGN INSIGHT Early design insight - speech to access GUI-based software Interface for a large-scale game

IPA EXPERIENCE Disparity between people’s mental models of IPAs & reality of interaction Human likeness can negatively affect IUX Embarrassment of public use Structure of multiple user interaction w/ Siri

CHALLENGES & ONGOING RESEARCH

CHALLENGES & ONGOING RESEARCH MORE THEORETICAL UNDERSTANDING FOR: 1. LANGUAGE PRODUCTION TO SYSTEMS 2. PERCEPTION OF SYSTEMS 3. DESIGN IN LIGHT OF THESE

Global Partner models

Local Partner models Knowledge Trust Style Colloquial Universal Social Reliability Functional Reliability Casual Formal

Proliferation of humanlike voices in non-human artefacts can create unrealistic expectations of capabilities Moore, R. K. (2017). Appropriate Voices for Artefacts: Some Key Insights. In 1st International Workshop on Vocal Interactivity in-and-between Humans, Animals and Robots.

POLITENESS & FACE Politeness linked to concept of face (Goffman, 1952; 1967) Social self-image dependent on societal norms and rules Usually best interest to save face

EXAMPLE No politeness Connect… Give each piece a twist… Attach… …so it’s in line with the feet Locate… …so the end is closest to the top of the body Politeness Just connect…. Just give each piece a little bit of a twist… Basically, attach…. …so it’s more or less in line with the feet Now just locate…. …the end should be closest to the top of the body

https://fineartamerica.com/featured/i-dont-care-if-she-is-a-tape-dispenser-i-love-sam-gross.html

KEY POINTS 1. Speech HCI fragmented 2. More theoretical development/application can help improve cohesion in the field 3. Theories can help explain & understand, but can also be redefined and re- conceptualised in HCI 4. Current work at HCI @ UCD looking at UCD looking design choices affecting partners models, language production, user perception

RELEVANT PAPERS Clark, L., Doyle, P., Garaialde, D., Gilmartin, E., Schlögl, S., Edlund, J., ... & Cowan, B. (2018). The State of Speech in HCI: Trends, Themes and Challenges. arXiv preprint arXiv:1810.06828. Murad, C., Munteanu, C., Clark, L., & Cowan, B. R. (2018). Design guidelines for hands-free speech interaction. Mobile HCI 2018 Adjunct (pp. 269-276). ACM. Clark, L. (2018). Social boundaries of appropriate speech in HCI: a politeness perspective. BCS HCI 2018. Clark, L., Cabral, J. & Cowan, B.R. (2018). The CogSIS Project: Examining the Cognitive Effects of Speech Interface Synthesis. BCS HCI 2018. Large, D. R., Clark, L., Quandt, A., Burnett, G., & Skrypchuk, L. (2017). Steering the conversation: a linguistic exploration of natural language interactions with a digital assistant during simulated driving. Applied Ergonomics, 63, 53-61. Clark, L., Ofemile, A., Adolphs, S. & Rodden, T. (2016). A Multimodal Approach to Assessing User Experiences with Agent Helpers. ACM Transactions on Interactive Intelligent Systems (TIIS), 6(4) 29. Clark, L. M. H., Bachour, K., Ofemile, A., Adolphs, S. & Rodden, T. (2014). Potential of Imprecision: Exploring Vague Language in Agent Instructors. HAI 2014. Tsukuba, Japan, ACM: 339-344. @lmhclark leigh.clark@ucd.ie @cogsis 
 @hci_ucd