Psychology-Driven Design of Intelligent Interfaces T. Metin Sezgin - - PowerPoint PPT Presentation

▶
psychology driven design of intelligent interfaces
SMART_READER_LITE
LIVE PREVIEW

Psychology-Driven Design of Intelligent Interfaces T. Metin Sezgin - - PowerPoint PPT Presentation

Feb 14, 2023 307 likes •1.25k views

Psychology-Driven Design of Intelligent Interfaces T. Metin Sezgin Assoc. Prof. College of Engineering Ko University http://iui.ku.edu.tr mtsezgin@alum.mit.edu BYOYO 01/07/20 Intelligent User Interfaces Group Dr. Metin Sezgin,

slide-1
SLIDE 1
  • T. Metin Sezgin
  • Assoc. Prof.

College of Engineering 
 Koç University 


http://iui.ku.edu.tr mtsezgin@alum.mit.edu BYOYO 01/07/20

Psychology-Driven Design of Intelligent Interfaces

slide-2
SLIDE 2
  • Dr. Metin Sezgin, Assoc. Prof.

Intelligent User Interfaces Group

▪ Areas of expertise

▪ Intelligent User Interfaces ▪ Machine learning ▪ Multimodal interfaces

MIT (MS ‘01) MIT (PhD ‘06) postdoc visiting appointments 2010 -- …

▪ 20+ graduate students ▪ ~15 TL million sponsored projects

▪International

▪European Union ▪CHIST-ERA ▪DARPA

▪National

▪Research Council of Turkey ▪Ministry of Science, Industry & Tech.

▪Industrial

▪Türk Telekom ▪Koç Sistem

slide-3
SLIDE 3

History of Human Computer Interaction

2007 ENIAC 1946 1989 Mac Portable 1981 IBM PC iPhone

Cheaper: 13000 times Smaller: 986530 times Faster: 6077922 times

10

17 Fold

cost of flops/grams

slide-4
SLIDE 4

Attempts at intelligent interaction

Television Control by Hand Gestures William T. Freeman, Craig D. Weissman
 MERL Report: TR94-24

slide-5
SLIDE 5

Attempts at intelligent interaction

Freeman ’94

Attempts at intelligent interaction have failed! Solution: leverage natural human behavior

Unidentified Samsung User ’14

slide-6
SLIDE 6

The Problem


Too little effort towards understanding interaction

slide-7
SLIDE 7

Strategy: Leverage natural human behavior

HCI

slide-8
SLIDE 8

Strategy: Leverage natural human behavior

HCI Machine learning

slide-9
SLIDE 9

Strategy: Leverage natural human behavior

HCI Machine learning Psychology

slide-10
SLIDE 10

Strategy

▪ Understand the human ▪ Perception ▪ Behaviour ▪ Computational models of ▪ Human perception ▪ Human behavior (intent) ▪ Build novel interfaces (HW & SW) ▪ Natural ▪ Intelligent ▪ Multimodal

slide-11
SLIDE 11

Case #1 ▪ Exercise

▪ Draw objects

▪ Observe human behavior

slide-12
SLIDE 12

Observe human behavior

slide-13
SLIDE 13

Case #1 ▪ Exercise

▪ Draw objects

▪ Observe human behavior ▪ Practical use

▪ Sketch recognition ▪ Auto-completion of drawings

slide-14
SLIDE 14

Online Sketch Recognition

slide-15
SLIDE 15

Offline Sketch Recognition

Work Funded under the National Science Foundation Priority Areas Call

slide-16
SLIDE 16

Auto-completion

slide-17
SLIDE 17
  • T. M. Sezgin and R. Davis, Sketch Recognition in Interspersed Drawings Using Time-

Based Graphical Models. Computers & Graphics Journal, Volume 32 , Issue 5, pp: 500-510 (2008). 
 Ç. Tırkaz, B. Yanıkoğlu, T. M. Sezgin, Sketched Symbol Recognition with Auto

  • Completion. Pattern Recognition, vol 45, issue 11, pp 3926-3937 (2012).

Auto-completion

slide-18
SLIDE 18

Auto-completion

▪ Drives multimedia retrieval UI ▪ iMotion European Commission ERA-NET Project

▪ U. Basel (Switzerland) ▪ U. Mons (Belgium)

Retrieval Engine

Grant: European Commission ERA-Net Program, CHIST-ERA Intelligent User Interfaces Call Intelligent Multimodal Augmented Video Motion Retrieval System

slide-19
SLIDE 19

Strategy: Leverage natural human behavior

slide-20
SLIDE 20

Case #2 ▪ Exercise

▪ Object manipulation

slide-21
SLIDE 21

Object Manipulation Virtual Interaction Task – Drag

slide-22
SLIDE 22

Virtual Interaction Task – Maximize

slide-23
SLIDE 23

Virtual Interaction Task – Minimize

slide-24
SLIDE 24

Virtual Interaction Task – Scroll

slide-25
SLIDE 25

Virtual Interaction Task – Free-Form Drawing

  • Ππ
slide-26
SLIDE 26

Virtual Interaction Task: Your turn

slide-27
SLIDE 27

Case #2 ▪ Exercise

▪ Object manipulation

▪ Observe human behavior

slide-28
SLIDE 28

Case #2

slide-29
SLIDE 29

Case #2

slide-30
SLIDE 30

Case #2

Ç. Çığ, T . M. Sezgin, Gaze-Based Prediction of Pen-Based Virtual Interaction

  • Tasks. International Journal of Human-Computer Studies, (2014).

European Patent Application, T . M. Sezgin, Ç. Çığ, Gaze Based Prediction Device, PCT/TR2014/00189, European Patent Office, May 2014.

slide-31
SLIDE 31

Case #2 ▪ Exercise

▪ Manipulate objects

▪ Observe human behavior ▪ Practical use

▪ Proactive UIs ▪ Intent recognition ▪ Fat finger problem

slide-32
SLIDE 32

Novel use of eye gaze
 How do I detect recognition errors?

slide-33
SLIDE 33

Novel use of eye gaze

▪ Immediate return to the misrecognition

slide-34
SLIDE 34

Novel use of eye gaze

▪ Immediate return to the misrecognition ▪ Double take at the misrecognition

slide-35
SLIDE 35

Novel use of eye gaze

▪ Immediate return to the misrecognition ▪ Double take at the misrecognition

slide-36
SLIDE 36

Research highlights

▪ Recognition technologies ▪ Perception-based ▪ Machine learning ▪ Multimodal interaction ▪ Development ▪ Evaluation

slide-37
SLIDE 37

Recognizing Sketches

Grant: Funded under the National Science Foundation Priority Areas Call

slide-38
SLIDE 38

Learning a scale of messiness

vs.

slide-39
SLIDE 39

Recognition with few examples, scarce resources

CLUSTER A CLUSTER C CLUSTER B

▪ Modeling styles ▪ Active learning ▪ Zero shot learning ▪ Self learning

slide-40
SLIDE 40

Pen-based interfaces

▪ Design ▪ E-learning ▪ Animation ▪ Entertainment

slide-41
SLIDE 41

Multimodal Storyboarding Assistant

slide-42
SLIDE 42

Intelligent User Interfaces

smart stylus rehabilitation of autism conditions multimedia retrieval gaze-based intent recognition affective robotics HRI

S E A R C H

slide-43
SLIDE 43

Grant: European Commission ERA-Net Program, CHIST-ERA Intelligent User Interfaces Call Joke and Empathy of a Robot/ECA: Towards Social and Affective Relations with a Robot

Affective interaction with robots

▪ Robots with a sense of humor ▪ JOKER – European Commission ERA-NET Project

▪ LIMSI/CNRS (France) ▪ Trinity College Dublin (Ireland) ▪ University of Mons (Belgium)

slide-44
SLIDE 44

Affective interaction with robots

Grant: European Commission ERA-Net Program, CHIST-ERA Intelligent User Interfaces Call Joke and Empathy of a Robot/ECA: Towards Social and Affective Relations with a Robot

slide-45
SLIDE 45

Affective interaction with robots

Grant: European Commission ERA-Net Program, CHIST-ERA Intelligent User Interfaces Call Joke and Empathy of a Robot/ECA: Towards Social and Affective Relations with a Robot

slide-46
SLIDE 46

Learning, vision, language

deep stroke segmentation learning visual attributes learning from few examples active learning explainable AI shape retrieval

slide-47
SLIDE 47

Looking forward

Medicine Social Sciences Arts

slide-48
SLIDE 48

Alumni Profiles

  • Dr. Yusuf Sahillioğlu, Visiting Researcher 

  • Assoc. Prof., Middle East Technical Univ.
  • Dr. Başak Alper, Postdoc


NASA - Jet Propulsion Laboratory Neşe Alyüz Çivitci, Postdoc
 Intel Labs, Intel Corporation Senem Ezgi Emgin, PhD Student Apple Zana Buçinca, MS Student Harvard University Çağlar Tırkaz, PhD Student Amazon Ayşe Küçükyılmaz, PhD Student Nottingham University (Asst. Prof.) Kurmanbek Kaiyrbekov, MSc Student John Hopkins University Cansu Şen, MSc Student University of Massachusetts Med. School Tuğrulcan Elmas, Summer Researcher École Polytech. Fédérale de Lausanne Arda İçmez, Summer Researcher Facebook Mustafa Emre Acer, Summer Researcher Google

slide-49
SLIDE 49

Acknowledgements

▪ Postdocs

▪ Basak Alper ▪ Nese Alyuz ▪ Yusuf Sahillioglu

▪ PhD students

▪ Sinan Tumen ▪ Berker Turker ▪ Ayse Kucukyilmaz ▪ Caglar Tirkaz ▪ Cagla Cig ▪ Ezgi Emgin

▪ MS students

▪ Serike Cakmak ▪ Ozem Kalay ▪ Cansu Sen ▪ Erelcan Yanik ▪ Atakan Arasan ▪ Banucicek Gurcuoglu ▪ Kemal Tugrul

▪ Undergraduate students

▪ Anil Uluturk ▪ Furkan Bayraktar ▪ Ozan Okumusoglu ▪ 30+

▪ Collaborators

▪ Berrin Yanikoglu ▪ Engin Erzgin ▪ Yucel Yemez ▪ Cagatay Basdogan

▪ Sponsors

▪ DARPA ▪ The European Commission ▪ TÜBİTAK ▪ Türk Telekom ▪ Koç Sistem ▪ Ministry of Science 
 Industry & Technology

slide-50
SLIDE 50

Questions

slide-51
SLIDE 51

Questions

slide-52
SLIDE 52

References

Invention Disclosures

Under review, O. Kalay., T. M. Sezgin, BBF # 2014.10.X 
 Koç University, Research, Project Development and Technology Transfer Directorate Gaze-Based Mode Inference for Pen-Based Interaction, Ç. Çığ, T. M. Sezgin, BBF # 2013.03.002
 Koç University, Research, Project Development and Technology Transfer Directorate Auto-Completion in Sketch Recognition, T. M. Sezgin, B.Yanıkoğlu, Ç. Tırkaz, BBF # 2011.03.X
 Koç University, Research, Project Development and Technology Transfer Directorate European Patent Application, T. M. Sezgin, Ç. Çığ, Gaze Based Prediction Device, PCT/TR2014/00189, European Patent Office, May 2014.

Publications

Ç. Çığ, T. M. Sezgin, Gaze-Based Virtual Task Predictor. Proceedings of International Conference on Multimodal Interfaces, Workshop Eye Gaze in Intelligent Human Machine Interaction: Eye-Gaze and Multimodality, Accepted for publication (2014). Ç. Çığ, T. M. Sezgin, Gaze-Based Prediction of Pen-Based Virtual Interaction Tasks. International Journal of Human- Computer Studies, Accepted for publication, (2014). Ç. Tırkaz, B. Yanıkoğlu, T. M. Sezgin, Sketched Symbol Recognition with Auto Completion. Pattern Recognition, vol 45, issue 11, pp 3926-3937 (2012).

slide-53
SLIDE 53

History of Human Computer Interaction

2007 ENIAC 1946 1989 Mac 
 Portable 1981 IBM PC iPhone

slide-54
SLIDE 54
slide-55
SLIDE 55
slide-56
SLIDE 56

Wizard of Oz Method

56

slide-57
SLIDE 57

The confession

Television Control by Hand Gestures William T. Freeman, Craig D. Weissman
 MERL Report: TR94-24

slide-58
SLIDE 58

The Problem


Too little effort towards understanding interaction

slide-59
SLIDE 59

Case study 3

slide-60
SLIDE 60

Case study 3

slide-61
SLIDE 61

Case study 3

slide-62
SLIDE 62

Case study 3

slide-63
SLIDE 63

Case study 3

slide-64
SLIDE 64

Case study 3

slide-65
SLIDE 65

Natural & Intelligent User Interfaces

▪ Understanding the user

▪ Natural modalities ▪ Collecting realistic data (observe the user in her space) ▪ Meet the user needs

▪ Real-time, seamless interaction ▪ Predictive interfaces

▪ Understanding machine learning

▪ Adaptation to the user ▪ Labeling large data sets (active learning) ▪ Getting better accuracies

▪ Classifier combination ▪ Feature selection

▪ Co-training, active learning

▪ Co-reference resolution

Important theme…

slide-66
SLIDE 66

Electrical Engineers Draw Sketches

slide-67
SLIDE 67

Electrical Engineers Draw Sketches

slide-68
SLIDE 68

Mechanical Engineers

slide-69
SLIDE 69

Software Engineers

slide-70
SLIDE 70

Sketch recognition

NPN Transistor

▪ Sketches are:

▪ Informal ▪ Messy ▪ Highly variable

▪ Focus:

▪ Iconic objects ▪ Compositional and hierarchical ▪ Online sketching (incremental)

▪ Our goal is to find:

▪ The correct segmentation ▪ The correct class

slide-71
SLIDE 71

After each stroke is drawn…

Sketch recognition at work

slide-72
SLIDE 72

Sketched Symbol Recognition with 
 Auto-Completion

… a list of top matches is produced. Predictive interfaces, Feature select Classifier combination, Co-training

slide-73
SLIDE 73

Input by stylus Recognition

Sketch recognition at work

Seamless integration, Natural modalit

slide-74
SLIDE 74

Machine learning technology & IUIs
 Designing interfaces and data collection

Realistic data, Seamless integration Wizard-of-Oz, Real-time interaction

slide-75
SLIDE 75

Machine learning technology & IUIs
 Data labeling

slide-76
SLIDE 76

Machine learning technology & IUIs
 Data labeling

Active learning

slide-77
SLIDE 77

Machine learning technology & IUIs
 User styles

CLUSTER A CLUSTER B

slide-78
SLIDE 78

Adaptation to user styles

Machine learning technology & IUIs
 Data labeling, User styles

slide-79
SLIDE 79

Led TV Tobii X120 Tablet

Multimodal Input and IUIs

Multimodality matters

slide-80
SLIDE 80

Multimodal Input and IUIs

slide-81
SLIDE 81

MIRA – Multi-Modal to Road Design Assistant

Natural modalities, Seamless integrat Coreference Resolution

slide-82
SLIDE 82

MISA – A Multi-Modal Approach to Storyboard Design

Speech Input Animation Recognition Play the Slides Animation Natural modalities, Seamless integrat Coreference Resolution

slide-83
SLIDE 83

Affect Recognition

slide-84
SLIDE 84

Helping Children with Autism Spectrum Conditions

slide-85
SLIDE 85

Helping Children with Autism Spectrum Conditions

slide-86
SLIDE 86

▪ 1% of the population ▪ Emotion recognition ▪ Display of emotions ▪ Learning through games ▪ Rehabilitation at a young age ▪ Interactive learning ▪ Formative assessment ▪ Approach ▪ Affect recognition ▪ Artificial intelligence ▪ Intelligent ingerfaces ▪ FP7 ASC-Inclusion ▪ International team (9 partners: Cambridge U., TUM … ) ▪ Academic, clinical, commercial impact ▪ Invaluable for the disadvantaged minorities

14/15

ASC-Inclusion

slide-87
SLIDE 87

Affective interfaces – animation

slide-88
SLIDE 88

Affective interfaces – recognition

slide-89
SLIDE 89

Affective interfaces – multiple modalities

slide-90
SLIDE 90

Collaboration and Negotiation:
 humans vs. computers vs. robots

Know thy customer! Modalities matte

slide-91
SLIDE 91

Goals ▪ Create IUI awareness ▪ Not just machine learning ▪ Million ways to do it wrong ▪ Showcase technology ▪ Sketch recognition ▪ Multimodal interfaces

▪ Eye-gaze ▪ Speech ▪ Sketching ▪ Affect ▪ Haptics

slide-92
SLIDE 92

Acknowledgements

▪ Postdocs

▪ Basak Alper ▪ Nese Alyuz ▪ Yusuf Sahillioglu

▪ PhD students

▪ Sinan Tumen ▪ Ayse Kucukyilmaz ▪ Caglar Tirkaz ▪ Cagla Cig ▪ Ezgi Emgin ▪ Emre Karaman ▪ Ferhat Cagan


▪ MS students

▪ Cansu Sen ▪ Burak Ozen ▪ Ozem Kalay ▪ Erelcan Yanik

▪ Atakan Arasan ▪ Banucicek Gurcuoglu ▪ Kemal Tugrul

▪ Undergraduate students

▪ Anil Uluturk ▪ Furkan Bayraktar ▪ Ozan Okumusoglu

▪ Collaborators

▪ Cagatay Basdogan ▪ Berrin Yanikoglu ▪ Engin Erzin ▪ Yucel Yemez

▪ Sponsors

▪ DARPA ▪ European Commission ▪ National Science Foundation ▪ Türk Telekom ▪ Koç Sistem ▪ Ministry of Science 
 Industry & Technology