Experimental Design & Evaluation 1. Introduction to ED&E - - PowerPoint PPT Presentation

▶
experimental design evaluation
SMART_READER_LITE
LIVE PREVIEW

Experimental Design & Evaluation 1. Introduction to ED&E - - PowerPoint PPT Presentation

Oct 31, 2023 225 likes •1.04k views

Experimental Design & Evaluation 1. Introduction to ED&E SunyoungKim,PhD Todays agenda Who are we? What is this course about? Syllabus Evaluation Class policy Introduction to ED&E

slide-1
SLIDE 1

Experimental Design & Evaluation

  • 1. Introduction to ED&E

SunyoungKim,PhD

slide-2
SLIDE 2

Today’s agenda

  • Who are we?
  • What is this course about?
  • Syllabus
  • Evaluation
  • Class policy
  • Introduction to ED&E
slide-3
SLIDE 3

Instructor

Sunyoung Kim

Office: Room 329 in SC&I Email: sunyoung.kim@rutgers.edu Website: http://www.sunyoung.kim.org

slide-4
SLIDE 4

Your turn

slide-5
SLIDE 5

Experimental Design & Evaluation

slide-6
SLIDE 6

Experimental Design & Evaluation

“A scientific procedure undertaken to make a discovery, test a hypothesis, or demonstrate a known fact”

slide-7
SLIDE 7

“The process of planning a study to meet specified

  • bjectives (of making a discovery, testing a hypothesis,
  • r demonstrating a known fact)”

Experimental Design & Evaluation

slide-8
SLIDE 8

“The making of a judgment about the amount, number, or value of something; assessment”

Experimental Design & Evaluation

slide-9
SLIDE 9

Planning a scientific study to make a discovery, test a hypothesis, or demonstrate a known fact & assessing the results of the study

Experimental Design & Evaluation

slide-10
SLIDE 10

The practice of designing interactive digital products, environments, systems, and services

Interaction Design (IxD)

slide-11
SLIDE 11

What are factors to consider for its success?

slide-12
SLIDE 12

The process of enhancing user satisfaction with a product by improving the usability, accessibility, and pleasure provided in the interaction with the product

UX Design (UxD)

slide-13
SLIDE 13

Planning a scientific study to make a discovery, test a hypothesis, or demonstrate a known fact & assessing the results of the study

Experimental Design & Evaluation

Planning a a hum human-sub subjec ect t stud study to ensure/improve user user sa sati tisfa sfacti tion

  • n with

th a intera nteracti tive e digita tal pr produ duct & assessing the usa usability ty, a , accessibi bility, a , and pl d pleasure when interacting with the product

slide-14
SLIDE 14

Interaction design cycle

slide-15
SLIDE 15

Syllabus

  • 1. Learn (understanding the users, tasks and contexts)
  • Data collection
  • Interview, Survey
  • Contextual Inquiry
  • Interpretation
  • Work model
  • Affinity diagram
  • Grounded Theory
  • Consolidation
  • 2. Design (building a system)
  • Persona, Scenario, Storyboard
  • 3. Measure (evaluating the system)
  • Heuristic evaluation
  • Think Aloud, usability testing
  • 4. Report
slide-16
SLIDE 16

Grading

  • 50% Project
  • 30% Quizzes
  • 20% Class participation
  • 10% Discussion and class activities
  • 10% Reading response
slide-17
SLIDE 17

Class Homepage

  • http://www.sunyoungkim.org/class/ede_sp17/
  • Access to slides- ID: rutgers PW: hci
slide-18
SLIDE 18

Project (50%)

  • Schedule and grades

Due ( by midnight) Assignment Weight

9/5 Team building check 9/12 9/19 Select your project topic: a problem, user, task, and context Submit an IRB 5% 10/3 Plan data collection 5% 10/17 Collect data 10% 10/24 Interpret data 10% 10/31 Consolidate findings - Modeling 10% 11/14 Prototype solutions 10% 12/5 Evaluate the solutions 10% 12/12 Final presentation 20% 12/19 Final report submission 20%

slide-19
SLIDE 19

Project: Goals

  • Acquire hands-on experiences of user-centered design process: data

collection, prototyping, analysis and report of the findings

  • Write an academic paper and submit it to CHI (https://chi2018.acm.org/)
  • Student Design Competition (https://chi2018.acm.org/authors/student-

design-competition/)

  • Late-Breaking Work (https://chi2018.acm.org/authors/late-breaking-work/)
  • Check out sample papers (Canvas > files)
slide-20
SLIDE 20

https://chi2018.acm.org/authors/student-design-competition/

slide-21
SLIDE 21

CHI 2017 SDC Finalists (https://chi2017.acm.org/assets/chi2017printprogforweb.pdf)

slide-22
SLIDE 22

CHI 2016 SDC Finalists (https://chi2016.acm.org/wp/wp-content/uploads/2016/05/chi2016-program-final-for-web-2.pdf)

slide-23
SLIDE 23

Grading: Quiz (30%)

  • 3 quizzes
  • Format: a number of short-answer/open-end questions
  • Schedule and points

Date Weight Topics 10/3 10% Understanding the context (data collection) 10/31 10% Interpreting and consolidating the collected data 12/5 10% Prototyping and Evaluating the system

slide-24
SLIDE 24

Grading: Class participation (20%)

  • 10% Discussion and class activities
  • 10% Reading response
  • Due Tuesday midnight, before class
  • Reading is posted on the website
  • Submit a reading response (a summary of reading) to Canvas

discussion board

  • Will be graded
  • 2 = good
  • 1 = partial
  • 0 = no submission
slide-25
SLIDE 25

Class policies

  • Be mindful during class time (aka no Facebook, Instagram, twitter,

snapchat, groupme, and so forth during class)

  • Respect and collaborate with your team members
  • All assignments are due by midnight before the class
  • Assignments will not be accepted late
  • Notify two days in advance if you have to miss taking a quiz
  • Cheating (official) will get you an F in the course
slide-26
SLIDE 26

Questions?

slide-27
SLIDE 27

Introduction to ED&E

slide-28
SLIDE 28

The practice of designing interactive digital products, environments, systems, and services

Interaction Design (IxD)

slide-29
SLIDE 29

What is a successful digital product? What used to be a successful digital product?

slide-30
SLIDE 30
slide-31
SLIDE 31
slide-32
SLIDE 32
slide-33
SLIDE 33

What is a successful digital product?

slide-34
SLIDE 34

Usability refers to the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of user Usability measures the quality of a user's experience when interacting with a product or system

Usability

  • Effectiveness: Can users successfully achieve their objectives?
  • Efficiency: How much effort is expended in achieving those
  • bjectives?
  • Satisfaction: Was the experience satisfactory?
slide-35
SLIDE 35

UI Hall of Fame or Shame?

slide-36
SLIDE 36

UI Hall of Fame or Shame?

slide-37
SLIDE 37

User-Centered Design (UCD)

slide-38
SLIDE 38

User-Centered Design (UCD)

slide-39
SLIDE 39
slide-40
SLIDE 40
slide-41
SLIDE 41

UI UI vs vs UX UX

User Interface User Experience

slide-42
SLIDE 42
slide-43
SLIDE 43

A world without UCD

slide-44
SLIDE 44

Why UCD?

slide-45
SLIDE 45

Interface Bloat

slide-46
SLIDE 46

Interface Bloat

slide-47
SLIDE 47

User-Centered Design (UCD)

Trying to satisfy the needs of all users may mean that you fail to satisfy the needs of any one user

slide-48
SLIDE 48

User-Centered Design (UCD)

slide-49
SLIDE 49

Why is User-centered approach important?

  • We interact with computing systems

in every facet of our everyday lives

  • When interacting with computing

systems, they do it through a user interface

  • How a user interacts with a computer

is different for each system

  • People do not accept products with

poor interfaces

  • If a system doesn’t work as expected,

it is not a user’s fault but a designer’s fault!

slide-50
SLIDE 50

Waterfall model

Traditional s/w design process

Waterfall model is bad for UI design because:

  • Users are not involved in validation until acceptance testing à So we won’t

find out until the end

  • UI flaws often cause changes in requirements and design à So we have to

throw away carefully-written and tested code

slide-51
SLIDE 51

Iterative model

  • Early focus on users and tasks
  • Plan to iterate with cheap, fast

prototypes

  • Evaluate frequently

UCD Process

Spiral model

slide-52
SLIDE 52

Data-Driven Design

An approach to defining a new system that collects multiple user- centered techniques into an integrated design process Not based on personal opinion, anecdotes, or unverifiable claims about what users would like but based on “DATA”!

slide-53
SLIDE 53

Benefits of Data

  • Requires design team to agree on common observations and their

representation

  • Reveals hidden aspects of work, that may be implicit in the user’s

understanding

  • Contextual Inquiry is a method for gathering and representing

data about the user and his/her work

slide-54
SLIDE 54

Interview

Models Models Models Models Sequence Notes

Interview

Models Models Models Models Sequence Notes

Interview

Models Models Models Models Sequence Notes

Interview

Models Models Models Models Sequence Notes

Interview

Models Models Models Models Sequence Notes

Models Models Models Models Sequence- flow model Affinity Idea Idea Idea Idea

Contextual Design

slide-55
SLIDE 55

Contextual Design: Challenge

How do we gather data from the users? How do we turn data into criteria? How do we design a system based on criteria?

slide-56
SLIDE 56

Understanding the users

slide-57
SLIDE 57

Interaction design cycle

slide-58
SLIDE 58
slide-59
SLIDE 59

Users Designer

We (designers) are not the users!

slide-60
SLIDE 60
slide-61
SLIDE 61

Who are the users?

Those who interact directly with the product Those who manage direct users Those who receive output from the product Those who make the purchasing decision Those who use competitors' product Stakeholders

  • Primary users: people who use the product directly
  • Secondary users: people who are affected by the product, or who

influence its development

slide-62
SLIDE 62

Who are the users?

  • Physical

§ Age § Sex § Culture § Physical ability & disabilities

  • Educational background
  • Computer/IT experience
  • Psychological

§ Motivation § Attitude

slide-63
SLIDE 63
slide-64
SLIDE 64

Identify potential participants

  • Which participants?
  • Gender
  • Age
  • Education level
  • Computer users
  • How many participants?
  • Resource (physical, financial)
  • Validity
  • How to recruit participant?
  • Informed consent
  • IRB
slide-65
SLIDE 65

What are their needs?

“Needs finding”

  • Expressed needs – what users say they want
  • Felt needs – users unsure what the system can do
  • Normative needs – professional view about the nature of the problem

and what may be needed Problems:

  • Users often don’t know what’s possible
  • Users often don’t know what they don’t know
  • Users often can’t articulate clearly what they need to achieve their goals
slide-66
SLIDE 66
slide-67
SLIDE 67

Methods: Understanding the user

Say Think Do Use Know Feel Dream

What people:

Interviews, survey Contextual inquiry Direct observation Participatory design

slide-68
SLIDE 68

Interaction design cycle

slide-69
SLIDE 69

Project

slide-70
SLIDE 70

Project

Team building Submit a team memo to Canvas by tonight with:

  • Name
  • Specialties
  • Role
  • Contact info
slide-71
SLIDE 71

Project: Goals

  • Acquire hands-on experiences of user-centered design process: data

collection, prototyping, analysis and report of the findings

  • Write an academic paper and submit it to CHI (https://chi2018.acm.org/)
  • Student Design Competition (https://chi2018.acm.org/authors/student-

design-competition/)

  • Late-Breaking Work (https://chi2018.acm.org/authors/late-breaking-work/)
  • Check out sample papers (Canvas > files)
slide-72
SLIDE 72

https://chi2018.acm.org/authors/student-design-competition/

slide-73
SLIDE 73

CHI 2017 SDC Finalists (https://chi2017.acm.org/assets/chi2017printprogforweb.pdf)

slide-74
SLIDE 74

CHI 2016 SDC Finalists (https://chi2016.acm.org/wp/wp-content/uploads/2016/05/chi2016-program-final-for-web-2.pdf)

slide-75
SLIDE 75

Project: Sample papers

  • Supporting Pakistani Farmers Through Digital Means: An Exploratory

Study

  • Contextual Inquiry of Future Commuting in Autonomous Cars
  • inSight: Kick-starting Communications For Elderlies Ageing In Place
  • Beam: A Mobile Application to Improve Happiness and Mental Health
  • Low-Income Parents’ Perceptions of Technology: Value-based Design

Insights

  • Bringing the Farmer Perspective to Agricultural Robots
  • Student Design Competition (https://chi2018.acm.org/authors/student-design-

competition/)

  • Late-Breaking Work (https://chi2018.acm.org/authors/late-breaking-work/)
slide-76
SLIDE 76

Project

Themes : Health, Wellbeing, or Sustainability, broadly speaking

  • Online prescription system (e.g., CVS pharmacy)
  • Electronic billing system (e.g., PSE&G)
  • Car insurance website (e.g., Geico)
  • Other…

Consideration factors

  • Interest: a problem that you are interested in!!!
  • Significance: an important problem to solve
  • Accessibility: easy access to potential/existing users
  • Feasibility: a problem that can be solvable
slide-77
SLIDE 77

Class exercise

Think about requirements/needs for

  • Online prescription system (e.g., CVS pharmacy)
  • Electronic billing system (e.g., PSE&G)
  • Car insurance website (e.g., Geico)
  • Other…
slide-78
SLIDE 78

Project

You have a week to decide which topic you will be working on throughout the semester. Choosing a right topic is critical! Sit together with your team members, research, brainstorm enough, share thoughts, check out other projects for ideas (chi15-17 lists are available on Canvas). By 9/12, submit a paragraph of your project topic to Canvas including:

  • What is the problem(s) that you are trying to solve?
  • Why is it an important problem?
  • What is the current status associated with the problem?
  • Who are the main users?
slide-79
SLIDE 79

Next class

  • Instructor out of town
  • A Guest lecturer (a PhD student) will teach you how to create an IRB
  • You need to complete a CITI program before next week

(https://orra.rutgers.edu/citi)

  • Reading: Lazar, Feng and Hochheiser, Research Methods in Human-Computer

Interaction, Ch.14-2

Following class

Readings

  • Müller,H., Sedley, A., & Ferrall-Nunge, E. Survey Research in HCI. In J. Olson

& W. Kellogg (Eds.), Ways of Knowing in HCI Research. New York:

  • Springer.Goodman et al (2012): Observing the User Experience Ch. 12,

Sections: When to Conduct Surveys and How to Field a Survey.