Human-Robot Interaction Elective in Artificial Intelligence - - PDF document

▶

Jan 20, 2024 95 likes •213 views

Human-Robot Interaction Elective in Artificial Intelligence Lecture 10 User Evaluation Luca Iocchi DIAG, Sapienza University of Rome, Italy HRI Evaluation HRI requires evaluation methods involving users Evaluation with users is not

SLIDE 1

Human-Robot Interaction

Elective in Artificial Intelligence Lecture 10 – User Evaluation

Luca Iocchi DIAG, Sapienza University of Rome, Italy

HRI requires evaluation methods involving users
Evaluation with users is not easy, because it always

involves some subjectivity

Many issues:
Trained vs. untrained users
Recruited vs. non-recruited users
Experimental protocol
L. Iocchi - User Evaluation

HRI Evaluation

SLIDE 2

Different kinds of interactions
Supervisor
Operator
Teammate (trained and knowledgeable about the robot)
Bystander (no trained and knowledgeable about the robot)

Bystanders need information about what the robot can do!

L. Iocchi - User Evaluation

HRI Evaluation

1. Subjects
2. Variables
3. Hypothesis
4. Experimental protocol
L. Iocchi - User Evaluation

HRI Experimental factors

SLIDE 3

Selection of subjects is a crucial factor.

sufficient sample set
independent identically distributed
…
L. Iocchi - User Evaluation

HRI Evaluation subjects

Variables: quantities to control and measure

Independent variables (conditions)

controlled during the experiment

Dependent variable (condition)

measured in the experiment

L. Iocchi - User Evaluation

HRI Evaluation variables

SLIDE 4

Typical HRI/HCI evaluation measures
Effectiveness (how much the task is completed)
Efficiency (how much time/resources are needed)
User satisfaction (user feelings about the process)
L. Iocchi - User Evaluation

HRI Evaluation measures

Hypotheses or research questions

H. Variable H is affected by variables/conditions

X, Y, and Z

Q. How variable H is affected by variables/conditions

X, Y, and Z? Null hypothesis: conditions X, Y, Z do not affect H

L. Iocchi - User Evaluation

HRI Evaluation hypotheses

SLIDE 5

Research question: how a user reacts when s/he encounters a robot in the corridor with respect to its motion behavior and the use of voice to acknowledge the presence of the user? Independent variables (conditions) 1. Robot motion behavior (possible values: straight, slow_down_and_move_right) 2. Robot vocalization (possible values: no speech, speech explaining that the robot would avoid the user) Dependent variable 1. User comfort: how much the user trusts the robot (i.e., s/he continues walking in the corridor in a natural way)

L. Iocchi - User Evaluation

HRI Evaluation example

Evaluation protocol
Within-subject (same user exposed to different

conditions, evaluating variation over time of user performance)

Between-subject (different users exposed to different

conditions, evaluating differences between users)

L. Iocchi - User Evaluation

HRI Evaluation protocol

SLIDE 6

Example of Within-subject protocol:

1. Assess a baseline performance on a basic system

(e.g., basic HRI) with a group of users

2. Evaluate a more sophisticated system

(e.g., HRI with a new feature A) with same group of users Result = improved performance of the new feature A Advantage: it requires less users Disadvantage: possibly biased results due to dependencies between different phases of the tests for the same user (e.g., fatigue and practice)

L. Iocchi - User Evaluation

HRI Evaluation protocol

Example of Between-subject protocol

1. Assess a baseline performance on a basic system

(e.g., basic HRI) with a first group of users

2. Evaluate a more sophisticated system

(e.g., HRI with a new feature A) with a second group

f users

Result = improved performance of the new feature A Advantage: it overcomes biased results Disadvantage: it requires more users, results may be affected by individual variability

L. Iocchi - User Evaluation

HRI Evaluation protocol

SLIDE 7

Analysis of quantitative data Several statistical techniques are used:

t-test
ANOVA
…

Choosing the right test to perform is a key issue

L. Iocchi - User Evaluation

HRI Analysis of data

General approach:

Specify the null hypothesis (there is no difference)
Collect data
Compute the p-value (using an appropriate tool)
Draw a conclusion

p<0.05 – statistically significant (*) p<0.01 – highly statistically significant (**) p<0.001 – extremely statistically significant (***)

L. Iocchi - User Evaluation

HRI Analysis

SLIDE 8

Student’s t-test is a statistical test to verify a statistical significance between two sets of measures. Example: measure the time to complete an HRI task in two different conditions A and B.

L. Iocchi - User Evaluation

HRI Analysis

Example: measure the time to complete an HRI task in two different conditions A and B. Repeat the test with condition A nA times and collect values TA = { t1

A, …, tnA A }

Repeat the test with condition B nB times and collect values TB = { t1

B, …, tnB B }

Compute the mean values µA and µB (with corresponding standard deviations) Is the difference significant or it is due to randomness?

L. Iocchi - User Evaluation

HRI Analysis

SLIDE 9

Example: measure the time to complete an HRI task in two different conditions A and B. Use a Between-subject protocol to collect data TA and TB Perform an independent (unpaired) t-test to assess significance in the results (rejection of null hypothesis). How to compute: Excel, R

L. Iocchi - User Evaluation

HRI Analysis

Excel T-test - Positive if P-value < 0.05

L. Iocchi - User Evaluation

HRI Analysis

Values Group A Values Group B 30.5 32 Mean A 31.05 31.5 28 Stddev A 1.83 34 31 Mean B 29.75 31 28 Stddev B 1.30 30.5 30.5 31 30 t-test P-value 0.08549 34 28.5 … …

SLIDE 10

Example: measure the time to complete an HRI task in three different conditions A, B, and C. Collect data TA , TB , and TC Perform a two-way analysis of variance (ANOVA) How to compute: Excel, R

L. Iocchi - User Evaluation