A Framework for Remote Usability Evaluation on Mobile Devices - - PowerPoint PPT Presentation

▶
a framework for remote usability evaluation on mobile
SMART_READER_LITE
LIVE PREVIEW

A Framework for Remote Usability Evaluation on Mobile Devices - - PowerPoint PPT Presentation

Jun 12, 2023 223 likes •785 views

Technische Universitt Mnchen A Framework for Remote Usability Evaluation on Mobile Devices Bachelorarbeit in Informatik Daniel Bader Technische Universitt Mnchen Introduction Usability - An indicator for the ease of use and

slide-1
SLIDE 1

Technische Universität München

A Framework for Remote Usability Evaluation

  • n Mobile Devices

Bachelorarbeit in Informatik Daniel Bader

slide-2
SLIDE 2

Technische Universität München

Daniel Bader 15.04.2011

Introduction

Usability

  • An indicator for the ease of use and acceptability of a system

Usability Evaluation

  • Methods for determining the usability of a system
  • Usability data are collected and analyzed by human evaluators

Common activities (Ivory and Hearst, 2001)

2

Common usability evaluation activities Capture Analysis Critique

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

slide-3
SLIDE 3

Technische Universität München

Daniel Bader 15.04.2011

Introduction

Automated Usability Evaluation

  • Some or all phases of an Usability Evaluation are automated
  • Various degrees of automation exist

Remote Usability Evaluation

  • Evaluators and test users are separated in space and/or time

Automated Remote Usability Evaluation

  • One of the “ultimate goals” in the field of usability evaluation

3

slide-4
SLIDE 4

Technische Universität München

Daniel Bader 15.04.2011

Introduction

Usability Evaluation is especially important on mobile devices

  • User interfaces on mobile devices became more complex
  • Mobile context creates new challenges:
  • different environments, distractions, ...

Current mobile devices have a rich ability to track their surroundings

  • Microphone, GPS, accelerometer, video camera, ...

Usability Evaluation can be (partly) automated

4

slide-5
SLIDE 5

Technische Universität München

Daniel Bader 15.04.2011

Problem statement

A lack of software support for Usability Evaluation on current mobile platforms

  • No automation support in existing tools
  • No support for built-in tracking capabilities

(microphone, accelerometer, GPS, ...)

  • No support for collecting user interactions

graphically (screenshots, video)

5

slide-6
SLIDE 6

Technische Universität München

Daniel Bader 15.04.2011

Proposed solution

The muEvaluationFramework (mobile usability Evaluation Framework)

An application-independent software framework for remote usability evaluation on mobile platforms which supports automation in all phases of a usability evaluation.

6

Common usability evaluation activities Capture Analysis Critique

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

slide-7
SLIDE 7

Technische Universität München

Daniel Bader 15.04.2011

Outline

✓ Introduction ✓ Problem statement

  • Proposed solution

– Requirements specification – System design – Object design

  • Prototype demo
  • Future work

7

slide-8
SLIDE 8

Common usability evaluation activities Capture Analysis Critique

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Technische Universität München

Daniel Bader 15.04.2011

Functional requirements

Capture phase

  • Session-based usability data collection
  • Support for multiple usability-data sources
  • User input events
  • Application events
  • Device sensor events
  • Live preview for collected events

8

slide-9
SLIDE 9

Common usability evaluation activities Capture Analysis Critique

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Technische Universität München

Daniel Bader 15.04.2011

Functional requirements

Analysis phase

  • Interpretation of the captured data
  • Detection of usability problems
  • e.g. Human Interface Guidelines (HIG) violations
  • Usability summary generation
  • Most used views and UI widgets
  • Heat maps for user input events
  • ...

9

slide-10
SLIDE 10

Common usability evaluation activities Capture Analysis Critique

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Technische Universität München

Daniel Bader 15.04.2011

Functional requirements

Critique phase

  • Report generation
  • Reports summarize the interpretation results
  • Reports can be viewed using a web browser
  • Report configuration
  • Selection of the report sections

10

slide-11
SLIDE 11

Technische Universität München

Daniel Bader 15.04.2011

Nonfunctional requirements

Minimal setup work required by the developer

  • Low entry barrier
  • Quick setup for existing projects
  • Goal: “link against one library and make one method call”

Wireless communication with the mobile device

  • Test users can move freely and do not feel restricted
  • Evaluation can be performed outside of a laboratory

Extensibility

  • Easily add new sensors, interpreters, and sections
  • Framework intended as a research platform

11

slide-12
SLIDE 12

Technische Universität München

Daniel Bader 15.04.2011

Use case model

12

<<actor>>

HostApplication ControlSession ConfigureReport ViewReport CaptureUsabilityData muEvaluationFramework TestUser Developer <<initiate>> <<initiate>> <<initiate>> <<participate>> <<participate>> <<initiate>>

slide-13
SLIDE 13

Technische Universität München

Daniel Bader 15.04.2011

Object model and Dynamic model

13

Capture Analysis Critique

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Conceptual sub-models for each of the three phases

slide-14
SLIDE 14

Technische Universität München

Daniel Bader 15.04.2011

Object model and Dynamic model

13

Capture Analysis Critique

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Conceptual sub-models for each of the three phases

slide-15
SLIDE 15

Technische Universität München

Daniel Bader 15.04.2011

Capture phase

14

HostApplication TestUser EvaluationSession Developer 4: stop() 3: interactWith() 2: participate() 1: start()

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

The life cycle of an EvaluationSession

slide-16
SLIDE 16

Technische Universität München

Daniel Bader 15.04.2011

Capture phase

15

+start() +stop() +selectSensors(sensorSet : Sensor []) EvaluationSession TestUser Developer +launch() +quit() HostApplication UsabilityData MobileDevice +attachTo(t : SensorTarget) +enable() +disable() Sensor monitors collects runs on uses is collected in is analyzed in participates in initiates interacts with

slide-17
SLIDE 17
  • timestamp : Time

<<abstract>> Event +attachTo(t : SensorTarget) +enable() +disable() <<abstract>> Sensor +addEvent(e : Event) SessionJournal UsabilityData * * {ordered} collects <<represents>> creates

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Technische Universität München

Daniel Bader 15.04.2011

Capture phase

16

Sensors and Events

slide-18
SLIDE 18

Technische Universität München

Daniel Bader 15.04.2011

Capture phase

17

<<abstract>> Sensor TouchInputSensor AudioSensor TextEntrySensor ScreenSensor

Sensor specializations

slide-19
SLIDE 19

<<abstract>> Event UserInputEvent CustomEvent ApplicationEvent UserInterfaceEvent DeviceSensorEvent

Technische Universität München

Daniel Bader 15.04.2011

Capture phase

18

Event specializations

slide-20
SLIDE 20

Technische Universität München

Daniel Bader 15.04.2011

Object model and Dynamic model

19

Capture Analysis Critique

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

slide-21
SLIDE 21

Technische Universität München

Daniel Bader 15.04.2011

Analysis phase

20

<<abstract>> InterpretationResult <<abstract>> Interpreter InterpreterController +addResult(r : InterpretationResult) ResultStore

  • timestamp : Time

<<abstract>> Event * * * 1..* stores results in controls * interprets interprets generates

Interpreters work on data collected in the Capture phase and generate InterpretationResults.

slide-22
SLIDE 22

<<abstract>> Interpreter OverviewInterpreter ViewChangesInterpreter ScreenshotInterpreter UIGuidelineChecker ViewDurationInterpreter TouchHeatmapInterpreter AnalysisStatisticsInterpreter

Technische Universität München

Daniel Bader 15.04.2011

Analysis phase

21

Interpreter specializations

slide-23
SLIDE 23

Technische Universität München

Daniel Bader 15.04.2011

Object model and Dynamic model

22

Capture Analysis Critique

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

slide-24
SLIDE 24

Technische Universität München

Daniel Bader 15.04.2011

Critique phase

23

Report InterpretationResult Developer HostApplication recommends changes to views summarizes

slide-25
SLIDE 25

+addSection(s : Section) Report <<abstract>> Section OverviewSection EventTimeline TouchHeatmaps UserNavigationPath 1..* {ordered}

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Technische Universität München

Daniel Bader 15.04.2011

Critique phase

24

Report document model

slide-26
SLIDE 26

Technische Universität München

Daniel Bader 15.04.2011

High-level dynamic model

25

Capture Analysis Critique TestUser Report ResultStore SessionJournal UsabilityData InterpretationResults Events

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Inputs and outputs of the three phases

slide-27
SLIDE 27

Technische Universität München

Daniel Bader 15.04.2011

System design

26

Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Subsystem decomposition of the framework

slide-28
SLIDE 28

Technische Universität München

Daniel Bader 15.04.2011

System design

27

<<device>> Mobile device <<device>> Desktop computer <<component>> CaptureLibrary <<component>> HostApplication <<component>> Critique <<component>> Analysis <<component>> CaptureServer <<artifact>> CaptureServer.py <<artifact>> libCapture.a <<artifact>> ReportGenerator.py <<manifest>> <<wireless network>> <<manifest>> <<manifest>> <<manifest>>

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Deployment of the framework

slide-29
SLIDE 29

Technische Universität München

Daniel Bader 15.04.2011

Object design

28

Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
slide-30
SLIDE 30

Technische Universität München

Daniel Bader 15.04.2011

CaptureLibrary - Method interception

29

UIWindow

+sendEvent() +... +... Original implementation

  • f sendEvent()

Event

UIWindow sendEvent() behavior before method interception

slide-31
SLIDE 31

Technische Universität München

Daniel Bader 15.04.2011

CaptureLibrary - Method interception

29

UIWindow

+sendEvent() +... +... Original implementation

  • f sendEvent()

Event

UIWindow sendEvent() behavior before method interception

slide-32
SLIDE 32

Technische Universität München

Daniel Bader 15.04.2011

CaptureLibrary - Method interception

29

UIWindow

+sendEvent() +... +... Original implementation

  • f sendEvent()

Event

UIWindow sendEvent() behavior before method interception

slide-33
SLIDE 33

Technische Universität München

Daniel Bader 15.04.2011

CaptureLibrary - Method interception

30

UIWindow

+sendEvent() +... +... Original implementation

  • f sendEvent()

UIWindow sendEvent() behavior after method interception Our implementation of sendEvent()

Event CaptureLibrary

slide-34
SLIDE 34

Technische Universität München

Daniel Bader 15.04.2011

CaptureLibrary - Method interception

30

UIWindow

+sendEvent() +... +... Original implementation

  • f sendEvent()

UIWindow sendEvent() behavior after method interception Our implementation of sendEvent()

Event CaptureLibrary

slide-35
SLIDE 35

Technische Universität München

Daniel Bader 15.04.2011

CaptureLibrary - Method interception

30

UIWindow

+sendEvent() +... +... Original implementation

  • f sendEvent()

UIWindow sendEvent() behavior after method interception Our implementation of sendEvent()

Event CaptureLibrary

slide-36
SLIDE 36

Technische Universität München

Daniel Bader 15.04.2011

CaptureLibrary - Method interception

30

UIWindow

+sendEvent() +... +... Original implementation

  • f sendEvent()

UIWindow sendEvent() behavior after method interception Our implementation of sendEvent()

Event CaptureLibrary Event

slide-37
SLIDE 37

Technische Universität München

Daniel Bader 15.04.2011

Object design

31

Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
slide-38
SLIDE 38

Technische Universität München

Daniel Bader 15.04.2011

Analysis subsystem

32

Usage of the Blackboard design pattern in the Analysis subsystem

ResultStore InterpreterController Interpreter +getResultsOfType() +addResult() <<Interface>> Blackboard +canContribute() +updateBlackboard() <<Interface>> BlackboardExpert +loop() +getReadyExperts() <<Interface>> BlackboardSupervisor supervises 1..* controls updates and accesses

slide-39
SLIDE 39

Technische Universität München

Daniel Bader 15.04.2011

Analysis subsystem

33

Dependencies between Interpreters are resolved automatically

InterpreterA InterpreterB InterpreterC InterpreterD

Generated order of execution:

slide-40
SLIDE 40

Technische Universität München

Daniel Bader 15.04.2011

Analysis subsystem

33

Dependencies between Interpreters are resolved automatically

InterpreterA InterpreterB InterpreterC InterpreterD InterpreterC InterpreterB InterpreterA InterpreterD

Generated order of execution:

{C} → {B} → {A, D}

slide-41
SLIDE 41

Technische Universität München

Daniel Bader 15.04.2011

Object design

34

Capture Critique Analysis ReportController ResultStorage AnalysisController Interpretation SectionGenerator CaptureLibrary CaptureServer Communication

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
slide-42
SLIDE 42

Technische Universität München

Daniel Bader 15.04.2011

Critique subsystem

35

  • body : String

Element

  • title : string
  • header : string
  • elements : Element[]

+__init__(title) +add_element(e) +generate_html(result_store) : string <<abstract>> Section OverviewSection NavigationPathSection TouchHeatmapSection ViewDurationSection UIGuidelineIssuesSection

  • title : string
  • header : string
  • footer : string
  • sections : Section[]

+add_section(s) +generate_html(result_store) : string +generate(result_store, filename) Report

  • reportFilename : string
  • eventsFilename : string
  • interpreters : string[]
  • sections : string[]

+addInterpreter(i : string) +addSection(s : string) +setEventsFilename(s : string) +setReportFilename(s : string) +generateReport() ReportGeneratorFacade 1..* 1..*

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
slide-43
SLIDE 43

Technische Universität München

Daniel Bader 15.04.2011

Prototypical implementation

Full vertical prototype for Apple iOS

  • Implemented in Objective-C and Python
  • Tested with the applications Wordpress for iOS and PlainNote

Capture support

  • Automatic network setup using Zeroconf/Bonjour
  • Custom, stateless XML-based network protocol
  • Data transmission is performed on background thread

Analysis support

  • Six interpreters were implemented

Critique support

  • Generates HTML output
  • Five section types were implemented

36

slide-44
SLIDE 44

Technische Universität München

Daniel Bader 15.04.2011

Demo

37

slide-45
SLIDE 45

Technische Universität München

Daniel Bader 15.04.2011

Future work

Add more sensors and interpreters

  • e.g. audio and video recording

Use the framework in a real world scenario Add support for Google Android

  • Java has rich support for reflection

Add ability to playback sessions

  • Play back captured user interactions on the device

Integrate framework in FastFix

  • Perform usability error detection
  • Provide access to user interaction data and context information

38

slide-46
SLIDE 46

Technische Universität München

Daniel Bader 15.04.2011

Thank you!

Questions?

39

slide-47
SLIDE 47

Technische Universität München

Daniel Bader 15.04.2011

Backup slides

40

slide-48
SLIDE 48

<<actor>>

HostApplication PreviewEvents SelectSensors StopSession StartSession ControlSession ConfigureReport ViewReport CaptureUsabilityData muEvaluationFramework TestUser Developer <<Extend>> <<initiate>> <<initiate>> <<initiate>> <<participate>> <<participate>> <<initiate>> Technische Universität München

Daniel Bader 15.04.2011

Use case model (with refinements)

41

slide-49
SLIDE 49

Technische Universität München

Daniel Bader 15.04.2011

Capture phase

42

Sensors and SensorTargets

  • isActive : boolean

+attachTo(t : SensorTarget) +enable() +disable() <<abstract>> Sensor HostApplication UsabilityData SensorTarget * 1..* *

  • bserves

collects monitors

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

slide-50
SLIDE 50

SessionJournal e1 : UserInputEvent TestUser a1 : Action Sensor 2: detect() 4: addEvent(e1) 3: create(time) 1: perform()

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)

Technische Universität München

Daniel Bader 15.04.2011

Capture phase

43

Event detection example

slide-51
SLIDE 51

Technische Universität München

Daniel Bader 15.04.2011

Capture phase

44

Capture phase Initialize SessionJournal Initialize Sensors Create Event Observe SensorTargets Continue with next Sensor ActiveSensors <<structured>> for each Sensor SessionJournal [targets did not change] Event [targets changed]

slide-52
SLIDE 52

Run analysis phase Initialize ResultStore Initialize Interpreters Update ResultStore Run Interpreter ResultStore <<structured>> Loop SessionJournal InterpretationResult Event

Technische Universität München

Daniel Bader 15.04.2011

Analysis phase

45

slide-53
SLIDE 53

Generate Report Add Section to Report Generate Section Select Section to generate ReportConfiguration <<structured>> Loop Report ResultStore InterpretationResult Section

Technische Universität München

Daniel Bader 15.04.2011

Critique phase

46

slide-54
SLIDE 54

Technische Universität München

Daniel Bader 15.04.2011

System design

47

The Capture package

Capture CaptureLibrary CaptureServer Communication <<abstract>> Sensor SensorTarget <<abstract>> Event SessionJournal NetworkConnection NetworkMessage CaptureManager ServerController 1..* 1..* * * * receives Events sends Events

  • bserves

creates

slide-55
SLIDE 55

Technische Universität München

Daniel Bader 15.04.2011

System design

48

The Analysis package

Analysis Capture AnalysisController Interpretation Storage InterpretationResult Interpreter InterpreterController ResultStore stores results in 1..* * * generates SessionJournal

Visual Paradigm for UML Standard Edition(TUM - Institut fuer Informatik - Lehrstuhl 1)
slide-56
SLIDE 56

Technische Universität München

Daniel Bader 15.04.2011

System design

49

The Critique package

Critique Analysis SectionGenerator ReportController ReportConfiguration ReportGenerator Report Section SectionGenerator 1..* 1..* uses generates ResultStore uses generates