Stop Using Your Gut: How to Efficiently Measure ROI Curtis Onuczko - - PowerPoint PPT Presentation

▶

Aug 11, 2023 21 likes •327 views

Stop Using Your Gut: How to Efficiently Measure ROI Curtis Onuczko Software Developer, BioWare Edmonton Introduction Create tools to improve processes Unguided decisions are dangerous May invest too much time Nobody uses tool

SLIDE 1

Curtis Onuczko

Software Developer, BioWare Edmonton

Stop Using Your Gut: How to Efficiently Measure ROI

SLIDE 2

Introduction

Create tools to improve processes
Unguided decisions are dangerous
May invest too much time
Nobody uses tool

Goals:

Accurately measure ROI
Develop process to ensure high ROI

SLIDE 3

Using your Gut – Hit or Miss

SLIDE 4

Using your Gut – Hit or Miss

Hit – B4Bug
High usage
ROI? Don’t know.

SLIDE 5

Using your Gut – Hit or Miss

Hit – B4Bug
High usage
ROI? Don’t know.
Miss – Automated regression tests
Large investment
Low ROI

SLIDE 6

Using your Gut – Hit or Miss

Hit – B4Bug
High usage
ROI? Don’t know.
Miss – Automated regression tests
Large investment
Low ROI
Insight – We can develop a process better than this

SLIDE 7

Process

Software engineers shadow Users
Identify repetitive patterns
Encapsulate patterns
Measure ROI

SLIDE 8

Shadowing

Why use software engineers?
Why shadow users?
Insight – Shadowing users is more cost-effective than

training them to be engineers

SLIDE 9

Shadowed Users on

Identified lowest hanging fruit
Editing Screenshots
File Attachments
Insight - Ensure you shadow, don’t interview

SLIDE 10

Edit Screenshots –

SLIDE 11

Edit Screenshots –

SLIDE 12

Edit Screenshots –

SLIDE 13

Measuring accurately

Use telemetry
Ongoing, accurate, and reliable
Easy to visualize
Benefits of measuring
Confident in results and decisions
Predictive

SLIDE 14

So what to measure?

Time Returned on Investment(ROI)
Task Complexity Reduction (TCR)

SLIDE 15

Time ROI

Time saved VS. time invested
Shows net gain or loss
Changes over time

SLIDE 16

Task Complexity Reduction (TCR)

Cognitive steps before VS. after
Measures change in cognitive workload
Shorter training times
Easier to context switch
Perform tasks in parallel

SLIDE 17

Edit Screenshots – ROI

2 4 6 8 10 12 14 16 18 20 Jul Aug Sep Oct Nov Dec Jan Time (Hours) Date Time Invested Time Saved

SLIDE 18

Edit Screenshots – TCR

Cognitive Steps Before – 10
Cognitive Steps After – 5
TCR of 50%!
5200 steps saved to date

SLIDE 19

Edit Screenshots - Insights

Time ROI is slow to aggregate
Ensure enough time for positive ROI
Bad marketing?
Other metrics besides ROI
TCR
User Perception – Future Work

SLIDE 20

Attach Files –

Attach files to bugs
Could attach files via internal tool

SLIDE 21

Estimating ROI

Attach files similar to edit screen shots encapsulation
Less anticipated use
Greater estimated investment
Insight - Edit screenshot telemetry can estimate the ROI
f attaching files

SLIDE 22

Attach Files – Estimated ROI

5 10 15 20 25 30 35 40 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Time (Hours) Date Time Invested Time Saved

SLIDE 23

Attach Files – Results

Estimated a smaller ROI
TCR was smaller (25%)
Dropped further development

SLIDE 24

Future Work

Define specific process for shadowing
Refine what a cognitive step is
Eg., Mouse click VS. navigate to file on disk
Measure user perception

SLIDE 25

Key Insights

SLIDE 26

Key Insights

Shadowing more valuable than interviewing

SLIDE 27

Key Insights

Shadowing more valuable than interviewing
ROI is slow to aggregate

SLIDE 28

Key Insights

Shadowing more valuable than interviewing
ROI is slow to aggregate
Can predict ROI using similar ROI data

SLIDE 29

Key Insights

Shadowing more valuable than interviewing
ROI is slow to aggregate
Can predict ROI using similar ROI data
Other metrics besides time ROI

SLIDE 30

Conclusion

Shadowing users and measuring ROI helps make more

informed decisions

Allows concrete data to be used in those decisions