Evaluating the Accuracy of Data Collection on Mobile Phones: - PowerPoint PPT Presentation

Evaluating the Accuracy of Data Collection on Mobile Phones: Collection on Mobile Phones: A Study of Forms, SMS, and Voice Somani Patnaik 1 , Emma Brunskill 1 , William Thies 2 1 Massachusetts Institute of Technology 2 Microsoft Research India ICTD 2009

Mobile Data Collection is in Style • Especially in the developing world – Mobile banking Mobile banking – Microfinance – Healthcare Healthcare – Environmental monitoring • Benefits: B fit No prior study of entry accuracy – Faster (on low-cost phones in developing world) – Cheaper Ch – More accurate

Data Collection on Mobile Phones OpenROSA FrontlineSMS Forms [Banks] Nokia Data Gathering [Nokia] RapidSMS [UNICEF] MobileResearcher [Populi net] MobileResearcher [Populi.net] Cell-Life in South Africa [Fynn] Jiva TeleDoc in India [UN Publications] [ ] Pesinet in Mali [Balancing Act News] Malaria monitoring in Kenya [Nokia News] Voxiva Cell-PREVEN in Peru [Curioso et. al]

Data Collection Data Collection on Mobile Phones on PDAs SATELLIFE OpenROSA FrontlineSMS Forms [Banks] EpiHandy Nokia Data Gathering [Nokia] EpiSurveyor [Datadyne] RapidSMS [UNICEF] Infant health in Tanzania [Shrima et al.] e-IMCI in Tanzania [DeRenzi et al.] e IMCI in Tanzania [DeRenzi et al ] MobileResearcher [Populi.net] MobileResearcher [Populi net] Cell-Life in South Africa [Fynn] Respiratory health in Kenya [Diero et al.] Jiva TeleDoc in India [UN Publications] [ ] Tobacco survey in India [Gupta] y [ p ] Pesinet in Mali [Balancing Act News] Ca:sh in India [Anantramanan et al.] Malaria monitoring in Kenya [Nokia News] Malaria monitoring in Gambia [Forster et al.] Voxiva Cell-PREVEN in Peru [Curioso et. al] Clinical study in Gabon [Missinou et al.] Tuberculosis records in Peru [Blaya et al.] Sexual surveys in Peru [Bernabe-Ortiz et al ] Sexual surveys in Peru [Bernabe-Ortiz et al.]

Data Collection Data Collection on Mobile Phones on PDAs SATELLIFE OpenROSA FrontlineSMS Forms [Banks] EpiHandy Nokia Data Gathering [Nokia] EpiSurveyor [Datadyne] RapidSMS [UNICEF] Infant health in Tanzania [Shrima et al.] e-IMCI in Tanzania [DeRenzi et al.] e IMCI in Tanzania [DeRenzi et al ] MobileResearcher [Populi.net] MobileResearcher [Populi net] Cell-Life in South Africa [Fynn] Respiratory health in Kenya [Diero et al.] Jiva TeleDoc in India [UN Publications] [ ] Tobacco survey in India [Gupta] y [ p ] Pesinet in Mali [Balancing Act News] Ca:sh in India [Anantramanan et al.] Malaria monitoring in Kenya [Nokia News] Voxiva Cell-PREVEN in Peru [Curioso et. al] Published Error Rates Malaria monitoring in Gambia [Forster et al.] Clinical study in Gabon [Missinou et al.] Tuberculosis records in Peru [Blaya et al.] Sexual surveys in Peru [Bernabe-Ortiz et al.]

Data Collection Data Collection on Mobile Phones on PDAs SATELLIFE OpenROSA FrontlineSMS Forms [Banks] EpiHandy Nokia Data Gathering [Nokia] EpiSurveyor [Datadyne] RapidSMS [UNICEF] Infant health in Tanzania [Shrima et al.] e IMCI in Tanzania [DeRenzi et al ] e-IMCI in Tanzania [DeRenzi et al.] MobileResearcher [Populi net] MobileResearcher [Populi.net] Cell-Life in South Africa [Fynn] Respiratory health in Kenya [Diero et al.] Jiva TeleDoc in India [UN Publications] [ ] Tobacco survey in India [Gupta] y [ p ] Pesinet in Mali [Balancing Act News] Ca:sh in India [Anantramanan et al.] Malaria monitoring in Kenya [Nokia News] Voxiva Cell-PREVEN in Peru [Curioso et. al] Published Error Rates Published Error Rates Malaria monitoring in Gambia [Forster et al.] Clinical study in Gabon [Missinou et al.] None? Tuberculosis records in Peru [Blaya et al.] CAM in India [Parikh et al.] Sexual surveys in Peru [Bernabe-Ortiz et al.]

Our Study • Compared three interfaces for health data collection Electronic Forms SMS Live Operator Append to current SMS: 11. Patient’s Cough: 13 lit 13 literate health t h lth No Cough ‐ Press 1 Rare Cough ‐ Press 2 workers & hospital Mild Cough ‐ Press 3 Heavy Cough ‐ Press 4 staff, Gujarat, India staff, Gujarat, India Severe Cough g ‐ Press 5 (with blood) — printed cue card— 4 2% 4.2% 4 5% 4.5% 0 45% 0.45% Error rate: Error rate: Result caused partners to switch from forms to operator • Recommendations: 1 Caution needed in deploying critical apps w/ non-expert users 1. Caution needed in deploying critical apps w/ non expert users 2. A live operator can be accurate and cost-effective solution

Context: Rural Tuberculosis Treatment • With local partners, working to improve CHINA NEPAL New Delhi Bihar tuberculosis treatment in rural Bihar India tuberculosis treatment in rural Bihar, India BANGLADESH INDIA INDIA Kolkata BURMA Mumbai THE PRAJNOPAYA FOUND THE PRAJNOP FOUNDATION ON Hyderabad Treatment Sites Bangalore Dalsingh Sarai Chennai Bih Bihar Sharif Sh if • Strategy: monitor patient symptoms remotely Health worker H lth k uploads symptoms Physician reviews, advises, schedules visits • Data uploaded: 11 questions, every 2 weeks – Patient ID P ti t ID ─ Temperature T t ─ Weight W i ht – Cough (multiple choice) ─ Symptoms (yes / no)

Design Space: Data Collection on Low End Phones Data Collection on Low-End Phones AUDIO Interactive Voice Spoken Live Response Dialog Operator Prompts VISUAL VISUAL SMS Electronic Voice-Activated Forms Forms less more less more interactive interactive interactive interactive TYPED SPOKEN Data Entry

Design Space: Data Collection on Low End Phones Data Collection on Low-End Phones AUDIO Live Operator Prompts VISUAL VISUAL SMS Electronic Forms less more less more interactive interactive interactive interactive TYPED SPOKEN Data Entry

1. SMS Interface • Pro: + Potentially cheapest + Potentially cheapest • Con: – Easiest to fake visits E i f k i i – Least reliable

2. Electronic Forms Interface • Pro: + Arguably more + Arguably more user friendly than SMS • Con: • Con: – Expensive handset

3. Live Operator Interface • Pro: Health Worker Operator Patient + Most flexible Q&A + Most flexible Q&A + No literacy required + Hard to fake visits + Hard to fake visits • Con: “Is the patient “Are you having having night sweats?” having night sweats? – C Cost of operator t f t night sweats? night sweats?” – Potentially slower “No, she isn’t.” “No, I’m not.”

Study Participants • 13 health workers and hospital staff (Gujarat, India) Age Education Cell Phone (Median) Experience 10 th – 12 th Health workers (6) 23 H lth k (6) 23 10 th 12 th H d Had used phone d h 12 th – D. Pharm. Hospital staff (7) 30 Owned phone • Within-subjects design • Training standard: • Training standard: two error-free reports on each interface on each interface – Health workers: big groups, 6-8 hours – Hospital staff: small groups, 1-2 hours

Results Append to current SMS: 11. Patient’s Cough: No Cough ‐ Press 1 Rare Cough ‐ Press 2 Mild Cough ‐ Press 3 Heavy Cough ‐ Press 4 Severe Cough ‐ Press 5 (with blood) — printed cue card— Electronic Forms SMS Live Operator Error rate Error rate 4 2% 4.2% 4 5% 4.5% 0 45% 0.45% (errors / entries) (12/286) (13/286) (1/ 220)

Results Health 7.6% 6.1% workers workers 3.2% Hospital 1.5% 1.3% staff staff 0% 0% Electronic Forms SMS Live Operator Error rate Error rate 4 2% 4.2% 4 5% 4.5% 0 45% 0.45% (errors / entries) (12/286) (13/286) (1/ 220)

Sources of Error Multiple Choice (SMS) (SMS) Numeric Multiple Choice (Forms)

Sources of Error Usability Barriers - small keys small keys - correcting mistakes - decimal point Correct Incorrect Multiple Choice 54 45 (SMS) (SMS) 62 62 826 826 Numeric 62 empty 68 67 68 93 69 59 98.5 98 Multiple Choice 98.7 98.687 (Forms) 100.2 100.0 100 3 100.3 103 103 “1003” 103 100.8 108

Sources of Error Usability Barriers - small keys small keys - correcting mistakes - decimal point - scrolling / selection Multiple Choice (SMS) (SMS) Correct Incorrect Numeric Mild None Heavy Mild Yes Yes No No No Yes Multiple Choice (Forms)

Sources of Error Usability Barriers - small keys small keys - correcting mistakes - decimal point - scrolling / selection Multiple Choice - SMS encoding (SMS) (SMS) Numeric Correct Incorrect “1” (none) “0” (disallowed) “1” (none) 1 (none) “0” (disallowed) 0 (disallowed) “1” (none) “0” (disallowed) “3” (mild) “0” (disallowed) “5” (severe) 5 (severe) empty empty Multiple Choice “6” (A. Khanna) “5” (A. Kumar) (Forms) “7” (A. Kapoor) “1” (A. Khan) “6” “6” “2” “2” “0000007” “000007”

Sources of Error Detectable Usability Barriers Errors - small keys small keys - correcting mistakes - decimal point - scrolling / selection Multiple Choice - SMS encoding (SMS) Numeric Multiple Choice (Forms)

Cost Comparison SMS Forms Live Operator Cost per interview Cost per interview C C S C S C (C V + C O ) T (C + C ) T Program variables Cost variables T time spent per interview T time spent per interview C S cost of an SMS cost of an SMS C C V cost of a voice minute C O cost of an operator minute C O cost of an operator minute