Sometimes There are Dumb Questions Garbage in-Garbage out: Why most surveys are worse than useless (except for yours of course ) Scott McIntyre, PhD Engineering & Experimental Psychologist
Project Lifecycle 1. Identify Customer Needs 2. Product Specifications 3. Concept Generation 4. Concept Selection 5. Concept Testing 6. Prototyping 7. Iteration 8. Product Delivery 9. Evaluation
Two concerns of this presentation 1. Identify Customer Needs 9. Evaluation BEST proposals collect objectively measurable pre & post data for accurate statistical evaluation of change
How do we GET IT? (research design) How will we USE IT? (statistical analysis)
Most common design to obtaining data? And maybe “required” in public outreach SURVEYS!! NO OFFENSE! EVERYONE uses poor surveys Except you
Scientists know what they are doing…right? Psychology-DSM “ should be used in research and evaluation as potentially useful tools to enhance clinical decision-making and not as the sole basis for making a clinical diagnosis.”
Survey Problems 1. Language 2. Memory 3. Bias 4. Prospective/predictive 5. Compound 6. Length/Time 7. Coding reliability 8. Scale/quality of data (subjective vs objective): – Nominal, ordinal, ratio 101. etc.
SCALE OF DATA: DEMO Do you like to cycle? Nominal/Categorical On a scale from 1-5, (1 being low and 5 being high) how much do you like to cycle? Ordinal/Rank How many minutes in the last month did you cycle? Ratio
Data analysis for project evaluation Danger!! Statistical Precision cannot make bad data good Poor quality no matter how you slice it
Statistical analysis Bad data Misleading data
Data collection for project evaluation 1. Nominal data (yes/no): “Do you ride a bike?” – Collapsing/equating/combining behavior very misleading—loss of information! – Two people, one who rides once/year and one who rides for 3 hours everyday are equal with nominal scale
Data collection for project evaluation 2. Ordinal/ranking data (1-5, 1-100): “How likely are you to…?” Subjectivity—Psychological difference between liking • something 3 or 4 vs 4 & 5 for SAME person is unknown let alone DIFFERENT people Collapsing data—info loss • Rank of finish in race not as sensitive as time!
Data analysis for project evaluation Nominal (Y/N) and Ordinal (1-5) data: subjective data and/or collapsed data + sensitive statistical analysis = MOST COMMON TYPES OF SURVEY QUESTIONS!!
Your project is science NOT twitter
1. What is the highest speed limit along your route to school? _____ MPH ( miles per hour ) Why rely on human memory or attention? Not posted Science says both are faulty. 3. Safety and comfort level: Define A) If you rode on the sidewalk, were the sidewalks comfortable and safe? Circle One: Always Sometimes Never “Sometimes comfortable” or safe? What do you do with that data?
Words or numbers, either way a problem
Science Imagine physicist wants to know “how fast is light?” Survey: • Is light fast? Y N • how fast is light? 1-Not very 3-quite a bit 7-super fast • Light travels at how many meters/second?
Data analysis for project evaluation For meaningful statistical analysis what you want is objectively measurable data (ratio)— time, distance, weight, speed, behaviors/unit of time
eg. Instead of “Do you ride a bike?” Try: How many minutes/hours per week/month/year do you ride your bike?
Statistical Analysis to Evaluate Change Ratio data allows for sensitive comparison across time and between groups Flagstaff Kingman Phoenix Lake Havasu 0 15 60 minutes per month
Professional survey consultants Name of firm covered to protect the guilty “When you are asking a number of questions based on a similar rating scale, …” make sure it’s an objective, ratio scale?? WHAT?... NO IDIOT! “be sure that the answer rating – whether it’s 1 to 5 or 0 to 10 – flows consistently …”
Endorsement “The expert analyses and recommendations we deliver serve as a blueprint for driving operational improvements to enhance the customer experience”
Should you survey at all? When was the last time you surveyed your pet? Despite that, are you meeting the needs of your pet? How know if … Hungry? Potty? Walk? Scratched? Survey?
Observe behavior
If for some reason you need to know if sidewalks are “comfortable” and by that you mean are they wide enough and are without major elevation changes…go measure them. If you rode on the sidewalk, were the sidewalks comfortable and safe? Circle One: Always Sometimes Never
Survey Problems 1. Language 2. Memory 3. Bias 4. Prospective/predictive 5. Compound 6. Length/Time 7. Coding reliability 8. Scale/quality of data (subjective vs objective): – Nominal, ordinal, ratio 101. etc.
Surveying Public Should Augment Your Expertise not Check a Box
Expertise • Any project involving humans needs experts in human behavior, research design, data collection and statistical analysis • Through no fault of their own, engineers and MBAs are likely not experts in any of those. • Caveat: Being a scientist and understanding humans is not highly correlated, r = 0.3, p > .05.
Don’t Follow the Herd Be Better!
Summary What do funders and you ultimately want to know? Did you make a difference? • Can’t know that objectively without good data for sensible statistical analysis Get objectively measurable data! Separate your proposal from the professional survey • “experts” If survey you MUST … survey well Find your Denise!?
Student project Assessing problem behavior in classroom Current inventory: subjective, tracking behavior problematic
Student redesign: quantitative, electronic, prefilled, auto calculating
QUESTIONS???
Recommend
More recommend
