SoberIT Software Business and Engineering Institute Design, sam - PowerPoint PPT Presentation

SoberIT Software Business and Engineering Institute Design, sam pling and im plem entation of a survey Jani-Pekka Jokinen Researcher Helsinki University of Technology / Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Motivation � Questions are not only there to be answered, but first of all to be posed Anonymous Researcher � Each stupidity left in the questionnaire will probably be seen by thousands of people making you look like an idiot The good old ’be scared of public humiliation’ -approach � If you don’t fall asleep, hopefully after this lecture you will: � know the common procedure how to put up a survey � know the common pitfalls and problems � realize the importance of iterating and piloting the questionnaire HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Outline � Background � Survey design � Sampling � sample size � response rate � Questionnaire design � Survey implementation � Wrap up HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Background � A survey is not just the instrument for gathering information. It is a comprehensive system for collecting information to describe, compare or explain knowledge, attitude and behaviour. � Uses numerical information � Often tries to identify causal relations between different phenomena HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Quantitative research types � (Classical test setting (to identify) causal relations) � (Ex post facto research (similar to classical testing)) � (Case studies (quantitative part is often in the preliminary phase or used to test the validity of the results)) � SURVEYS HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute W hat is a survey? � Survey process has clearly-defined activities: 1. Setting specific, measurable objectives (what do we want to know) 2. Planning and scheduling the survey 3. Ensuring that appropriate resources are availabe (mailing, coding etc.) 4. Designing the survey 5. Preparing the data collection instrument 6. Validating the instrument 7. Selecting participants 8. Administrating and scoring the instrument 9. Analysing the data 10. Reporting the results (Pfleeger & Kitchenham 2001) HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Survey design � Similar to designing an experiment: design must match the objectives survey data and analysis answer to the questions posed � Usually a survey has two goals 1. attempting to describe a phenomenon of interest 2. aims to assess the impact of some intervention (requires an experimental design to support hypothesis testing) HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Keep in m ind � Bias (not biased by any particular opinion or aspect) � ”In your opinion, do single mothers need more financial support from the government?” –asked from a group of single mothers or fundamental Catolists � ”In your opinion, is the availability of risk finance as a significant barrier for the emergence of new software product companies? � Appropriate (makes sense in the context of the population) � What expenses belong to the A-group in your consumption in ABC cost approach? –asked from teenagers � Cost-effective (limitations in researchers resources as well as respondents) � cost come not just from mailing, acquiring contacts but also from reminders, re-postings, coding the data.... � how many minutes would you spend on a badly designed questionnaire (normally max. 15-20 min.) � confidence aspests (who will have the access to the data, what will it be used for..) HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Sam pling in theory � You must define a target population : the target population is the group or the individuals to whom the survey applies � Very often we have to survey just a subset of the population, called a sam ple : is it representative? � We need to know about the phenomenon we’re studying forehand (literature, previous studies etc.) � Sample size depends on the variance of the phenomenon, costs, information available etc. � Target population should arise from the survey objectives, not from a sense who is available to answer your questions HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Sam pling in practice � How to select a sample on a company level? � Tilastokeskus industry classifications � Other industry classifactors (Mailer Oy has more accurate classification) � Industry listings � How many good data points are you aiming for? � How to guarantee that you can generalize results from your sample? � How representative is your sample? � Can the target population answer your research questions? HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Sam pling m ethods � Probabilistic � Simple random sample � Stratified random sample (subgroups called strata) � Systematic sampling (every nth member) � Cluster-based sampling (belonging to a defined group) � Non-probabilistic � Convenience sampling (who is available) � Snowball sampling (letting your friend down) � Quota sampling (non-probabilistic version of random sampling) � Focus groups HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Sam ple size and response rate � How many respondents to target? � If taken from mathematical point of view: � consider Type I error (probability of falsely rejecting the null hypothesis) and Type II error (probability of falsely accepting the null hypothesis), variance and the effect size are considerable issues � Sample size varies from tens to thousands, depending on the issue, target population, needs etc. � if comparing different groups, each groups should have app. 30 data points in order to make statistical analyses � Response rate can range from 10 % to 70 % .... � A low response rate can ”destroy” a good survey HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Questionnaire � it is normally bad or satisfactory, make it the latter (have you seen a good one?) � give instructions � use enough spacing and different styles � be reasonable and considerate if possible � motivate! HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Questionnaire design � Common procedure: 1. Search the relevant literature 2. Construct an instrument 3. Evaluate the instrument 4. Document the instrument HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Exercise 1 � Take a look at the questionnaire: � Are there some things that might cause bias in the survey? � Do all the questions make sense? � Is it “cost-effective” enough that you would answer? What kind of motivation they are using? � What do you think about the questionnaire in general? HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute W hat to ask and how � A good question: � clear � unamibiguous � not leading � concentrating on essentials � asks only one issue � easy to answer � Understanding the respondent’s know-how, feelings and time consumption � Ask an appropriate number of questions (DIFFICULT) � Standardizing responses (i.e. scale 1-7 vs. open question) HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Different types of questions I � Structured � nominative scale (are you? 1 Man 2) Woman) � dichotomy scale (select all of the following that suits you.. 1) bored 2) happy 3) scared..) � preference scale (put in order the following, where 1. is the most desirable and 3 the least...I want in life __money, __power, ___happiness) � ordinal scale, like Likert-scale (do you find this lecture? 1....very boring 7....very interesting) � to add IDK (I don’t know) or not part dilemma � scale 1...4, 1...5, 1...7 dilemma � relational scale (has an absolute zero-point; how much is your annual salary?) HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Different types of questions I I : open questions � can give more detailed information, but! � more time taking for the respondents � more difficult to classify the responses � more time consuming to code USE OPEN QUESTIONS WITH CAUTION! � combinating open and closed questions is common (which of the following cough drops do you use? 1) Mentos 2) Pastirol 3) Sisu 4) other, what________) HELSINKI UNIVERSITY OF TECHNOLOGY

SoberIT Software Business and Engineering Institute Ups � For how many different operating systems you sell your main product to? � How many (% ) of your revenues came from already existing customers? � Did you smoke when getting married? � Many more... HELSINKI UNIVERSITY OF TECHNOLOGY