SoberIT Software Business and Engineering Institute Design, sam - - PowerPoint PPT Presentation

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Design, sam pling and im plem entation of a survey

Jani-Pekka Jokinen

Researcher Helsinki University of Technology / Software Business and Engineering Institute

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Motivation

• Questions are not only there to be answered, but first of all to be posed

Anonymous Researcher

• Each stupidity left in the questionnaire will probably be seen by thousands of

people making you look like an idiot The good old ’be scared of public humiliation’ -approach

• If you don’t fall asleep, hopefully after this lecture you will:
• know the common procedure how to put up a survey
• know the common pitfalls and problems
• realize the importance of iterating and piloting the questionnaire

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Outline

• Background
• Survey design
• Sampling
• sample size
• response rate
• Questionnaire design
• Survey implementation
• Wrap up

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Background

A survey is not just the instrument for gathering

• information. It is a comprehensive system for collecting

information to describe, compare or explain knowledge, attitude and behaviour.

Uses numerical information Often tries to identify causal relations between different

phenomena

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Quantitative research types

(Classical test setting (to identify) causal relations) (Ex post facto research (similar to classical testing)) (Case studies (quantitative part is often in the preliminary

phase or used to test the validity of the results))

SURVEYS

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

W hat is a survey?

• Survey process has clearly-defined activities:

1.

Setting specific, measurable objectives (what do we want to know)

2.

Planning and scheduling the survey

3.

Ensuring that appropriate resources are availabe (mailing, coding etc.)

4.

Designing the survey

5.

Preparing the data collection instrument

6.

Validating the instrument

7.

Selecting participants

8.

Administrating and scoring the instrument

9.

Analysing the data

• 10. Reporting the results

(Pfleeger & Kitchenham 2001)

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Survey design

• Similar to designing an experiment: design must match

the objectives survey data and analysis answer to the questions posed

• Usually a survey has two goals
• 1. attempting to describe a phenomenon of interest
• 2. aims to assess the impact of some intervention (requires an

experimental design to support hypothesis testing)

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Keep in m ind

• Bias (not biased by any particular opinion or aspect)
• ”In your opinion, do single mothers need more financial support from the government?” –asked from a group
• f single mothers or fundamental Catolists
• ”In your opinion, is the availability of risk finance as a significant barrier for the emergence of new software product

companies?

• Appropriate (makes sense in the context of the population)
• What expenses belong to the A-group in your consumption in ABC cost approach? –asked from teenagers
• Cost-effective (limitations in researchers resources as well as respondents)
• cost come not just from mailing, acquiring contacts but also from reminders, re-postings, coding the data....
• how many minutes would you spend on a badly designed questionnaire (normally max. 15-20 min.)
• confidence aspests (who will have the access to the data, what will it be used for..)

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Sam pling in theory

• You must define a target population: the target population is the group
• r the individuals to whom the survey applies
• Very often we have to survey just a subset of the population, called a

sam ple: is it representative?

• We need to know about the phenomenon we’re studying forehand

(literature, previous studies etc.)

• Sample size depends on the variance of the phenomenon, costs,

information available etc.

• Target population should arise from the survey objectives, not from a

sense who is available to answer your questions

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Sam pling in practice

• How to select a sample on a company level?
• Tilastokeskus industry classifications
• Other industry classifactors (Mailer Oy has more accurate classification)
• Industry listings
• How many good data points are you aiming for?
• How to guarantee that you can generalize results from your sample?
• How representative is your sample?
• Can the target population answer your research questions?

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Sam pling m ethods

• Probabilistic
• Simple random sample
• Stratified random sample (subgroups called strata)
• Systematic sampling (every nth member)
• Cluster-based sampling (belonging to a defined group)
• Non-probabilistic
• Convenience sampling (who is available)
• Snowball sampling (letting your friend down)
• Quota sampling (non-probabilistic version of random sampling)
• Focus groups

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Sam ple size and response rate

• How many respondents to target?
• If taken from mathematical point of view:
• consider Type I error (probability of falsely rejecting the null hypothesis) and Type II error

(probability of falsely accepting the null hypothesis), variance and the effect size are considerable issues

• Sample size varies from tens to thousands, depending on the issue, target

population, needs etc.

• if comparing different groups, each groups should have app. 30 data points in order to make

statistical analyses

• Response rate can range from 10 % to 70 % ....
• A low response rate can ”destroy” a good survey

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Questionnaire

it is normally bad or satisfactory, make it the latter (have

you seen a good one?)

give instructions use enough spacing and different styles be reasonable and considerate if possible motivate!

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Questionnaire design

• Common procedure:

1.Search the relevant literature 2.Construct an instrument 3.Evaluate the instrument 4.Document the instrument

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Exercise 1

Take a look at the questionnaire:

Are there some things that might cause bias in the survey? Do all the questions make sense? Is it “cost-effective” enough that you would answer? What

kind of motivation they are using?

What do you think about the questionnaire in general?

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

W hat to ask and how

• A good question:
• clear
• unamibiguous
• not leading
• concentrating on essentials
• asks only one issue
• easy to answer
• Understanding the respondent’s know-how, feelings and time consumption
• Ask an appropriate number of questions (DIFFICULT)
• Standardizing responses (i.e. scale 1-7 vs. open question)

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Different types of questions I

• Structured
• nominative scale (are you? 1 Man 2) Woman)
• dichotomy scale (select all of the following that suits you.. 1) bored 2) happy 3) scared..)
• preference scale (put in order the following, where 1. is the most desirable and 3 the

least...I want in life __money, __power, ___happiness)

• rdinal scale, like Likert-scale (do you find this lecture? 1....very boring 7....very

interesting)

• to add IDK (I don’t know) or not part dilemma
• scale 1...4, 1...5, 1...7 dilemma
• relational scale (has an absolute zero-point; how much is your annual salary?)

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Different types of questions I I : open questions

• can give more detailed information, but!

more time taking for the respondents more difficult to classify the responses more time consuming to code

USE OPEN QUESTIONS WITH CAUTION!

• combinating open and closed questions is common (which of the

following cough drops do you use? 1) Mentos 2) Pastirol 3) Sisu 4) other, what________)

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Ups

For how many different operating systems you

sell your main product to?

How many (% ) of your revenues came from

already existing customers?

Did you smoke when getting married? Many more...

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Survey im plem entation

Reserve enough time for iterating the questionnaire Resourve enough time for piloting the questionnaire: you

always get surprises (How old are you? ___born in 1945

• r I’m 58 years old..)

Remember: after the questionnaire is out, there is nothing

you can do about it

Iterate iterate iterate and iterate! And then pilot!

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

I m plem enting the survey: tasks

• Design and piloting
• Gathering the contact information
• Printing materials
• cover letter (signatures, motivation etc.)
• questionnaire (id, two-sided?stapled? font?)
• envelopes (pre-paid, pick-up, HUT-logo...?)
• return envelopes (contract with the post if return contract is used)
• Posting (who will do, quality control, risk management etc.)
• Follow-up calls (randomly selected)
• Follow-up mailing (who have already answered, unknown addresses..)
• Coding the database and finding the coders
• Coding the data

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Coding the data

Normally some statistical analysis software is used (SPSS,

SAS etc.)

The data should be coded so that the questionnaires can

be as easily and accurately typed in as possible

How to minimize typing errors? How to make judgement calls?

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Check list for a questionnaire design

• Will it keep the respondent interested to the end?
• Are the questions really understandable? Are terms well known and explicit?
• Is the importance of the question vital, very important or NTK? (Forget those nice

to knows)

• Is the questionnaire short enough (absolute maximum 10 pages..)
• Don’t hug the whole world, keep the number of instances involved in research minimized
• Give the respondent enough instructions (Pick one, mark three…

)

• How can the data from the questionnaires be transferred to the database?

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Questions?

SoberIT

Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

• On SE survey research
• Pfleeger and Kitchenham. 2001. Principles of Survey Research. Parts 1-

5.Software Engineering Notes.

• http: / / www.honeybee.helsinki.fi/ users/ niskanen/ kotu/ survey.htm
• Survey Techniques
• Fink. 1995. The survey handbook. Sage Publications.
• Bourgue and Fielder. 1994. How to conjduct self-administred and mail surveys.

Sage publications.

• Questionnaire design
• Sudman and Bradburn. 1982. Asking Questions. A practical guide to

questionnaire degin. Jossey-Bass Publishers.

References