From offline to online: Retaining quality while transferring an - - PowerPoint PPT Presentation

From offline to online: Retaining quality while transferring an established survey

ASC - Smart research technology for cost- conscious times 27th April 2012

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Background

3

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

• Run since 1991
• Monitors public’s opinions and attitudes:
• TV programme standards
• Protection of Children
• News – sources, impartiality
• Take up and usage of devices / services
• Purpose
• Methodology

Ofcom’s Media Tracker survey - background

Methodology UK nat rep, adults 16+ n = c.1,700 – 2 waves per year Long interview (c. 35 minutes) F2F Oversample and monitor certain demographic groups

4

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Why did Ofcom want to explore possibility of transferring Media Tracker online?

5

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Why did Ofcom want to explore possibility of transferring Media Tracker online?

6

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Seeking to obtain enhanced value for money…

MR spend

Sources: Ofcom Tech Tracker; MRS

2.16 2.08 2.06 2005 2006 2007 2008 2009 2010 2011 UK Internet penetration UK MR spend (£bn)

7

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Seeking to obtain enhanced value for money - at the same time viability of a natrep online survey has potentially increased

MR spend vs. internet penetration

Sources: Ofcom Tech Tracker; MRS

58% 61% 65% 67% 73% 75% 78% 2.16 2.08 2.06 2005 2006 2007 2008 2009 2010 2011 UK Internet penetration UK MR spend (£bn)

8

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Why did Ofcom want to explore possibility of transferring Media Tracker online?

Maximise value Economic climate Internet penetration Explore feasibility and practicalities

9

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Speed

Reduce elapse time of project

Cost-effectiveness

Reduce cost and thus increase value

Ability to use visual stimuli

Engages respondents and makes the survey more interesting

And others!

Key benefits from transferring to online

10

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Usage of technology Usage of media Attitudes towards media Internet usage

Everyone

People who use the internet are not the same as people that don’t use the internet

Internet users

Internet users that are on online panels are not the same as people that aren't on online panels

Internet users not on a panel

Internet users that aren't on online panels but fill in surveys, are not the same as internet users that aren't on online panels but don’t fill in surveys

Key concerns - representivity

11

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

82% 96% 94% 89% 73% 45% 87% 76% 70% 89%

All Adults 16-34 35-44 45-54 55-64 65+ ABC1 C2DE Freeview

• nly

Pay TV

Ever use internet

Key concerns – Internet penetration varies between key groups

Source: Ofcom Media Tracker 2011

Internet users are younger

12

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

82% 96% 94% 89% 73% 45% 87% 76% 70% 89%

All Adults 16-34 35-44 45-54 55-64 65+ ABC1 C2DE Freeview

• nly

Pay TV

Ever use internet

Key concerns – Internet penetration varies between key groups

Source: Ofcom Media Tracker 2011

Internet users are younger and watch TV differently

13

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

82% 50% 61% 38%

86% 55% 69% 42% Ever listen to radio Sexually explicit programmes should never be shown on TV (DISAGREE) Aware TV channel blocking Aware Ofcom responsible regualting TV

All adults Internet users

Key concerns – behavioural and attitudinal representivity

Source: Ofcom Media Tracker 2011

Internet users behave differently….

14

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

82% 50% 61% 38%

86% 55% 69% 42% Ever listen to radio Sexually explicit programmes should never be shown on TV (DISAGREE) Aware TV channel blocking Aware Ofcom responsible regualting TV

All adults Internet users

Key concerns – behavioural and attitudinal representivity

Source: Ofcom Media Tracker 2011

Internet users behave differently and have different attitudes….

15

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

82% 50% 61% 38%

86% 55% 69% 42% Ever listen to radio Sexually explicit programmes should never be shown on TV (DISAGREE) Aware TV channel blocking Aware Ofcom responsible regualting TV

All adults Internet users

Key concerns – behavioural and attitudinal representivity

Source: Ofcom Media Tracker 2011

Internet users behave differently and have different attitudes….

16

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

82% 50% 61% 38%

86% 55% 69% 42% Ever listen to radio Sexually explicit programmes should never be shown on TV (DISAGREE) Aware TV channel blocking Aware Ofcom responsible regualting TV

All adults Internet users

Key concerns – behavioural and attitudinal representivity

Source: Ofcom Media Tracker 2011

Internet users behave differently and have different attitudes and knowledge

17

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

82% 50% 61% 38%

86% 55% 69% 42% Ever listen to radio Sexually explicit programmes should never be shown on TV (DISAGREE) Aware TV channel blocking Aware Ofcom responsible regulating TV

All adults Internet users

Key concerns – behavioural and attitudinal representivity

Source: Ofcom Media Tracker 2011

Internet users behave differently and have different attitudes and knowledge

18

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

56% 20% 5% 1% 17% 80% 5% 5% 2% 0% Every day Several times a week At least once a month Several times a year Never

All adults Internet users And of course – they use the internet more frequently

Key concerns – behavioural and attitudinal representivity

Source: Ofcom Media Tracker 2011

19

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Media Tracker particularly prone to highlights differences in answers concerning: Sources of news Attitudes towards news sources TV reception method Devices used for watching TV Internet usage

Set quotas to ensure demographic match Sample from multiple sources Include specific additional attitudinal and behavioral questions Weight the data

How to ensure representative (as far as possible)

20

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Media Tracker particularly prone to highlights differences in answers concerning: Sources of news Attitudes towards news sources TV reception method Devices used for watching TV Internet usage

Set quotas to ensure demographic match Sample from multiple sources Include specific additional attitudinal and behavioral questions Weight the data

How to ensure representative (as far as possible)

21

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Data checked for: Speeding Flat lining Contradictory answers Sensible open-ended responses Responses compared to information already held Multiple responses from same IP address Not as regulated as F2F or telephone Self-completion Participation is incentivised Self-screening

Other concerns - quality

22

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Data checked for: Speeding Flat lining Contradictory answers Sensible open-ended responses Responses compared to information already held Multiple responses from same IP address Not as regulated as F2F or telephone Self-completion Participation is incentivised Self-screening

Other concerns - quality

23

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Not as established as F2F or telephone Non-acceptance that it is as valid or comparable as offline methods If weighting is required, it can be difficult to explain to audiences what has been done and why it is necessary TV and Radio broadcasters Advertisers News suppliers Lobbying groups The general public!

Other concerns - acceptance

24

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Not as established as F2F or telephone Non-acceptance that it is as valid or comparable as offline methods If weighting is required, it can be difficult to explain to audiences what has been done and why it is necessary TV and Radio broadcasters Advertisers News suppliers Lobbying groups The general public!

Other concerns - acceptance

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

The test

26

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Objectives To assess the differences in results between an online approach and the current face to face approach To develop a process to “correct” for these differences To evaluate if this process can be used for the next wave of the Media Tracker

The test

If migration is to take place it is imperative that Ofcom is confident there has been no step change in results.

27

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Online 728 interviews

232 from panel, 496 from river sampling

Quotas set to match F2F proportions Geographic quotas set at regional level instead of postcode sector Questionnaire adapted to account for

• nline self-completion aspect

Face to face 890 interviews Quotas and sampling as per usual Method(s) - fieldwork

28

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

The technical bits!

• A measure of how much weighting has reduced the effective sample size
• A weighting efficiency of 50% means a sample of 1,000 is an effective

sample size of 500 for the purpose of calculating confidence intervals Weighting efficiency

• Enables you to weight your data to a number of different profiles
• The more profiles you use, the more you reduce weighting efficiency
• … but you have data that aligns with the profiles you have set

Rim weighting

• A technique to identify what differentiates one or more groups from others

Discriminant analysis

29

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

The test

• Undertake the survey both online and offline
• Merge the data
• Discriminant analysis
• Rim weight online sample to the profile of the
• ffline sample for influential variables
• Compare results
• Start point: weight both samples by conventional

demographics

30

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v330

15% 28% 26% 38% 9% 15% 4% 6% 45% 11% F2F Online

Every day Several times a week At least once a month Several times a year Never

Frequency of watching recorded TV on PVR or on demand service

Source: Q21 On average, how often, if at all, do you watch television that is not live scheduled TV (Base: All Respondents watching TV (1591/872/719)

Online has higher frequency of watching recorded/catch up TV

31

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v331

70% 89% 20% 7% 9% 4% F2F Online

Yes No Don't know

Aware of catch up services such as iPlayer

Source: Q88 You may be aware that it's possible to watch programmes online or downloaded programmes from TV broadcaster websites (such as BBC iPlayer, ITV player, 4OD, Demand Five, or Sky Player). Were you already aware of this? (Base: All Respondents (1618/890/729)

Online has higher awareness of catch up services on PC

32

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v332

53% 92% 93% 89% 18% 5% 4% 9% 4% 3% 3% 2% 2% 23% F2F Online River Panel

Every day Several times a week At least once a month Several times a year Never

Source: Q90 How often do you use the internet on any device (at home/elsewhere net) Base: All Respondents (890/720/487/233)

One reason for the differences is frequency of internet use

Frequency of using the internet (any location)

33

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

First discriminant analysis

33 33

• Top discriminator: Frequency of using the internet
• However, rim weighting on this reduces weighting

efficiency from 78% to 10%

• This is unacceptable as the effective sample size

is reduced to 10% of its original value

• So ‘frequency of internet’ question removed and

search for proxies

34

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v334

199 60 52 37 10 Frequency of viewing television that is not live scheduled TV Awareness of catch-up TV services on PC? Activities undertaken whilst watching TV As far as you are aware is the internet regulated? Working status Importance

Most important discriminators

35

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v335

23% 33% 30% 45% 41% 42% 20% 13% 15% 17% 13% 13% Online

• riginal

F2F Online with five rims

Freeview Satellite Cable Other

Main method of TV reception

Source: Q11/12 Which if any of these types of television does your household receive? If more than one, which would you regard as your main types of television? (Base: All Respondents watching TV(1591/872/719)

Online now matches on key questions – main TV viewing method

36

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v336

78% 36% 30% 23% 22% 20% Demographics Frequency of viewing television that is not live scheduled TV Awareness of catch-up TV services on PC? As far as you are aware is the internet regulated? Working status Activities undertaken whilst watching TV

% weighting efficiency of online sample

There is a downside – the weighting efficiency drops to 20%

37

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Comparison of online and offline results

37 37

With these five rims, online results compare well to offline results In practice, we could:

• Run an online survey with a large sample size
• Large enough to generate same effective sample size as current
• ffline survey
• Put the four questions on a consumer omnibus
• Rim weight the online survey by demographics plus these variables

This process would fairly well replicate an offline survey

• At lower overall cost and with faster turnaround

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Conclusions

39

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Conclusions and key learnings for organisations

It is possible to get results from an online sample to match those from an

• ffline sample

The (relative) success of which will depend of the topic of the survey Media and technology-related topics are among the least suitable for transfer online Extensive weighting is required, reducing effective sample size

• Could be countered by increasing sample size
• Or boosting with offline sample

Some questions would still need to be asked offline in order to provide profile to weight the online data.

40

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Conclusions and key learnings for organisations

Due to the extensive scrutiny and multiple stakeholders involved with public sector research, online may not be suitable for many surveys

• Particularly tracking surveys where continuity is important
• And surveys where the topic is particularly likely to be affected by the

data collection method

41

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

3 final questions

What’s the topic? What’s the audience? What’s the budget/cost?

42

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

Questions

43

Shared\Execs\Tim Barber\My Documents\Powerpoint\Ofcom\J11426 – Online Pilot\MRS conference presentation – 12.01.12 v3

For further information…

Please contact us if you have any questions or comments

Tim Barber, Research Director Tel : 020 7490 9121 tim.barber@bdrc-continental.com Dave Chilvers, Director Tel : 020 7490 9111 dave.chilvers@bdrc-continental.com Web www.bdrc-continental.com Sumran Kaul, Senior Audience Analyst, Ofcom Tel : 020 7981 3827 sumran.kaul@ofcom.org.uk Web www.ofcom.org.uk