Recommended Practices for the design of business surveys - - PowerPoint PPT Presentation

Recommended Practices for the design of business surveys questionnaires

Stefania Macchia

macchia@istat.it (Istat, National Accounts and Economic Statistics Department)

NTTS 2015 Conference Bruxelles, 10-12 March

2

ISTAT Business Statistical Portal implements a new approach for the organization and management of data collection processes. It constitutes a single entry point for web- based data collection from enterprises according to a “business-centric” perspective.

Process 1

Businesses register

Process n Business es Statistical Portal

Data capturing systems Communicatio n

The context

 High integration among all the phases of the productive processes: survey design, data collection, frame, design of electronic questionnaires, elaboration, estimations, informative «return»

 Rationalization and intensification of the informative flows among «stakeholders» involved in statistical processes (production lines, statistical registers, data capturing systems, dissemination and communication, statistical units)  Introduction of new management procedures characterised by high interrelation degree among Portal, production lines, production support facilities  High potential in terms of systemic efficiency

Main Features:

Harmonisation of business surveys questionnaires

3 The new Business Portal architecture implies the acceleration of the harmonization

• f survey questionnaires, both in terms of concepts treated and features of

electronic questionnaires, with the perspective of:  reducing respondent burden in terms of duplication of information requested  adopting a common language of communication with enterprises through a standard approach for questionnaires design, which takes into consideration the peculiarities of the different business surveys questionnaires (SBS and STS), with attention to the variables definitions management and to the structure of questions to collect information  enhancing the level of quality (in terms of response rates, but also homogeneity of collected data).

Manual of Recommended Practices for the design of business surveys questionnaires

Manual of Recommended Practices for the design of business surveys questionnaires

Not all the aspects of the structured process to design questionnaires have been treated because, in this context, the aim was not to plan new surveys ex novo, but to restructure and harmonize surveys to be included in Business Portal. The manual is concentrated on:  The definition of the questionnaire schemes  All the aspects of the questionnaire design related to the data capturing technique

4

GSBPM v.5: 2.2 Design variable descriptions 2.3 Design collection

Questionnaire Electronic Questionnaire design design

‘The sub-process 2.2 may need to run in parallel with sub- process 2.3, as the definition of the variables to be collected, and the choice of collection instrument may be inter-dependent’

Recommended Practices for the design of business surveys questionnaires

In defining Recommended Practices, it has been considered how some aspects of the design of questionnaires and of the features to be implemented in the electronic questionnaire have an impact on the steps of the response process, represented in a ‘hybrid response model’ based on a cognitive approach.

5

• Encoding in Memory / Record Formation
• Selection / Identification of Respondent(s)
• Assessment of Priorities (Motivation)
• Release of the Data

Comprehension Retrieval Judgement Communication.

 The questions wording, their structuring, the layout and all the instructions and information at disposal

• f respondents related to each question  impact on

the comprehension and retrieval processes.  The skipping and consistency rules implemented in electronic questionnaire  support for the judgement step  the electronic questionnaire design  impact on the communication process, as it predefines the structures according to which responses have to be given.  the questionnaire design  impact on the Encoding in memory/Record formation step because as more the way data are stored in the respondent data base is considered in structuring questions, as more the effort needed to provide data is contained. Ibrid response model (2000 Sudman, 2001 Wilimack, Nichols)

Manual of Recommended Practices for the design of business surveys questionnaires

6

First section of the Manual: General principles and particular ones related to the CAWI technique (shortly described) Principles defined according to a strategy which should:  Encourage responses  Contain responses errors  Reduce the effort necessary to provide answers  Give respondents a feedback

Very important general principle, consistent with an integrated approach:  contain information to be collected, using sources already available like:  Administrative/fiscal sources  Data collected in other surveys  Official registers.

Manual of Recommended Practices

7

The set of recommendations provided are organised in paragraphs according to specific homogeneous themes: 1) Design of questionnaire sections/Web pages Suggestions are given on how designing the sections, taking into considerations the Web pages, which in turn, can be composed by more than one computer screen-shots. 2) Questions design Identification of the structure to be used to provide the answer: suggestions are provided on quantitative questions (how to manage decimal values, scale factors, measure units, etc.), closed questions (when using radio-buttons, drop-down-boxes, check-boxes), free texts questions to be coded (when it is preferred to use assisted coding functions) and, finally, questions organised in tables/arrays. 3) Use of classifications When classifications are used to classify textual responses, suggestions are given on different algorithms for textual matching which can be used.

Manual of Recommended Practices

8

4) The structure for variables definitions Harmonising the definitions in terms of their content is surely the first problem to be faced but how this content is expressed and structured has in impact on the comprehension process as well. For this reason a common modular structure for definitions of variables has been defined 5) The management of skipping rules The automatic management of skipping rules is a strength of data collection computer assisted techniques. Concerning business surveys questionnaires, different suggestions are given for skipping rules related to questions belonging to the same section/web page or to different section/web pages. 6) The checking plan Suggestions are given on the management of the consistency rules, taking into consideration aspects, like: rules regarding variables of the same web-page or different

• nes, number of checking rules on a single web-page, relation between skipping and

checking rules, etc. 7) The screen design Standards already defined for software package used in the Business Portal architecture have been adopted. Some details have been provided on graphic symbols to be used to link definitions, instructions, helps, etc.

Manual of Recommended Practices

9

8) The guide and instructions for respondents A set of documents is always provided to respondents (e.g. brief description of the survey, normative aspects concerning confidentiality, compilation guide, etc.). Common structures, templates and access modes have been designed for these documents, because they can have an impact on the comprehension process as well. 9) Questions to estimate the perceived respondent burden The migration of surveys questionnaires in the new generalised sw system for electronic questionnaires implementation is a good occasion to collect information on respondent burden directly from respondents. For this purpose a set of questions has been proposed, which distinguish possible burden due to difficulty in retrieving the requested information and in filling in the questionnaire. The responses to these questions could be analysed together with other indications derivable from paradata. 10) The use of paradata for data collection monitoring and optimising questionnaire design The Business Portal already produces some indicators for the monitoring purpose. A set of

• ther indicators has been identified which can be used by the researcher to optimise the

questionnaire design, specifically when some ‘alerts’ are highlighted by the answers given to questions on the respondent burden (e.g. number of entries for web-pages, connection time for the entire questionnaire or for each web-page, length of navigation routings, etc.).

10

10

The structure for variables definitions The harmonisation of the contents to be observed

In business statistics some variables can be collected in different surveys

• questionnaires. This can depend on different informative needs (structural and short-

term surveys), while in other situations it could sometime result to be redundant. On the other hand, big business agencies are often involved in more surveys. Definitions of the same variables could be different:

in terms of their content because EU regulations contain different specifications for structural and short-term surveys, or in terms of wording, because expressed differently by each survey responsible.

Contents harmonisation in terms of variables definitions and questions wording is the pre-requirement for a two steps activity:

 Improvement of the harmonisation and comparability of collected data  Reduction of redundancies and respondent burden

11

11 A common modular structure for definitions of variables has been defined, which will be a standard for all variables definitions.

The same informative content can be presented to respondents through different wordings, different structures and different layouts. All these aspects have an impact on the comprehension process.

The structure for variables definitions The harmonisation of the contents to be observed

A Definition (core of the definition = real meaning, expressed in a simple, unambiguous and synthetic way) B Inclusions/Exclusions clauses C National peculiarities D Peculiarities of sectors/surveys

This structure is a useful tool for the improvement of harmonisation and comparability of collected

• data. It makes it easier:

 converging on the core of the definition  identifying differences of informative needs and, consequently, structuring questions so as to reduce redundancies in data collection

12

12 Two variables, used by different survey, have been considered with the purpose of harmonising definitions and testing the standard structure:  Number of employees  Turnover

Sources used to produce harmonised definitions  Eurostat Regulations on SBS and STS surveys  Definitions already used in Istat surveys (SBS and STS)  Indications produced by ESSNet project on consistency (WP3)  Definitions of Istat Business register  Definitions of Business census.

The structure for variables definitions The harmonisation of the contents to be observed

REFERENCES

14

• J.Bethlehem, S.Biffignandi (2011) Handbook of Web Surveys, eds Wiley
• Eurostat (2006) Handbook of Recommended Practices for Questionnaire Development and

Testing in the European Statistical System http://ec.europa.eu/eurostat/ramon/statmanuals/files/Handbook_of_Practices_for_Quest.pdf

• Eurostat (2014) Memobust handbook - Handbook on Methodology of Modern Business

Statistics http://www.cros-portal.eu/content/memobust

• N.Fazio, M.Murgia, A.Nunnari (2013) The Business Statistical Portal: a new way of
• rganising and managing data collection process for business surveys in ISTAT Proceedings
• f the UNECE Seminar on Data Collection, Ginevra
• Sudman, S., D. K. Willimack, E. Nichols and T.L.Mesenbourg (2000) “Exploratory

Research at the U.S. Census Bureau on the Survey Response Process in Large Companies,” Proceedings of the Second International Conference on Establishment Surveys, American Statistical Association,