Conducting survey methods research in an official statistics - - PowerPoint PPT Presentation

Conducting survey methods research in an

• fficial statistics production environment:

Why is it hard? How do you make it work?

Jaki S. McCarthy USDA’s National Agricultural Statistics Service B d d Based on a poster presented at the Journal Of Official Statistics 30th Anniversary Conference June 2015

Research in Survey Methods is Key to Improving Statistics y p g

• Survey methods research can be conducted outside the statistical

production environment

– Stand alone experiments – Analysis of existing data Analysis of existing data

• Pros

N i t i d t – No impact on on-going data – May be easier to manipulate conditions

• Cons

– Doesn’t mimic real survey conditions – Results often out of context, infeasible or do not translate to operations

Research embedded in production provides unique benefits, but……

h i d i i !

• Research in production is HARD!
• What can we do to address that?
• What can we do to address that?

COMMUNICATION!

• COMMUNICATION
• COMMUNICATION
• COMMUNICATION

Wh i it h d? Why is it hard? Wh t d t dd th t? What can you do to address that?

“The two words 'information' and 'communication' are often used interchangeably, but they signify quite different things. Information is giving out; communication is getting through ” Information is giving out; communication is getting through.

• --Sydney J. Harris

No benefit is seen in doing research

• Choose your problems wisely!

y p y

– Understand what problems are important to management or ti l t ff

• perational staff

– Vet research with operational staff – Enlist program management support for research – Clearly state how your research outcomes can be beneficial

Or…..clear benefits are perceived so operational staff want to jump right to production j p g p

• Explain how research can demonstrate benefits (or costs)

p ( )

– Be clear when answers are not known – Understand potential negative impacts that should be evaluated – Explain how research can MEASURE benefits and costs – Or be used to evaluate what NOT to do Or be used to evaluate what NOT to do

Research staff do not understand production environment

• Do your homework!

y

– Researchers must fully understand how surveys work, what is required, survey timelines and dependencies Know who in your organization is responsible for what – Know who in your organization is responsible for what – Build relationships with operational staff – go grab a beer!

Concern about negative impacts

• n survey operations
• n survey operations
• Understand what is required BOTH to conduct research and

q to implement findings

– Do as much as you can to reduce effort needed from operational staff staff – Find operational partners to help you understand how to get research embedded in operations – Make sure any findings CAN be implemented – Work with operational staff to understand HOW findings would be implemented – Be willing to do development work beyond your research

Operational staff are busy getting their jobs done

• Share your results and then share them again

y g

– Bring research findings to operational staff – Document results and archive them where people can find them C i h hi k h ld h ( i h – Communicate more than you think you should have to (with everyone)

Concern about disrupting time series, even if improvements can be made p

• Plan for phased implementation of research, if necessary

p p , y

– Remember research results are not research implementation – With your results, develop a realistic plan for phased implementation C l f h h dl di i i i i – Create a plan for how to handle disruption in time series

You can’t do anything without good communication!

• You can do the best research and be making the strongest

g g intellectual argument, but if readers don't get past the third paragraph you've wasted your energy and valuable ink. Carl Hiaasen Carl Hiaasen