Improving Data Quality in the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS) Peter Einaudi, RTI International Jonathan Gordon, RTI International Stephanie Eckman, RTI International Herschel Sanders, RTI International Mike Yamaner, National Science Foundation www.rti.org RTI International is a registered trademark and a trade name of Research Triangle Institute.
What is the GSS? Establishment survey of all Survey of Graduate Students U.S. academic institutions and Postdoctorates in Science that grant graduate degrees and Engineering (GSS) in science, engineering, and health (SEH) fields – Sponsored by the National Center for Science and Engineering Institutions report enrollment Statistics (NCSES) within the and financial support data for National Science Foundation graduate students, postdoctoral (NSF) and the National Institutes researchers, and doctorate- of Health (NIH) holding nonfaculty researchers (NFRs)
Sample data grid Voluntary survey High response rates – 98.6% RR3 in 2016 Burdensome – 56.4 hours average Electronic Data Interchange (EDI) limited prior to 2016
Improving the value of the data – Users want to analyze master’s and doctoral data separately Aligning taxonomy of disciplines across federal surveys Key Challenges Ensuring all eligible units are reported and classified appropriately Mitigating response burden and maintaining high response rates
Collect Separate master’s and doctoral more enrollment data data Increase use of EDI Goals of Reduce Simplify coding of academic units Burden Redesign Eliminate data entry errors Improve Align request with administrative data Quality Improve coding of disciplines
Proposed Solutions Simplify and expand use of EDI Replace GSS codes with Classification of Instructional – Align data request with Programs (CIP) respondents’ administrative data systems – Already utilized and maintained – Reduce reliance on manual by institutions for mandatory data entry reporting – Facilitate separate master’s – Improved comparability with and doctoral reporting without other federal surveys added burden – More granular coding scheme – Shifts burden of taxonomy changes from data reporters to data collectors
Fall 2015– Fall 2016– Fall 2017– Summer 2016 Summer 2017 Summer 2018 Institutional site visits Pilot Full implementation Recordkeeping and data Feasibility reporting practices Estimate reporting Coordinator survey burden Understand impact of changes
Fall 2015– Spring 2016– Fall 2017– Summer 2016 Summer 2017 Summer 2018 Institutional site visits Pilot Full implementation Recordkeeping and data Feasibility reporting practices Estimate reporting Coordinator survey burden Understand impact of changes
What we learned Coordinators can distinguish Most coordinators willing to between students enrolled use EDI in master’s and doctoral – Concerns about programming programs in their records effort required – Privacy concerns about transmitting individual-level data CIP codes available in student information systems – Codes less available to report postdocs
Integrating lessons learned into the Redesign Flexibility in field coding method Allay concerns about privacy – CIP codes strongly encouraged for – Avoid Personally Identifiable student data Information – GSS or CIP codes for – Option to aggregate data postdoctoral data before transmission Make EDI as simple as possible – Excel templates – Eliminate additional programming steps to format data
Fall 2015– Spring 2016– Fall 2017– Summer 2016 Summer 2017 Summer 2018 Institutional site visits Pilot Full implementation Recordkeeping and data Feasibility reporting practices Estimate reporting Coordinator survey burden Understand impact of changes
Trial Run: Pilot Data Collection GSS 2016 Ran parallel to GSS 2016 survey cycle – Stratified random sample of – Data request included: coordinators Separate reporting of master’s and doctoral students Master’s-only institutions ( n = 15) Use of CIP codes to report student data Doctorate-granting institutions Use of EDI to transmit data with 15 or fewer units ( n = 25) Doctorate-granting institutions with over 15 units ( n = 40)
Pilot Results 98.7% of coordinators Nearly 90% of schools able to upload at least reported student data some data with CIP codes All schools with both Most coordinators reported master’s and doctoral similar or lower response students able to report burden compared to the these data separately previous year
Average response burden for pilot coordinators
Fall 2015 – Spring 2016 – Fall 2017 – Summer 2016 Summer 2017 Summer 2018 Institutional site visits Pilot Full implementation Recordkeeping and data Feasibility reporting practices Estimate reporting Coordinator survey burden Understand impact of changes
Preparation for full implementation Targeted Communication Training – Mail – Webinars – E-mail – Training videos – Conference presentations – Sandbox Redesign of Survey Website – Highlight changes – Taxonomy tool
2017 Response method
Average response burden
Impact on response Response rates declined… but more data – Increase in school-level nonresponse Item nonresponse declined – Despite increase in items Lower item nonresponse with EDI
Item nonresponse rates by section and response method
Discussion Understand your Pilot your changes respondents and their data systems Communicate early and often Design around constraints: one size does not fit all
Thank You peinaudi@rti.org Peter Einaudi jgordon@rti.org Jonathan Gordon myamaner@nsf.gov Mike Yamaner
Recommend
More recommend
