Evaluating the Impact of Your Intervention: From Research Question - - PowerPoint PPT Presentation

Evaluating the Impact of Your Intervention: From Research Question to Dissemination

Jeffrey T. Olimpo, Ph.D. BUILDing SCHOLARS e-Conference

Disclaimer

This e-Conference, meeting materials, and related content are solely the responsibility of UTEP and do not necessarily represent the official views of the National Institutes of Health.

• What is discipline-based education research (DBER)?
• What is the process for conducting DBER work?
• Where and in what ways can I share my findings?

Session Overview

• DBER studies “investigate learning and teaching

in a discipline from a perspective that reflects the discipline’s priorities, worldviews, knowledge, and practices.” (NRC, 2012)

• In a more practical sense, DBER studies

examine how students (and/or other key stakeholders) come to understand and apply the knowledge, concepts, and practices of a particular discipline.

DBER Defined

Research Process

Identification of a Research Question Formulation of a Problem Study Design & IRB Review Observation in the Classroom Data Collection & Analysis Synthesis of Results Incorporation of Theory Dissemination

• What am I interested in exploring? In what context?
• Who will be involved in the study, and who will I be

collecting data from throughout the study?

• What data will I collect? How and when will data

collection take place?

• How do I plan to disseminate the results of my study?

Getting Started: Questions to Consider

• Multiple outcomes can be explored, including the impact
• f your intervention on participants’:
• Content knowledge and/or conceptual understanding
• Attitudes, motivations, beliefs, and/or perceptions
• Professional skills development
• Level of engagement within a specific learning environment
• Depending upon the central focus of your research,

multiple study populations (e.g., students; faculty; administrators; external stakeholders) could be included as part of your work.

Identifying a Focus for Your Research

• Should be aligned to your research question(s) and take

into account the number of individuals you anticipate enrolling in your study (i.e., sample size)

• Use of previously-validated measures is preferable.
• Quantitative Approaches:

Data Collection Methods

** Pay Attention to Number of Constructs/Factors Included ** Concept Inventories (e.g., IMCA) Likert-item Questionnaires/Surveys Scenario-based Prompts (e.g., E-EDAT) “Visualization” Tasks (e.g., SPFA)

• Qualitative Approaches:
• Coding of participant responses to open-ended prompts
• Interviews (semi-structured/focus group)
• Participant observations or evaluation of student artifacts
• Mixed methods research = quantitative + qualitative

Data Collection Methods (Cont’d)

Pulling It All Together

Research Question Instrument(s) Methods of Analysis Study Population(s) Timing of Data Collection

Pulling It All Together

Research Question Instrument(s) Methods of Analysis Study Population(s) Timing of Data Collection What impact does the XYZ activity have

• n students’

development of scientific process skills? Scientific Process Flowchart Assessment (SPFA) Paired t-tests with Bonferroni correction, as needed All student participants Pre-/post-activity What aspect of the XYZ activity did students find most beneficial, and why? Open-ended prompt Content analysis of students’ responses to the prompt All student participants Post-activity

• Your Institutional Review Board (IRB) oversees all

research involving human subjects.

• The IRB review process is important for…
• Adhering to ethical research practices
• Obtaining and renewing grants
• Presenting and/or publishing your work
• Protection of investigators and institutions

The IRB Process

Exempt Review Expedited Review Full Review

• DBER/SoTL Conferences:
• National Association for Research in Science Teaching (NARST)
• Society for the Advancement of Biol. Educ. Research (SABER)
• AAC&U Transforming STEM Higher Education Conference
• Council for Undergraduate Research (CUR)
• DBER/SoTL Journals:

STEM Education Conferences/Journals

Contact Information

Jeffrey T. Olimpo, Ph.D.

• Dept. of Biological Sciences

University of Texas at El Paso jtolimpo@utep.edu