Using the Wheel Without Reinventing It: How to examine Noyce - PowerPoint PPT Presentation

Using the Wheel Without Reinventing It: How to examine Noyce projects Part II Presented by: John Sutton, Ph.D. Brandie Ward Emma Espel, Ph.D. 1 1

Goals We hope you leave the webinar better prepared to answer two questions: What instruments could I use for my Noyce project evaluation? 2

Who are we? External evaluators from RMC Research Corporation in Denver, CO 6 current Noyce projects evaluations across CO, NE, and LA • 1 capacity building project • 2 scholarship and stipend projects 2 TF/MTF projects • 1 evaluation of the Midwest Regional Connections • Noyce project 3

4 4

Outline Brief review of Noyce grant types Common guidelines for what makes a good instrument Rubric review Instruments & Dialogue Let’s share! 5 5

Capacity Scholarship & Building Stipend (S & S) Noyce Project Types Teaching Fellows (TFs) Conferences/ /Master Teaching Events Fellows (MTFs) 6

Webinar Poll #1 What is your primary role with a Noyce project? a. PI or Co-PI b. Evaluator c. Project Staff d. Other? 7 7

What would you do? Chat z Imagine yourself…as the evaluator for a TF/MTF program funded for five years. Your evaluation plan includes annual online surveys of teachers about their content knowledge, instructional strategies used, and demographics. After three years of program implementation and evaluation, the TF/MTF program leaders would like the survey to collect information about teachers’ agreement with productive or unproductive statements about the teaching and learning of mathematics. They are also interested in the extent to which their students are achieving the Standards for Mathematical Practices. How might you customize the survey to include these new scales, three years into the project? What are some considerations you need to think about when adding the scales? 8

Perhaps… Retrospective & Concurrent • Identify participants’ agreement with the statements prior to participation in the program Ask participants to reflect on whether or not their • agreement has changed over the past three years. Include items with concurrent wording in all • upcoming assessments in order to monitor moving forward. Do any previous measures tap similar constructs? • 9 9

Common guidelines for what makes a good instrument 10 10

Chat z What qualities do you look for in a good instrument? 11 11

Perhaps… Instrument content that helps answer evaluation questions Appropriate for the context type of respondents • Appropriate psychometric properties reliability, validity, wording • 12 12

Review of our evaluation rubric 13 13

Chat z Provide an example of an evaluation question you may commonly use across any type of evaluation that addresses stakeholders’ attitudes & beliefs about the project. 14 14

Scholarship/ Events (e.g., Capacity Building TF/MTF Stipend Program Conferences) Construct To what extent… are scholarships/ Implementation stipends awarded according to review criteria? do scholars perceive do participants in Attitudes & do stakeholders view do TFs/MTFs apply new the project as sponsored activities the project as a viable ideas, knowledge, and Beliefs implementing a high perceive new ideas, option for partner information in classroom quality teacher knowledge, or institutions? practice? preparation program information as useful focused on STEM? and applicable? does the scholarship Content project have an impact Knowledge on scholars’ Praxis scores? do TFs/MTFs impact Student student outcomes? Outcomes Satisfaction are participants satisfied with event activities? Sustainability has progress been made towards sustaining or “scaling 15 15 up” project activities?

Scholarship/ Events (e.g., Capacity Building TF/MTF Stipend Program Conferences) Construct To what extent… are scholarships/ Implementation stipends awarded according to review criteria? do scholars perceive do participants in Attitudes & do stakeholders view do TFs/MTFs apply new the project as sponsored activities the project as a viable ideas, knowledge, and Beliefs implementing a high perceive new ideas, option for partner information in classroom quality teacher knowledge, or institutions? practice? preparation program information as useful focused on STEM? and applicable? does the scholarship Content project have an impact Knowledge on scholars’ Praxis scores? do TFs/MTFs impact Student student outcomes? Outcomes Satisfaction are participants satisfied with event activities? Sustainability has progress been made towards sustaining or “scaling 16 16 up” project activities?

Capacity Building Scholarship/ Stipend Program TF/MTF Events (e.g., Conferences) Construct To what extent… do stakeholders perceive the do stakeholders perceive the do stakeholders perceive the do stakeholders perceive the Attitudes & project as implementing a project as implementing a project as implementing a event as high quality ? Beliefs high quality teacher high quality teacher high quality teacher preparation program focused preparation program focused preparation program focused How does on STEM? on STEM? on STEM? the program impact does the program impact does the event impact does the program impact does the program impact attitudes and scholars’ interest in teaching participants’ interest in STEM? students’ interest in STEM? TFs/MTFs’ interest in teaching beliefs? STEM? STEM? does the program impact does the program impact does the event impact does the program impact students’ motivation to students’ motivation to teach participants’ motivation to TFs/MTFs’ motivation to teach engage in STEM? STEM? engage in STEM? STEM? do recruited students feel they does the program impact does the event impact does the program impact have the ability to engage in participants’ self-efficacy in participants’ self-efficacy in participants’ self-efficacy in STEM ( self-efficacy )? learning STEM? STEM? teaching STEM? do stakeholders view the do scholars feel prepared to do participants feel better do TFs/MTFs feel prepared to project as a viable option for teach (e.g., STEM content, prepared to more effectively teach (e.g., STEM content, partner institutions instructional practices, participate in and contribute instructional practices, (preparedness)? teaching in high needs to their program activities? teaching in high needs schools)? schools)? do challenges exist in utilizing do challenges exist in do challenges exist in new ideas, knowledge, or transferring what TFs/MTFs transferring what scholars are information from events (and are learning into the learning into the classroom what are they)? classroom (and what are (and what are they?) they?) do attitudes and beliefs about do attitudes and beliefs about teaching and learning of STEM teaching and learning of STEM change? change? 17 17

Capacity Building Scholarship/ Stipend Program TF/MTF Events (e.g., Conferences) Construct To what extent… do stakeholders perceive the do stakeholders perceive the do stakeholders perceive the do stakeholders perceive the Attitudes & project as implementing a project as implementing a project as implementing a event as high quality ? Beliefs high quality teacher high quality teacher high quality teacher preparation program focused preparation program focused preparation program focused How does on STEM? on STEM? on STEM? the program impact does the program impact does the event impact does the program impact does the program impact attitudes and scholars’ interest in teaching participants’ interest in STEM? students’ interest in STEM? TFs/MTFs’ interest in teaching beliefs? STEM? STEM? does the program impact does the program impact does the event impact does the program impact students’ motivation to students’ motivation to teach participants’ motivation to TFs/MTFs’ motivation to teach engage in STEM? STEM? engage in STEM? STEM? do recruited students feel they does the program impact does the event impact does the program impact have the ability to engage in participants’ self-efficacy in participants’ self-efficacy in participants’ self-efficacy in STEM ( self-efficacy )? learning STEM? STEM? teaching STEM? do stakeholders view the do scholars feel prepared to do participants feel better do TFs/MTFs feel prepared to project as a viable option for teach (e.g., STEM content, prepared to more effectively teach (e.g., STEM content, partner institutions instructional practices, participate in and contribute instructional practices, (preparedness)? teaching in high needs to their program activities? teaching in high needs schools)? schools)? do challenges exist in do challenges exist in utilizing do challenges exist in transferring what scholars are new ideas, knowledge, or transferring what TFs/MTFs learning into the classroom information from events (and are learning into the (and what are they?) what are they)? classroom (and what are they?) do attitudes and beliefs about do attitudes and beliefs about teaching and learning of STEM teaching and learning of STEM change? change? 18 18