NASA STEM Engagement Performance Assessment & Evaluation - - PowerPoint PPT Presentation

National Aeronautics and Space Administration

NASA STEM Engagement Performance Assessment & Evaluation Briefing

August 9, 2019

2

Rob LaSalvia, Chief, NASA Glenn Office of Education Rick Gilmore, Evaluation Manager Paragon TEC

• Dr. Catherine Graves, Program Manager/Sr. Evaluation Specialist
• Dr. Tara Strang, Evaluation Specialist
• Dr. Anne Kotynski Gooding, Evaluation Specialist

Clarence Jones, Technology Coordinator/OEPM Data Manager

NASA STEM Engagement Performance Assessment, Evaluation, & Information Management (PAEIM) Team

NASA STEM Engagement Performance Assessment & Evaluation Briefing

Agenda

• NASA STEM Engagement FY 2018

Performance Data Summary (Preliminary)

• NASA STEM Engagement Performance &

Evaluation Strategy

• Higher Education Challenges,

Competitions, and Internships Multiple Case Study Summary of Findings

• NASA Office of STEM Engagement

(OSTEM) Programs Principal Investigator (PI) Effective Practices Panel Discussion Session

FOOD FOR THOUGHT

Ask Questions At: jsc.cnf.io Don’t forget to vote responses up!

3

4

NASA STEM Engagement FY 2018 Performance Data Summary (Preliminary)

FY 2018 NASA STEM Engagement Performance Assessment

5

Internships and Fellowships Research and Development

Collaborators

7,205

In Fiscal Year 2018, NASA provided 7,205 internships and fellowships to higher education students across all institutional categories and levels. These significant awards provided a total of over $14M in direct financial support to higher education students.

27.3% of higher education internships

and fellowships were awarded to racially

• r ethnically underrepresented student

participants, compared to 24.5% for the national average of STEM degree enrollees. Additionally, 39.6% of the Agency’s higher education internships and fellowship positions were filled by women. NASA’s performance in providing

• pportunities for learners to contribute to

NASA’s aeronautics, space, and science missions and work is assessed across peer- reviewed publications and technical paper presentations directly resulting from research funded by NASA STEM Engagement grants and awards to higher education institutions. Space Grant, MUREP, and EPSCoR grantee and awardee institutions reporting 1,324 peer-reviewed publications and technical papers and presentations in FY 2018. NASA’s Office of STEM Engagement collaborators are funded and unfunded and located in all 50 states, DC, GU, PR, and VI. Collaborators include: government agencies, industry, formal and informal education institutions including museums, science centers, planetariums, and youth- serving organizations, non-profit, and

• ther education organizations.

Collaborators extend the reach of NASA STEM engagement opportunities by supporting the execution of an

• pportunity. In FY 2018 OSTEM

collaborated with 2,214 institutions and

• rganizations.

2,214

1,324

4

6

NASA STEM Engagement Performance and Evaluation Strategy

Legislative Directives that Guide the Office of STEM Engagement’s Work

7

Public Law 85-568 The National Aeronautics and Space Act of 1958.

Created NASA and directed that the agency should pursue several goals, including the pursuit of educational activities.

Public Law 111-352 (H.R. 2142) Government Performance and Results Act – Modernization Act (GPRAMA) of 2010.

Requires quarterly performance assessments of Government programs for purposes

• f assessing agency performance and improvement.

Reauthorization Act of 2015 (H.R. 1806) America COMPETES Act.

Funds investments in science and engineering research and STEM education from kindergarten to the postdoctoral level. Established the National Science and Technology Council’s Committee on STEM Education (CoSTEM) to: (1) collaborate with the STEM Education Advisory Panel and other outside stakeholders to ensure the engagement of the STEM education community, and (2) review evaluation measures used for federal STEM education programs.

Public Law 114-329 (S. 3084) American Innovation and Competitiveness Act.

The act authorizes creation of an interagency advisory panel and working groups to consider education for STEM fields and supports the coordination of citizen science and crowdsourcing by Federal agencies to accomplish their missions.

Public Law 114-264 (S. 1550) Program Management Improvement Accountability Act (PMIAA).

Enhances accountability and best practices in project and program management throughout the federal government.

H.R. 4174 Foundations for Evidence-Based Policymaking Act of 2017.

Requires the measurement of the federal government’s effectiveness through the use of program evaluation, continuous improvement, policy-relevant research, and cost-benefit analyses by qualified researchers and institutions.

H.R. 4887 Grant Reporting Efficiency and Agreements Transparency (GREAT) Act.

Requires the creation of a comprehensive and standardized data structure, or “taxonomy”, covering all data elements reported by recipients of federal awards, including both grants and cooperative agreements. 7

8

Strategic Alignment of Studies (Illustrative)

9

Strategic Alignment of Studies (Illustrative)

10

Performance Assessment & Evaluation Overview

10

11

FY19 & FY20 Office of STEM Engagement Performance Assessment Model

12

NASA Higher Education Challenges, Competitions, and Internships Multiple Case Study Summary of Findings

Introduction

13

PURPOSE:

The overarching purpose of this basic research study was to define the key constructs underpinning NASA’s goals for STEM engagement in relation to higher education activities. NASA’s goals for STEM engagement put forth that STEM engagement activities should:

• A. Enable contributions to NASA’s work;
• B. Build a diverse, skilled future STEM workforce; and
• C. Strengthen understanding of STEM through powerful

connections to NASA.

QUESTIONS:

• 1. In what ways are NASA STEM engagement activity higher

education participants contributing to NASA’s aeronautics, space, and/or science missions?

• 2. What evidence-based and practice-based effective strategies

for STEM learning are used in the design and implementation

• f NASA STEM engagement higher education activities?
• 3. What evidence-based and practice-based effective strategies

for the recruitment and retention of participants from groups historically underrepresented or underserved in STEM fields are used in the design and implementation of NASA STEM engagement higher education activities?

UTILIZATION-FOCUSED APPROACH:

Designed for the actual use and application of evaluation findings (Patton, 2015)

13

Methodology & Data Collection

In what ways are NASA STEM engagement activity higher education participants contributing to NASA’s aeronautics, space, and/or science missions?

• Open-ended questionnaires
• Document (content) analysis of activity plans, performance and evaluation reports,

presentations, and mission directorate strategic planning documents, goals, and priorities

• Literature review

Qualitative Analysis What evidence-based and practice-based effective strategies for STEM learning are used in the design and implementation of NASA STEM engagement higher education activities?

• Open-ended questionnaires
• Document (content) analysis of activity plans, performance and evaluation reports,

presentations, and mission directorate strategic planning documents, goals, and priorities

• Literature review

Qualitative Analysis

What evidence-based and practice-based effective strategies for the recruitment and retention of participants from groups historically underrepresented or underserved in STEM fields are used in the design and implementation of NASA STEM engagement higher education activities?

• Open-ended questionnaires
• Document (content) analysis of activity plans, performance and evaluation reports,

presentations, and mission directorate strategic planning documents, goals, and priorities

• Literature review

Qualitative Analysis

EVALUTION QUESTIONS DATA SOURCE ANALYSIS

14

15

NASA Internships

X-Hab COSGC RASC-AL NCAS

eXploration Systems & Habitation Academic Innovation Challenge Colorado Space Grant Consortium Statewide program that provides students access to space through higher education courses, telescope and satellite programs, and interactive

• utreach programs.

Provide post-secondary students with experiences that prepare them for our Nation’s future space programs. Depends on specific activity; includes science investigations, technical papers, technology. Revolutionary Aerospace Systems Concepts Academic Linkage Higher education competition with applications to human space exploration. Supports the development

• f deep space exploration

architecture with the goal

• f advancing human

spaceflight missions. Prototypes that respond to challenge. NASA Community College Aerospace Scholars

Study Participants

Activity managers and key support staff associated with five instrumental cases

Competitive awards to support educational

• pportunities that provide

unique NASA-related research and operational experiences . Leverage NASA’s unique mission activities to enhance and increase the capabilities, diversity, and size of the nation’s next generation workforce needed to enable future NASA discoveries Mentor-defined; includes science investigations, technical papers, technology. Bridge strategic knowledge gaps and increase knowledge in capabilities and technology risk reduction related to NASA’s vision and missions. Prototypes that respond to challenge. Higher education challenge designed to develop strategic partnerships and collaborations with universities. Interactive online learning

• pportunity and in-person

experience at a NASA Center. To encourage community college students in STEM to complete their 2-yr degree, go on to a 4-yr degree and pursue a STEM career. To provide tools and motivation for students to pursue other STEM undergraduate education

• pportunities.

Online course & engineering design challenge.

NASA STEM Engagement Strategy Alignment

16

Goal Objective Internships X-Hab COSGC RASC-AL NCAS

Enabling Contributions to NASAs work. 1.1 Students contribute to NASA’s endeavors in exploration and discovery. X X X 1.2: Research and development capacity of educational institutions is enhanced, enabling broad and diverse contributions that directly address NASA priorities. ’ X Build a Diverse, Skilled Future STEM Workforce 2.1: A broad and diverse set of students are attracted to STEM through NASA

• pportunities.

X X X 2.2: Students, including those from underrepresented and underserved communities, explore and pursue STEM pathways through authentic learning experiences and research opportunities with NASA’s people and work. X X X 2.3: The portfolio of NASA STEM engagement opportunities meets agency workforce requirements and serves the nation’s aerospace and relevant STEM needs. X X X X X 2.4: Strategic partnerships with industry, academia, non-profit organizations and educational institutions enhance and extend the impact of NASA’s efforts in STEM engagement. X X X X Strengthen Understanding of STEM through Powerful Connections to NASA. 3.1: Youth are introduced to STEM concepts and content through readily available NASA STEM engagement resources and content. X X 3.2: Students gain exposure to STEM careers through direct and virtual experiences with NASA’s people and work. X X X

NASA STEM Engagement Performance Goal Alignment

17

Performance Goal Annual Performance Indicator Internships X-Hab COSGC RASC-AL NCAS

3.3.5: Provide

• pportunities for students

to contribute to NASA’s aeronautics, space, and science missions and work in exploration and discovery. Conduct a multiple case study focused on NASA STEM engagement higher education challenges, competitions, and internships to build knowledge about how these activities: a) contribute to NASA's aeronautics, space, and science missions; b) align to evidence-based effective practices for STEM learning; and c) recruit and retain participants from groups historically underrepresented and/or underserved in STEM fields. X X X X X Contribute to American technical capability by supporting the release

• f at least one exit presentations and peer-reviewed research

publications through STEM engagement investments. X X X 3.3.3. Provide

• pportunities for students

to engage with NASA’s aeronautics, space, and science people, content, and facilities in support of a diverse future NASA and aerospace industry workforce. Provide significant, direct student awards in higher education to (1) students across all institutional categories and levels (as defined by the U.S. Department of Education), (2) racially or ethnically underrepresented students (Hispanics and Latinos, African Americans, American Indians, Alaska Native, Native Hawaiians and Pacific Islanders), (3) women, and (4) persons with disabilities, at percentages that meet or exceeded at the national percentages for the science and engineering graduates, as determined by the most recent, publicly available data from the U.S. Department of Education’s National Center for Education Statistics for a minimum of two of the four categories. X X X

Higher Education STEM Engagement Logic Model

18

• NASA-unique resources (mission

needs, facilities, subject matter experts, mentors)

• Literature on evidence-based

practices

• Recruitment plans for

underrepresented students

OUTPUTS (Activities)

• Internships
• Challenges and competitions
• Research opportunities
• Education programs (on-site and
• nline learning)
• Industry collaborations
• Outreach programs
• Mentorship
• Recruitment

SHORT-TERM OUTCOMES (Impacts)

• Low-cost technology

development

• Students’ increased knowledge in

and exposure to technologies, capabilities, and operational approaches related to human spaceflight

• Students’ increased workplace-

relevant skills (professionalism, teamwork, critical thinking)

• Innovative strategic partnerships
• Students support faculty and SME

research

LONG-TERM OUTCOMES (Goals)

• Increased student persistence in

academic STEM disciplines

• Students’ career development that

strengthens NASA and the nation’s future STEM workforce

• Increased representation of

diverse student groups pursuing STEM academic disciplines and careers

• Support to raise NASA’s

Technology Readiness Levels

INPUTS (Resources)

• NASA-unique resources (mission

needs, facilities, subject matter experts, mentors)

• Literature on evidence-based

practices

• Recruitment plans for

underrepresented students

Theory of Change: If higher education students participate in NASA authentic STEM experiences (e.g., internships, competitions,

challenges, and activities), NASA will benefit from their contribution to missions and a diverse group of students will be better equipped to persist in STEM academic and career pursuits.

Contributions to NASA Missions

• Activities that produce

knowledge or products that will be used by NASA

• Activities that support NASA

SME work and research tasks

• Activities that promote NASA’s

future workforce

Cross-Case Synthesis: Practice-Based Findings for Replication

Practice-based Effective Strategies for STEM Learning

Recruitment & Retention of Students Historically Underrepresented

• Application of evidence-based

practices

• Mentor-defined
• Task-defined
• Workshop model
• Mentorship
• Partnership with MUREP and/or

MUREP-funded activities

• Engagement with relevant societies

for target groups of students

• Student or alumni support

recruitment

• Hands-on recruitment content
• Prolonged period of engagement

with target population

• Faculty engagement
• Diverse program

leadership/mentorship

• Minority Serving Institution focus

19

Cross-Case Synthesis: Alignment to Evidence Based Practice

20

Categories Evidence-based Effective Strategies List NASA Higher Education Challenges, Competitions, & Internships Internships X-Hab COSGC RASC-AL NCAS 1. Practices for higher education student activities that contribute to STEM a. Work-based learning X X X b. Academic undergraduate research experience X X X X X 2. Practices for higher education student STEM learning a. Active learning X X X X X b. Engagement in research practices X X X X X 3. Practices for recruitment/retention of students historically underrepresented a. Institutional supports for STEM readiness b. Evidence-based learning practices X X X c. Research experiences X X X d. Mentorship X X X e. Engagement with 2-year and/or Minority Serving Institutions X X X

Recommendations

21

A series of evidence-based and practice-based effective strategies for enabling higher education students to contribute to NASA’s missions were developed based on the study’s findings.  Prioritize work-based learning and undergraduate research experiences that enable higher education students to contribute to NASA’s missions.  Build upon good practices identified in the case studies to create a framework for challenge/competition development that ensures alignment with documented mission needs.  Develop a set of internal standards for activity developers to ensure that work-based learning experiences align with the needs of NASA’s workforce and literature.  Support activity managers of new and existing higher education student experiences with alignment of design and operations to evidence and practice based strategies  Build upon collaborations with MUREP as a mechanism to support diversity goals.  Identify and align practical experiences to institutional supports within universities (course design, faculty research efforts, etc.).  Use future evaluation work to support NASA in refining student higher education experiences.  Refine objective and measures associated with higher education student activities to recommended logic model.  Survey NASA SMEs and other relevant workforce about higher education students’ contributions.  Conduct a longitudinal examination of work-based learning or research experiences on student participants’ STEM persistence and career outcomes.  Conduct an examination of work-based learning or research experiences on student participants’ science identity and other short-term outcomes of interest

22

NASA Office of STEM Engagement (OSTEM) Programs Principal Investigator (PI) Effective Practices Panel Discussion Session

Overview of OSTEM Programs Principal Investigator (PI) Effective Practices Panel Discussion Session

23

• How does your program engage students in NASA’s missions,

people, content, and facilities in support of a diverse future NASA and aerospace industry workforce? (Engagement of diverse audiences)

• How does your program contribute to NASA’s missions and

work in exploration and discovery? (Contributions to NASA research)

Goal

Highlight some example effective practices, activities, and/or strategies that the Space Grant, MUREP and EPSCoR programs implement in support of NASA STEM Engagement performance goals and objectives.

Key Question(s)

Guest Panelists Introduction

Kuldeep Rawat, Ph.D.

Dean, School of Science, Aviation, Health, and Technology, Elizabeth City State University and MUREP Aerospace Academy (MAA) Director

ksrawat@ecsu.edu https://www.ecsu.edu/academics/programs/n asa-aero-acad/index.html

Nathan L. Harshman, Ph. D.

Professor and Department Chair of Physics, American University and Director, District of Columbia Space Grant Consortium

harshman@american.edu https://www.dcspacegrant.org/

Edward F. Duke, Ph.D.

Professor, South Dakota School of Mines and Technology, and South Dakota EPSCoR and Space Grant Consortium Director

Edward.Duke@sdsmt.edu http://sdspacegrant.sdsmt.edu/Default.htm

24

Space Grant Guest Panelist

NASA STEM Engagement Performance Goal Support:

• Engagement of diverse audiences
• Contribution to NASA research

Nathan L. Harshman, Ph. D.

Professor and Department Chair of Physics, American University and Director, District of Columbia Space Grant Consortium

harshman@american.edu https://www.dcspacegrant.org/

25

Partners

Washington D.C. Space Grant Consortium

• Georgetown University
• Space Science

Professionals International (Mid- Atlantic Section)

• Smithsonian National Air

and Space Museum

26

Program Facts

Team

• Nate Harshman, Director
• Megan Kemble, Asst.

Director

• Eric Day, Program Manager

Lead Institution American University Washington D.C.

Affiliates

• American University
• Catholic University
• Gallaudet University
• George Washington

University

• Howard University
• Trinity Washington

University

• University of District of

Columbia

• INSPIRE, Inc.
• Partnerships with MSIs and close contact with affiliates
• Integration of diversity goals within proposal and review process
• Focus on disseminating research-based practices for mentorship

Key Strategies FY18 Diversity Data

53.3 46.7

34% Underrepresented Target 53% Underrepresented Actual 59% Female Target 62% Female Actual

61.7 38.3

MUREP Guest Panelist

NASA STEM Engagement Performance Goal Support:

• Engagement of diverse audiences
• Contribution to NASA research

Kuldeep Rawat, Ph.D.

Dean, School of Science, Aviation, Health, and Technology, Elizabeth City State University and MUREP Aerospace Academy (MAA) Director

ksrawat@ecsu.edu https://www.ecsu.edu/academics/programs/n asa-aero-acad/index.html

27

Partnerships

STEM Ecosystem

• School districts
• Non-profits
• Governmental

agencies

• Private
• rganizations
• Other STEM

enrichment programs

• Advisory boards

28

Program Highlights

Mobile STEM Lab Roadshow-in-a-Box that helps providing exposure and create awareness about the program. Residential Camp For student in rural areas, plagued with lack of transportation.

Activities

• Theme-based with focus on engineering design

challenge and team-based competitions

• Topics focused on NASA Aeronautics and NASA

Science mission directorates

• Capitalize on students’ interest - robotics and

drones, 3D printing, Hot-air balloon, and Rocketry 3 Strategies to Implement Program Activities 1. Nurturing students’ enthusiasm for STEM 2. Improving students’ competence in STEM 3. Interesting students in research or other STEM- related careers

• Seek additional funding opportunities.
• Leverage NASA grant, project, and results to secure additional funding

Seeking Additional Funding

FY16 - FY18 Diversity Data

ECSU MUREP Aerospace Academy (MAA)

• 2,622 K-12 Students

60% Underrepresented Target 66% Underrepresented Actual 50% Female Target 51% Female Actual

66 34 51 49

EPSCoR Guest Panelist

NASA STEM Engagement Performance Goal Support:

• Contribution to NASA research

Edward F. Duke, Ph.D.

Professor, South Dakota School of Mines and Technology, and South Dakota EPSCoR and Space Grant Consortium Director

Edward.Duke@sdsmt.edu http://sdspacegrant.sdsmt.edu/Default.htm

29

2012 – 2019 Case Study (Dr. Alevtina Smirnova)

South Dakota NASA EPSCoR FY2007 – FY2018

2012: Travel grant to NASA Glenn, presentation on advanced batteries and fuel cells 2013:

• SD NASA EPSCoR Research seed grant ($50,000),

collaborate with NASA Glenn

• SD NASA EPSCoR and NSF EPSCoR sponsor advanced

battery workshop

• NASA Glenn Faculty Fellowship (summer)

2013 – 2014: Students work at NASA Glenn; 6 students receive SD Space Grant Fellowships 2014 – 2018: NASA EPSCoR Research Award for Advanced Batteries ($750,000), collaborate with NASA Glenn 2018: DoD SBIR Phase 1 and 2 ($1.15 million) 2019: NSF IUCRC Center for Green Solid-state Electric Power Generation and Storage (includes NASA Glenn)

Research Infrastructure Development

• 21 Research Seed Grants

($570,000)

• 5 Tribal College Research

Collaboration Grants ($56,000)

• 76 Research Planning trips (136

state researchers, nine NASA Centers)

Competitive Awards

• 12 NASA EPSCoR Research Awards

($9 million)

• Minority-Serving Institution Faculty

Engagement Grant ($250,000, Oglala Lakota College)

• 2 Rapid Response Research Grants

($200,000)

• 94 Faculty and post-doctoral researchers
• 208 graduate students and 192 undergraduate students
• 440 peer-reviewed journal articles
• 579 other publications and presentations
• 8 patent applications, 4 patents, 2 start-up companies
• 392 external collaborations (NASA, other federal, academia, industry)
• More than $60 million follow-on grants

Impact

30

Panel Discussion Wrap Up

31

FOOD FOR THOUGHT

Ask Questions At: jsc.cnf.io Don’t forget to vote responses up!

32

Questions

Questions regarding the NGS Pilot Evaluation should be addressed to: Rob LaSalvia, Chief NASA Glenn Office of Education Robert.F.LaSalvia@nasa.gov Rick Gilmore, Evaluation Manager Richard.L.Gilmore@nasa.gov

33