AACT E s Bor tner Data Initiatives Data System I ntegr ation - - PowerPoint PPT Presentation

AACT E ’s Bor tner Data Initiatives

Data System I ntegr ation and Stakeholder E ngagement: Documenting E ffective P r actice and P er sistent Challenges

Mar ch 2, 2017 AACT E Annual Meet i ng T ampa, F l

• r

i da

www.aacte.org

NW NWMSU: : Assess ssme ment t System S Status us Prio rior to Nov

• v. 16

16 AAC AACTE Visit it

• Chal

hallenge enges i in n Qual uality A Assuran ance

– Integrating data from campus, program, state, testing systems into one quality control system – Engaging internal and external stakeholders in review, analysis and use of data for improvement

www.aacte.org

Data S System I Integration a n and Stakehol

• lder Engagem

emen ent

• Look

Looking ng t to

• the

he fiel eld f for

• r s

sol

• lut

utions ns and and innov nnovation

– Peer-directed case-studies will yield reports – Collective reflection will yield a framework – Participants from nine institutions/entities

• Brigham Young University
• Northern Michigan University
• Northwest Missouri State University
• The Ohio State University
• University of Delaware
• University of Louisville
• University of Texas Rio Grande Valley
• SUNY System Administration
• Michigan Department of Education

www.aacte.org

Doc

• cum

umen enting ng Effec ective P Prac actice and and Per ersisten ent Chal halleng enges es

• Report on Site Visit to Northwest Missouri State University

– Erica Brownstein, The Ohio State University – Christine Gorowara, University of Delaware; – Sungti Hsu, AACTE – Mark LaCelle-Peterson, AACTE – Joseph Lubig, Northern Michigan University; – Michael McBride, Northwest Missouri University; – Manish Sharma, University of Louisville; – Tim Wall, Northwest Missouri University; – Catherine Wigent, Michigan State Department of Education;

www.aacte.org

Data S System I Integration a n and Stakehol

• lder Engagem

emen ent

• Agenda f

genda for

• r t

the he se sess ssion

– Overview of the NWMU Data System since 2002 – Highlights of Topics Explored

• Preprogram RTI System
• Proactive Advisement

– Impact for Host Institution – Reflection from the visitors

Overview

Northwest Data Systems Overview

Northwest Missouri State University School of Education

www.aacte.org

2002: 2002: Tim Wal Wall, A Asse ssessment nt C Coor

• ordina

nator

• r
• Revamping the Praxis Score Record Keeping System:

Automatic Updating in three ways:

– 1. Student by student

• 2. Program by program
• 3. Across the Unit
• Teacher Work Sample

– Low scores on student teacher ability to use assessment – Focused on improving this

www.aacte.org

• The Northwest Education Programs began to systematically

use data and information proactively to support student learning and retention.

• Data use to solve challenges. EX: too many credit hours,

too low GPA; test score gaps by grouping; lack of information for advisors; need for a system.

• Example:
• Solution: development of Teacher Education Appeals

committee

– Recruitment, retention, support, application of remediation before…

The Begi he Beginning of

• f D

Dat ata Sy Systems at at N NW- 1999 1999-200 004

www.aacte.org

Development nt o

• f a

a System t throug ugh I h Iteration

• n,

Pr Professiona nal Lear Learning, and and Har ard Wor

• rk

1999 – “state of the art”

www.aacte.org

Development nt o

• f a

a System t throug ugh I h Iteration

• n,

Pr Professiona nal Lear Learning, and and Har ard Wor

• rk

2002 version of a dataset report

www.aacte.org

Development nt o

• f a

a System t throug ugh I h Iteration

• n,

Pr Professiona nal Lear Learning, and and Har ard Wor

• rk

2003 version of an advising report

www.aacte.org

Development nt o

• f a

a System t throug ugh I h Iteration

• n,

Pr Professiona nal Lear Learning, and and Har ard Wor

• rk

2003 –unpacking the national standards and implementing a system

www.aacte.org

Emphas asis o

• n stud

udent nt s suppor

• rt. - 2002

2002 “Qua Quant w with a a hear heart”

www.aacte.org

Un Unit a t asse sessm ssment s t syst stem- a step f p forward ben benchmarks and gat and gates- 2001 2001-200 004

Systems thinking (from 2002-2004)

www.aacte.org

Avo voiding ng t the he teacher eacher educat education of n off-ramp mp

18

Passing Score= 164

Color Key t o Underst and t his report : 1st t ime t est t aker

Not a first time test-taker Passed on 2nd attempt Took Praxis test not in their field so they were not entered into database Yellow is passed but below National average Green is at or above National Average RED = Failed Sep-04 Nov-04 # # # Mar-05 Apr-05 Jun-05 Student Score# Student Scor # Studen Score # Student Score # Student Score # Student Score Student 1 127 1 Student 2 172 1 Student 3 182 1

total # of students this administration total # of students this administrationtotal # of students this administration total # of students this administration total # of students this ad

Student 4 169 1

total passed total passed total passed total passed total passed

Student 5 177 1

total pass % total pass % total pass % total pass % total pass %

Student 6 164 4

1st time students 1st time students 1st time students 1st time students 1st time students

Student 7 160 1

1st time passed 1st time passed 1st time passed 1st time passed 1st time passed 1st-time pass rate 1st-time pass rate 1st-time pass rate 1st-time pass rate 1st-time pass rate total # of students this a 6 1st time above 50th % 1st time above 50th % 1st time above 50th % 1st time above 50th % 1st time above 50th % total passed

5

total % above 50th % total % above 50th % total % above 50th % total % above 50th % total % above 50th % total pass %

83%

Avg Score 1st time test taken

#

Avg Score 1st time # REF! Avg Score 1st time test ta# REF! Avg Score 1st time test ta# REF! Avg Score 1st time test t 1st time students

5

Avg Score all students

#

Avg Score all stude # DIV/0! Avg Score all students

# REF!

Avg Score all students

# REF!

Avg Score all students 1st time passed

4

1st-time pass rate

80%

1st time above 50th % 2 total % above 50th %

40%

Avg Score 1st time test taken

165

Avg Score all students

165

total # of students

6

total attempts

6

total passed

5

total pass %

83%

1st time students

5

1st time passed

4

1st-time pass rate

80%

1st time above 50th % 2 total % above 50th %

40%

Avg Score 1st time test

165

Avg Score all attempts

165 2004-2005 El. Ed. Totals

19

• Dr. Wall’s Work at Missouri

Western

www.aacte.org

www.aacte.org

www.aacte.org

New ew C Cha halleng nges es

• Missouri Educator Gateway Assessments (MEGA)
• Annual Performance Report (APR)

www.aacte.org

MEGA ( (Missouri Educator G Gatew eway ay A Assessment nts)

Teacher Candidate MEGA

Required for Admittance Required Prior to Student Teaching Completed During Student Teaching

MEP

X

MoGEA

X

Missouri Content Assessments

X

MoPTA

X

MEES

X

www.aacte.org

Annua nnual P Per erfor

• rmanc

nce R Repor eport ( (APR)

• Accountability report, state level, on completers
• Currently, four indicators

– Certification assessment (MoCA, from MEGA) – Content course GPA – Survey results, first year teachers (self and principal responses)

• Tiered-reporting structure, 1-4
• Results shared with media
• Tiers 3 and 4:

– Considerations for remediation – Potential closure

www.aacte.org

Our ur Asses ssessmen ent G Goal

• als

Regular and Systematic Use of Data

Courses Programs Clinical Experiences

Data Sharing Coursework revised in Fall 2012 to align with MoSPE Analyzed during ASET meeting, also sent to coordinators Methods courses, cooperating teachers, ST data Analysis Standards and assessments reviewed during program meetings Analysis during or following ASET meeting set by calendar Analyzed in ASET, shared with Field Experience Director and coordinators Results Curriculum and campus assessments modified (TWS/MoPTA) Discussed if necessary, and changes made (Praxis, Advanced, Dispositions, TK20) Director makes staff decisions, coordinators review courses (BTAP %)

www.aacte.org

Steps eps Taken t aken to

• Impact

pact S Shor hortcom

• ming

ngs

• Tk20, purchased in Fall 2011, piloted in Spring 2012, fully

implemented in Fall 2012

– Student, program and unit data is analyzed….

• Assessment System Evaluation Team (ASET)

– April 2012, 1st meeting – Calendar developed to analyze assessment data and review and modify assessment processes

www.aacte.org

Asses essmen ent S Systems and and Ev Evalu luatio ion Team Team ( (ASET) Rev eview ew C Cycle e Cal alenda endar for

• r A

Asses essment D Dat ata

Fall Spring 1st

Meeting

Annual Report, MoPTA, MEES & Other Student Teaching data Advanced Program Artifacts & Field Experience Data 2nd

Meeting

Diversity Hours, Diverse Experiences & Dispositions (MO Educator Profile) MoGEA & GPA & build up use of grades into teacher ed, w/ new competencies and phases 3rd

Meeting

ACT, student demographics & Retention (& MCA, major and minor areas and testing areas) MO Content Assessment (major and minor areas and testing areas) & Faculty Periodic Involvement

www.aacte.org

Where We Are: Strengths:

• 2006 Christa McAuliffe Award Recipient
• NCATE Standard 2 (Assessment) Target Level, 2014

www.aacte.org

2006 Christa McAuliffe Award Recipient

The Christa McAuliffe Award honors exemplary teacher education programs at public colleges and universities that that can document the success of their graduates in improving P-12 pupil learning outcomes.

www.aacte.org

Data S System I Integration a n and Stakehol

• lder Engagem

emen ent

• Chal

hallenge enges i in n Qual uality Assur uranc nce

– Integrating data from campus, program, state, testing systems into one quality control system

• Still, many different sources besides Tk20

– Engaging internal and external stakeholders in review, analysis and use of data for improvement

• Clinical partnerships
• ASET team

www.aacte.org

NWMSU: U: Our A r AACTE TE V Visi isit

www.aacte.org

Topics Explored

• Preprogram RTI System
• Program Redesign and Continuous Improvement
• Data Use for Continuous Improvement at Northwest: 2002-2016
• Teacher Education Admittance Committee Meeting
• Field Experiences and Data Usage
• Assessment System Evaluation Team (ASET) Meeting
• Clinical Partnerships
• Ploghoft Diversity Lecture

www.aacte.org

Example of Topics Explored

• Preprogram RTI System

• Mrs. Jill Baker

Instructor, Department of Professional Education Departmental Academic Advisor bakerj@nwmissouri.edu

• Mr. Kirk Skoglund

Talent Develop Center kskog@nwmissouri.edu

Interventions and Support for Pre-Admission Teacher Candidates at Northwest Missouri State University

History and Rationale

No Child Left Behind Act- NCLB (2001)- Federal legislation * Highly qualified teachers provision * Increased accountability to state and federal agencies * Increased high-stakes testing Individuals with Disabilities Education Act- IDEA (2004)- Federal legislation * RTI became acceptable to include in determination criteria for special education placement * Increased monies spent for RTI in schools * RTI shifts to a ‘for everyone’ model for academics rather than solely special education placement/behavior model Response to Intervention- RTI (1970s) * Roots in special education/behavior * Became common in regular classrooms in the early 2000’s as a model for identifying students at risk for academic difficulty and failure * ‘Intervention time’ became common in schools as a means to ‘catch’ students at risk for failure so that they could perform well on high-stakes testing

Foundations of Response to Intervention- The basis of our work

From: Center on Response to Intervention

Increasing Level of Support

Data Used To Determine Placement On The Tiers

Screening

• ACT Score
• High School GPA
• College Student Inventory (CSI) Data
• Status as an ASSIST student (conditional admission)
• Previous semester’s GPA (2nd semester freshmen and older)

Progress Monitoring

• 4 week grade checks
• Mid-term grade checks
• 12 week grade checks
• Semester GPA
• Northwest 360 qualitative data
• Qualitative data from other Northwest entities

Multi-level Prevention System

• University Seminar
• CSI and Mid-Year Student Assessment (MYSA)
• Freshman connections activities
• Behavior Intervention Team (BIT)
• Northwest 360 notes
• Academic Success Coaching
• Talent Development Center (TDC)
• Teacher Education Admissions Committee
• Teacher Education Guidance Committee
• Residential Life

Tracking and moving students within the tiers Data Point #1- Confidential Interventions Document

Data Point #2- Northwest Success 360 (Starfish)- Comprehensive data from all aspects of the student’s University experience.

Data Point #3 (when needed)- Northwest Behavior Intervention Team (BIT): Provides comprehensive support for faculty with concerns about student behavior, emotional wellbeing, and attendance.

Data Point #3 (when needed)- Northwest Behavior Intervention Team (BIT): Provides comprehensive support for faculty with concerns about student behavior, emotional wellbeing, and attendance.

What the data has told us so far….

• Comprehensive progress monitoring of this population of

students is critical given the high-stakes nature of the program

• Placing students in intervention tiers allows the academic

advisor to focus on key students who need critical support

• The intervention tiers and 360 allow the academic advisor

to address problems early before they become career- ending issues

• Data-driven decisions regarding course loads, etc. can be

more easily made when continual progress monitoring

• ccurs

www.aacte.org

Impact f t for Ho Host st Instituti tion

• Some weaknesses:

– Our faculty are extremely engaged, but need more outside engagement – Still many silos – Data bottlenecks

• Possible solutions

– Cross Over opportunities – ASET adjustments and focus

• Extraordinary feedback from visiting team- Haikus

www.aacte.org

Extraordinar ary f feedb dback f from visiting ng team- Ha Haiku kus

• Rating:
• Give feedback to all. Celebrate wins! Set goals.
• Hold onto the best.
• Design:
• Dare we build without stopping to honor constraints?
• Our dreams require it.
• Data
• Data sands shifting. Sandstorm of new assessments.
• Systems sift the gems
• Who should get to be a teacher in Missouri?
• All we can support

www.aacte.org

Reflection n from

• m t

the V Visitor

• rs