Evaluation of software tools supporting outcomes- based continuous - PowerPoint PPT Presentation

Evaluation of software tools supporting outcomes- based continuous program improvement processes: Part 3 Jake Kaupp & Brian Frank

Episode iii I ’ v e b e e n a t t h i s a w h i l e

The promise “Academic analytics combines select institutional data , statistical analysis, and predictive modelling to create intelligence upon which students, instructors, or administrators can change academic behaviour ”

The pitfall

The problem “ Why so much collection — but so little utilization — of data? Most institutions have routinized data collection , but they have little experience in reviewing and making sense of data .” - C. F. Blaich & Wise, 2011

Inform the community and raise awareness of available tools, their strengths and limitations . Encourage the use of e ff ective practise in conjunction with educational technology

5 ! Assessment Platform Designed according to effective practise Portfolio encourages authentic assessment Accommodates program & course assessment Manages process planning & curriculum improvement Templated accreditation reports

Analytics System Cloud-based platform Focus on student success & degree pathways Engagement & persistence are the main predicted elements Draws factors from diverse sources (LMS, SIS, etc.) 3 ! Variable granularity for views & reporting

4 ! Hybrid, closest to an analytics system Focuses on UX & flexibility Newcomer to education field Emphasis on controlled process with user & role-defined workflows Excel powered graphics Very customizable

3 ! Curriculum mapping system Program & competency focused Multiple stakeholder views Encourages constructive alignment Flexible metadata mapping capabilities planned Reporting in development

3 ! R: Statistical Computing Language Pipeline for data analysis [1] RStudio: IDE for R Visualize One of the fastest growing languages for data analysis, analytics & visualization Tidy Transform Extensible, modular, framework based Model Excellent for customized, need- based and ad-hoc reporting [1] Wickham, H. (2015). Big data pipelines. Presented at the Workshop on Visualization for Big Data Strategies and Principles, Toronto.

Templated Reporting Performance Histograms Graduate Attribute Course Report: MECH 216 2014 − 2015 Jake Kaupp 100 May 22nd, 2015 75 Course Mapping Tables 50 Data IN Acquisition 25 Course Indicator Short Assessment Assessor Date Instructor Number of Students Description Assessed Comments Assessed 0 MECH APSC-2- Data lab TA NA NA 163 100 216 IN-2 Acquisition reports MECH APSC-2- Uncertainty lab TA NA NA 163 75 216 IN-5 reports MECH APSC-2- Draw lab TA NA NA 163 Draw 50 IN Conclusions 216 IN-6 Conclusions reports MECH APSC-2- Write Clearly lab TA NA NA 163 25 216 CO-3 reports 0 MECH APSC-2- Technical lab TA NA NA 163 100 216 CO-6 Graphics reports 75 Count 50 IN Uncertainty 25 0 100 75 Technical 50 CO Graphics 25 0 100 75 50 Write CO Clearly 25 0 Not Marginal Meets High Mastery Demonstrated Expectations Quality Performance

Dashboard Style Information Program Overview Historical 2013 − 2014 2012 − 2013 2013 − 2014 500 Mastery 400 300 Technical CO APSC − 1 − CO − 3 High writing 200 Quality First Year 100 0 Meets 500 Expectations 400 300 Marginal DE APSC − 1 − DE − 5 Convergent 200 100 Not 0 Demonstrated 500 400 Mastery 300 LL APSC − 1 − LL − 6 Self − regulation 200 High 100 Second Year Count Quality 0 500 Meets 400 Expectations 300 Modelling PA APSC − 1 − PA − 4 process 200 Marginal 100 0 Not 500 Demonstrated 400 300 Model PA APSC − 1 − PA − 7 evaluation Mastery 200 100 High 0 Quality Third Year 500 400 Meets Evidence 300 PA APSC − 1 − PA − 8 based Expectations 200 conclusions 100 Marginal 0 Not Marginal Meets High Mastery Not Marginal Meets High Mastery Not Demonstrated Expectations Quality Demonstrated Expectations Quality Demonstrated Performance Mastery High Fourth Year Quality Meets Expectations Marginal Not Demonstrated PA DE CO LL Attribute

Episode iv Maybe Blackboard will speak with me this time