Closing the Loop Closing the Loop Closing the Loop Closing the - - PDF document

10/19/2010 1

Closing the Loop Closing the Loop Closing the Loop Closing the Loop Closing the Loop Closing the Loop Closing the Loop Closing the Loop (Continuous Quality Improv (Continuous Quality Improvement ement (Continuous Quality Improv (Continuous Quality Improvement – ement – – C – CQI) CQI) CQI)

Prof Dr Shahrin Mohammad

23h Oct 23h Oct 2010 2010

Head Computer Lab 1988 – 1990 Head of Dept (Structures and Materials) 1997 – 1998 IT Manager 1998 ‐ 2001

About Myself

IT Manager 1998 2001 Deputy Dean (Academic) 2001‐2005 Quality Mgt Rep ISO 9001:2000 2001‐2005 Academic Quality Director of UTM 2005‐2007 Deputy Dean (Academic) 2008‐2009 Dean Faculty of Civil Eng 2009‐ present Certified ISO 9001:2000 Lead Auditor, Facilitator/Trainer ISO 9001:2000, Internal Auditor Engineering Accreditation Council Panel of Trainers Ministry of Higher Education ‐ facilitator on MQF Credit System Facilitator/Trainer on Outcome based Approach Facilitator/Trainer on Quality Assurance for Academic Programme Facilitator/Trainer on Academic Auditing Secretary Academic Performance Audit Panel MQA Associate Director for Engineering Accreditation Dept

10/19/2010 2

What are the expected outcomes/results of this workshop? Begins with the end in mind p p 1. 2. 3. 4. 5. 6. Learning Outcomes At the end of the workshop you are able to;

■

Explain what and why CQI

■

Describe some best practices in implementing CQI

■

Identify current status of CQI at UTeM

■

Initiate the strategies and action plan to closen the

■

Initiate the strategies and action plan to closen the gaps to improve the current status

10/19/2010 3

Introduction Introduction Introduction Introduction

2009 : MALAYSIA AS A MEMBER OF THE WASHINGTON ACCORD

An endorsement that the engineering education system has demonstrated a strong long-term commitment strong, long-term commitment to quality assurance in producing engineers ready for industry practice in the international scene.

10/19/2010 4

Students

ACCREDITATION CRITERIA

CQI - Continuous Quality Improvement PEO - Programme Educational Objectives PO P O t

PEO & PO Students Staff Curriculum

PO - Programme Outcomes QMS - Quality Mgmt System

PO QMS Facilities

STAKEHOLDER’S NE AND INSTITUTION MISSION LEADERSHIP, GOVERNANCE and QMS(C5)

EAC-BEM ACCREDITATION PRACTICES IN A NUTSHELL

PEO and PO DEVELOPMENT (S3 S4) PROGRAMME REVIEW FOR ST EEDS NAL FE S FACILITIES (C4) DEVELOPMENT (S3,S4) ACADEMIC STAFF (C3) CURRICULUM DESIGN (C1)

TEACHING AND LEARNING ACTIVITIES (C2,C3)

STUDENTS (C2) CQI(S4,C4,C5)

PO ASSESSMENT

(S3) TAKEHOLDER’S SATISFAC

OUTPUT (GRADUATES) INPUT

8

EEDBACKS FROM STAKEHOLDER SUPPORTING UNIT AND STAFF(C2,C3) CTION

INPUT

10/19/2010 5

Focus

E i i d i d l l i i d

• Engineering education content and level are maintained
• Programme Continual Quality Improvement (CQI)
• Outcome-based Engineering Education (OBE)

Programme

• Systematic (QMS)

Closing the loop Closing the loop Closing the loop Closing the loop (Continuous (Continuous Continuous Quality Continuous Quality Quality Quality Improv Improvement) ement) Improv Improvement) ement) Improv Improvement) ement) Improv Improvement) ement)

10/19/2010 6

What is CQI ?

• The culture, strategies and methods necessary for continual

improvement in meeting and exceeding customers' expectations. (www.european‐quality.co.uk/index.php)

• A management approach to improving and maintaining quality that

emphasizes internally driven and relatively continuous assessments

• f potential causes of quality defects, followed by action aimed

either at avoiding decrease in quality or else correcting it an early stage. www.qaproject.org/methods/resglossary.html

What is CQI ?

The systematic pursuit of excellence and satisfaction of the needs of stakeholders, in a dynamic and competitive environment competitive environment

10/19/2010 7

Facts on CQI

• exists only when it is done systematically

i d i b h i f i ti

• is a dynamic behavior of an organization
• does NOT result from external factors
• can NOT be achieved in isolation
• exists when the continuous pursuit of excellence

motivates and guides the philosophies, planning, policies and processes of the organization

• Focus of CQI in education is NOT on the

curriculum

• Foundation of CQI is Assessment

Facts on CQI

• Quality improvement comes from within an institution

y p

• Continuous improvement requires integration of defined
• bjectives, performance metrics, & regular assessment
• Continuous improvement is cyclical. Assessment of

performance is the baseline for future assessment

• Essential to synchronize educational objectives, mission of

IHL and needs of stakeholders to achieve CQI IHL and needs of stakeholders to achieve CQI

• CQI can be proven through well documented report and

data.

10/19/2010 8

■ Success is achieved through meeting the needs of those

Facts on CQI

g g we serve. ■ Most problems are found in processes. CQI does not seek to blame, but rather to improve processes. ■ It is possible to achieve continual improvement through small, incremental changes using the scientific method. ■ Continuous improvement is most effective when it ■ Continuous improvement is most effective when it becomes a natural part of the way everyday work is done.

Programme Educational Objective

CQI Stage/level

PObj assessment methods

CQI

PO setting and TnL activities PO assessment methods CQI

CO setting and TnL activities

Data Analysis

Data Analysis

CO assessment methods Data Analysis CQI

10/19/2010 9

• I dentify program objectives (P-
• bj)
• I dentify program outcomes

Programme Outcomes CQI Cycle

• Communicate

the results

• I dentify best

practices

• I dentify new

project (PO)

• Map P-Obj with PO
• Map CO with PO
• Specify Curriculum content, TnL

approaches, Performance Criteria and level

• Develop assessment plan

PLAN DO CHECK ACTI ON

• Analyse the data
• Share results

C lt project

• Make necessary

changes to the plan

• Carry out TnL approaches at course
• Compare results

with what has been plan using CQI tools

• Review the gap
• What can be

learn and need to be monitored level

• I mplement out-of-class activities
• Assess the students learning
• External examiners visit
• Carry out stakeholders survey
• I nvolvement of external advisors
• Collect appropriate data
• Carry out Academic Auditing

UTM CQI Model

• Map CO with PO
• Specify Course content, TnL

approaches, Performance

Course Outcomes CQI Cycle

• Communicate

the results

• I dentify best

practices

• I dentify new

project Criteria and level

• Develop course assessment

plan

PLAN DO CHECK ACTI ON

• Analyse the data
• Share results

C lt project

• Make necessary

changes to the plan

• Carry out TnL approaches at course

level

• Assess the students learning
• Collect appropriate data
• Compare results

with what has been plan using CQI tools

• Review the gap
• What can be

learn and need to be monitored

UTM CQI Model

10/19/2010 10

UTM Academic review cycle

Eg : report

• Employability rate at a particular Convocation
• Graduate on Time (GoT) rate

Self‐review is reflected from the regular analyses made every semester to check on the academic performance of students and programmes. It is a normal practice for almost every Faculty to report on the following output parameters:

• Graduate‐on‐Time (GoT) rate
• Performance Status, based on KS (conditional pass) and KG (fail)
• Performance Status, based on CPA (cumulative point average) and GPA (gross point

average).

• % courses obtaining > 3.00, exceeding Faculty’s KPI
• Mean CGPA of students, against Faculty’s KPI
• Rate of “all passed”‐ % students passing all courses taken for a particular semester
• % students exceeding Faculty’s KPI
• % students repeating a particular Course

% students repeating a particular Course

• Assessment of all Course Outcome against Faculty’s KPI
• Assessment of all Programme Outcomes against Faculty’s KPI
• Generic Skill attainment against Faculty’s KPI
• Appropriateness of credit hours against student learning time (SLT)
• Assessment of Final Examination questions against the Bloom Taxonomy
• Customer complaints, against KPI or customer charter

10/19/2010 11

Online Integrated Quality Assurance Online Integrated Quality Assurance Monitoring System Monitoring System

Teaching Document Life-Cycle

10/19/2010 12

Best Practices Best Practices Best Practices Best Practices

Different Levels of Learning Outcomes Different Levels of Learning Outcomes Different Levels of Learning Outcomes Different Levels of Learning Outcomes

Educational Goals Broad/Long term Program Objectives Programme Outcomes or Programme Learning Outcomes Course Outcomes or Upon graduation Upon course completion Few years after Graduation – 4 to 5 years Course Learning outcomes or Learning Objectives Weekly/Topic Outcomes Upon weekly/topic completion

10/19/2010 13

The three main stages in the Teaching and Learning processes

L i L i

Bloom’s Taxonomy Program Outcomes I nstructor’s goals

Pl i Implementation

1

What do I want my students to b bl t d What TL activities do I adopt to achieve the intended LO? 3 domains of

Students Students Learning Learning Outcomes Outcomes

Classroom Assessment technique I nstructional technology Lectures Labs

Planning stage Implementation stage

2

be able to do as a result of my teaching? educational goals (cognitive, psychomotor and affective)

Felder & Brent, JEE, Jan., 2003

Assessment Assessment

Test Other measures Surveys

I nstruction I nstruction

Other techniques Active & Cooperative learning Problem- based learning

Assessment stage

3

Can my students do what I want them to be able to do?

Pla Planning Stage nning Stage Pla Planning Stage nning Stage Pla Planning Stage nning Stage Pla Planning Stage nning Stage

• Setting Learning Outcomes
• Documentations
• Programme Specification
• Course Outlines
• Policy and Code of Practices in Teaching and Learning
• Preparation for Curriculum Delivery
• Assessment Plan

10/19/2010 14

PO1 Knowledge in science and civil engineering PO2 U t h i kill d d t l PROGRAMME OUTCOMES - Keywords PROGRAMME OUTCOMES - Keywords PO2 Use techniques,skills and modern tools PO3 Analyse/Intepret Data, Conduct Experiment and Design System PO4 Communication Skills PO5 Critical Thinking and Problem Solving Skills PO6 Team-working Skills PO7 Information Management & Lifelong Learning Skills PO7 Information Management & Lifelong Learning Skills PO8 Entrepreneurship Skills PO9 Leadership Skills and Pro-activeness PO10 Ethics and Integrity

CUSTOMERS, STAKEHOLDERS, CONSTITUENTS

Professional Bodies

Relationships between subject LO’s, Course LO’s PO’s and compliance to the stakeholders

Governmnent Institutional requirements Industries etc

PRORGRAME EDUCATIONAL OBJECTIVES

PObj-1 ….. etc PObj-2 PObj-3 PObj-4 PObj-5 ….. etc PO-1 PO-2 PO-3 PO-4 PO-5 ….. etc

PROGRAM OUTCOMES

Programme Specs Management support and commitment

Course CO-1 ….. etc Course CO-2 Course CO-3 Course CO-4 Course CO-5

COURSE OUTCOMES

Course Outlines

10/19/2010 15

CO-PO Mapping CO-PO Mapping

10/19/2010 16

Lecture plan or course outline

10/19/2010 17

Lecture plan or course outline

Performance criteria at PO level

10/19/2010 18

Performance criteria at PO level Performance criteria at CO level ?

Performance criteria

Lecture plan or course outline

10/19/2010 19

Assessment Plan

PO1 PO2

Assessment Plan

PO3 PO4

10/19/2010 20

Assessment Plan

PO5 PO6 PO7

Curiculum Curiculum Curiculum Curiculum Deliv elivery ery Deliv Delivery ry

10/19/2010 21

Effective Classroom Strategies

• Democratic instructional approaches are superior

to indoctrinational approaches for imparting to indoctrinational approaches for imparting generic skills to students

41

Problem Based Learning - PBL

• Instructors assume the role of

facilitators and coaches rather facilitators and coaches rather than lecturers and order givers, requiring students to take much

• f the responsibility for their own

learning

42

10/19/2010 22

E-

• Learning UTM (http://elearning.utm.my)

Learning UTM (http://elearning.utm.my) E-

• Learning UTM (http://elearning.utm.my)

Learning UTM (http://elearning.utm.my) 44

10/19/2010 23

E-Learning UTM (http://elearning.utm.my)

45

OTHER APROACHES

• Projects on Professional Practices
• Seminars
• Engineering ethics
• Laboratory works

46

y

10/19/2010 24

Assessment Stage Assessment Stage Assessment Stage Assessment Stage

A Model Hierarchy

Mission of the Institution Vision of the Programme Programme Objectives

Need Assessm ent and Evaluation

48 Programme Outcomes Course Outcomes

INSPIRING CREATIVE AND INNOVATIVE MINDS

10/19/2010 25

P A A

Gatekeeping

When and How to Assess ?

Individual Improvement

PURPOSE OF ASSESSMENT

To help them to improve To give feedback to them To place/class them To grade/class them To pass/fail them

C C O G N N I T T I Y H H C C H H O M M O T T A A F F F F E C C T T I V

Individual

? Who to Assess? Programme Improvement Programme Evaluation

To continue/stop the programme To acreditate programme To improve the programme To monitor the progress

I V E T T O R R E Why Assess Learning/Teaching (Formative) y Accountability (Summative)

Group

W

• I dentify program objectives (P-
• bj)
• I dentify program outcomes

Programme Outcomes CQI Cycle

• Communicate

the results

• I dentify best

practices

• I dentify new

project (PO)

• Map P-Obj with PO
• Map CO with PO
• Specify Curriculum content, TnL

approaches, Performance Criteria and level

• Develop assessment plan

PLAN DO CHECK ACTI ON

• Analyse the data
• Share results

C lt project

• Make necessary

changes to the plan

• Carry out TnL approaches at course
• Compare results

with what has been plan using CQI tools

• Review the gap
• What can be

learn and need to be monitored level

• I mplement out-of-class activities
• Assess the students learning
• External examiners visit
• Carry out stakeholders survey
• I nvolvement of external advisors
• Collect appropriate data
• Carry out Academic Auditing

UTM CQI Model

10/19/2010 26

Various methods/ tools of Assessment and Evaluation Evaluation by Lecturers for each course Evaluation by Students and the Faculty

• n overall competencies

Technical Competencies Generic Skills Technical Competencies Generic Skills Competencies (PO1, PO2,PO3) (PO4-PO10) Competencies (PO1, PO2,PO3) (PO4-PO10) Develop assessment based

• n Blooms

Taxonomy Develop Performance criteria for each skill Entry Self Assessment Students Portfolio Surveys/Feedbacks from stakeholders Examination Results External examiners Concentrate on i t t Group subjects that dd d th External examiners External advisor feedbacks/report Students self-assessment Academic Advisor Evaluation quizzes, tests, projects, design, examinations etc addressed the generic skills Forms or records used in the assessment

Assessment- data analysed

PO1 PO2

10/19/2010 27

Assessment- data analysed

PO3 PO4

Assessment- data analysed

PO5 PO6 PO7

10/19/2010 28

Closing the loop Closing the loop Closing the loop Closing the loop

Programme Educational Objective

CQI Stage/level

PObj assessment methods

CQI

PO setting and TnL activities PO assessment methods CQI

CO setting and TnL activities

Data Analysis

Data Analysis

CO assessment methods Data Analysis CQI

10/19/2010 29

• Map CO with PO
• Specify Course content, TnL

approaches, Performance

Course Outcomes CQI Cycle

• Communicate

the results

• I dentify best

practices

• I dentify new

project Criteria and level

• Develop course assessment

plan

PLAN DO CHECK ACTI ON

• Analyse the data
• Share results

C lt project

• Make necessary

changes to the plan

• Carry out TnL approaches at course

level

• Assess the students learning
• Collect appropriate data
• Compare results

with what has been plan using CQI tools

• Review the gap
• What can be

learn and need to be monitored

UTM CQI Model

• I dentify program objectives (P-
• bj)
• I dentify program outcomes

Programme Outcomes CQI Cycle

• Communicate

the results

• I dentify best

practices

• I dentify new

project (PO)

• Map P-Obj with PO
• Map CO with PO
• Specify Curriculum content, TnL

approaches, Performance Criteria and level

• Develop assessment plan

PLAN DO CHECK ACTI ON

• Analyse the data
• Share results

C lt project

• Make necessary

changes to the plan

• Carry out TnL approaches at course
• Compare results

with what has been plan using CQI tools

• Review the gap
• What can be

learn and need to be monitored level

• I mplement out-of-class activities
• Assess the students learning
• External examiners visit
• Carry out stakeholders survey
• I nvolvement of external advisors
• Collect appropriate data
• Carry out Academic Auditing

UTM CQI Model

10/19/2010 30

Closing the loop at CO level

Example

Closing the loop at PO level

PO1 PO2

10/19/2010 31

Closing the loop at PO level

PO3 PO4

Closing the loop at PO level

PO5 PO6 PO7

10/19/2010 32

Thank you k you Thank you k you

10/19/2010 2

Lecture plan or course outline p

10/19/2010 3

Performance criteria at PO level Performance criteria at PO level

10/19/2010 4

Performance criteria at PO level Performance criteria at PO level

10/19/2010 5

Performance criteria at CO level ? Performance criteria at CO level ?

Performance criteria

Lecture plan or course outline

10/19/2010 6

Assessment Plan Assessment Plan

PO1 PO2

10/19/2010 7

Assessment Plan Assessment Plan

PO3 PO4

10/19/2010 8

Assessment Plan Assessment Plan

PO5 PO6 PO7

10/19/2010 9

Assessment- data analysed Assessment data analysed

PO1 PO2

10/19/2010 10

Assessment- data analysed Assessment data analysed

PO3 PO4

10/19/2010 11

Assessment- data analysed Assessment data analysed

PO5 PO6 PO7

10/19/2010 12

Closing the loop at PO level Closing the loop at PO level

PO1 PO2

10/19/2010 13

Closing the loop at PO level Closing the loop at PO level

PO3 PO4

10/19/2010 14

Closing the loop at PO level Closing the loop at PO level

PO5 PO6 PO7