Program Program Learning Learning Outcomes: Outcomes: Closing - - PowerPoint PPT Presentation

Program Program Learning Learning Outcomes: Outcomes: Closing Closing the the Lo Loop

• p

Amal BouZeineddine bouzein@aub.edu.lb Associate Director Center for Teaching and Learning (CTL) American University of Beirut June 19, 2019

By the end of this session, we will be able to:

• 1. Articulate program learning outcomes
• 2. Assess program learning outcomes
• 3. Develop an action and improvement plans for

assessing program learning outcomes Session Learning Outcomes

Int Introd roduct uction ion: A Case Case for for Le Learning arning Outcomes Outcomes

Year

Students enrolled in higher education worldwide

1970

32.5 million

2000

100 million

2010

178 million 4.3% average annual growth in tertiary enrollment compared to the 1.6% average annual growth in the world population

• ver the same period (UNDP, 2012)

Number of higher education students is forecast to further

expand to reach 263 million by 2025 (British Council and

IDP Australia, cited in Davis, 2003 and Daniel, 2009).

A Case Case fo for Learni Learning ng Ou Outcom tcomes: es: Today’s HEIs HEIs

• Today HEIs are more diversified and are closer to a

patchwork model attended by larger segments of the population.

• Higher education today is characterized by

➢ massive expansion; ➢ more diverse profiles of HEIs, programs and their

students;

➢ broader adoption and more integrated use of

communications and educational technologies;

➢ greater internationalization;

Today’s HEI HEIs

• Internationalization of higher education features in the past 25 years, especially in

the European context, has been accelerating rapidly. Internationalization can be

defined as “the process of integrating an international, intercultural or global dimension into the purpose, functions or delivery of tertiary education” (Knight, 2003). Internationalization is manifested in:

➢ Students’ mobility ➢ Scholars’ mobility ➢ Curricula: Programs’ content and delivery ➢ Institution: Cross-border mobility of HEIs and/or their programs ➢ labor market → global economic integration

employers in OECD countries increasingly need to look abroad for talent as their own graduates are insufficient to replace those going into retirement.

• HEIs Ranking
• Student Centeredness
• Evidence-based teaching and learning

Student Student Mobili Mobility ty

• International student mobility has

increased over the past three decades, from 0.8 .8 mi million llion students worldwide in 1975 75 to 4. 4.1 million illion in 20 2010 10 (Figure 1.2) driven by large increases in student mobility from China, India and European countries in particular (OECD, 2012).

• Growth is projected to continue in the

future to reach approximately 5.8 mi million llion aro around und 202 2020 (Böhm et al., 2004) and 8 mil illion ion by by 2025 025 (Altbach and Bassett, 2004).

Mobili Mobility ty of

• f Academic

Academic Staff taff

• Data are scarce on the international mobility of academic

staff.

• This is further complicated by the multiple forms of mobility,

from short-term moves of a few days/weeks to longer movements of one year or more.

• Available evidence suggests that the main internationalization
• f faculty consists of short-term leave, exchange visits and

research collaborations (Enders and de Weert, 2004).

Inte Intern rnatio ationa nalizat lizatio ion of

• f Curricul

Curricula

• The internationalization of curricula gained momentum in

the past decade with greater emphasis on the teaching of foreign languages and broader international perspectives in the content of programs and curricula (Van der Wende, 2001; Harman, 2006).

• This phenomenon has been most evident in Europe with the

Bologna Process aimed at establishing a European Higher Education Area (EHEA) and enhancing the comparability and compatibility of higher education structures and degrees in Europe by 2010 (Bologna Secretariat, 1999).

Inst Instituti itution Mob

• bili

ility

• The past 15 years have also seen the emergence and growing development
• f off
• ff-sho

shore re de deliv iver ery of

• f edu

education cation by HEIs. Australia and the United Kingdom pioneered this movement in the 1990s, but the United States has also become a major force in this area and a number of other countries – especially European countries – have joined this trend and, since 2000, set up campuses abroad and other trans-national arrangements (OECD, 2008, OECD, 2004; McBurnie and Ziguras, 2001).

• the majority of trans-national operations take the form of joint programs

with foreign HEIs through franchise arrangements with a partner institution in the international students’ home country.

• More recently, virtual HEIs which operate exclusively online have
• emerged. Recent trends suggest that although program and institution

mobility is not yet that important, these forms of internationalization are expected to outpace the delivery to international students onshore.

Qu Qual ality ity Assu Assura rance ce - Qu Quali lity ty Ed Education ucation???? ?????

Ran Ranki kings ngs

• The spread of global rankings illustrates the need for systems,

HEIs and individuals, to gain access to internationally comparable data and information on university achievement. These developments have driven a “rankings movement” that has considerably sharpened focus on data-driven cross- institutional comparison.

Student Student- centered centered vs.

• vs. Te

Teacher acher - Cen Centere tered

• The paradigm that has governed our colleges has been this: A college

is an institution that exists to provide instruction. Subtly but profoundly, we are shifting to a new paradigm: A college is an institution that exists to produce learning. It takes the means or

method-called "instruction" or "teaching" - and makes it the college's end or purpose. *…+ We

We now

• w see

see th that at ou

• ur mi

mission sion is is not not inst instruct ructio ion but but rather rather th that of

• f producing

roducing le learn arning ing with ith every every stu student dent by by what whatever ver me mean ans work

• rk best

best (Barr and Tagg, 1995).

• Students’ learning outcomes are a key factor of institutional

performance, and hence of aggregate system performance.

Evidence Evidence-based based teaching teaching

• Evidence-based teaching (EBT) is a concept or approach teachers

and researchers in education adopt to improve or modify teaching and learning techniques. This concept could be from either a qualitative research data on students or the personal experience teachers acquired as a result of continuous study of profiles, attitudes, characteristics and academic potentials of students within the educational sector.

• Evidence is obtained through various forms of assessment – which

may include teacher observation, tests, peer assessment and practical performance – and constitutes the information and data that is used to gauge the educational attainment and progress of individuals; groups; and cohorts; and increasingly, the effectiveness of programs and performance of educational systems

Why Why PLO PLO’s and nd Assessment Assessment of

• f PL

PLO’s? s?

PLO Assessment Roadmap

Goals, Objectives, Outcomes

http://www.assessment.uconn.edu/primer/goals1.html

Vision Vision an and Miss Mission ion

VIS ISIO ION

(The

The Dream Dream) MISSION (The

The Dest Destination nation)

Vision ion: This utopian dream is generally described by one or more phrases or vision statements, which are brief proclamations that convey the community's dreams for the future. Without a vision, your program lacks direction. If you don't have a common, agreed-upon destination, then everyone is left to his or her own devices to imagine one—a scenario that results in unharnessed and unfocused efforts, with everyone believing that what he or she is doing is right.

• We aspire to inspire a passion for learning.
• Our programs nurture critical thinkers and self directed individuals

Missio Mission: The program mission is a broad statement of what the program is, what it does, and for whom it does it. It should provide a clear description of the purpose

• f the program and the learning environment. The mission statement should

reflect how the program contributes to the education and careers of students graduating from the program. (Selim and Pet-Armacost, 2004, p.17)

Ex Example mples of

• f Missi

Mission

• n Stat

Stateme ment nts

BA BA in in Edu ducat cation/ ion/Elementary lementary Mis Mission ion St Stat ateme ement nt The Bachelor of Arts in Education/Elementary Program aims at developing students’ knowledge base for teaching and competence in professional practice as well as promoting a commitment to personal professional development and active participation in the professional community. Through fulfillment of coursework, field-based experiences, and professional community service activities students are prepared to enter the field of teaching and/or graduate studies in education as reflective practitioners, literate in information and communication technology, and critical thinkers committed to the human and moral values of lifelong learning, integrity, innovation, civic responsibility, and leadership. Ba Bachelor chelor of

• f Engineering

ngineering (BE) Ma Majo jor: r: Chemi Chemical cal Eng ngin inee eeri ring ng Miss Mission ion The mission of chemical engineering in SFEA is to provide a stimulating and supportive environment for quality education; to prepare graduates for career opportunities in a rapidly changing world by fostering the development

• f professionalism, leadership qualities and

ethical behavior, and to contribute to expanding the knowledge in chemical engineering and its related fields.

Program Program Lea Learn rning ing Outcomes Outcomes

Let’s Talk

➢

What are Program Learning Outcomes (PLO’s)?

➢ Why should we assess PLO’s? ➢ How do we assess PLO’s?

Developing Developing Program Program Learni Learning ng Outcomes Outcomes

PLO’s are statements of expectations about what

students should know and be able to do as a result

• f their learning experiences in a given academic

program. PLO’s are what a program considers students will be able to perform/ do as a result of program design. What are Program Learning Outcomes (PLO’s)?

BEST BEST WAY WAY is to have a combination of both → Get everyone involved; this ensures buy-in and ownership → Talk with team members about setting PLO’s and how they assess department success

TOP-DO DOWN Broad set of goals/PLO’s are identified by administration; faculty create LO’s for these PLO’s in their courses Top down allows for a holistic approach ensuring people are not pulling in different directions Promotes cohesion BOTT TTOM OM-UP UP Members create PLO’s based on already existing courses and CLO’s Bottom up builds buy- in, ownership and commitment to learning outcomes; people get to reflect

• n their learning
• utcomes and how

they relate to the

• rganizational PLO’s

PLO’s an and Bac Backwa kward rd Des Desig ign: n: Wigg Wiggin ins and McTig McTighe he (2005 2005)

r al

1 1 1 E

Ideal Ideal Student Student and nd Developing Developing Learning Learning Outcomes Outcomes (PLO PLO’s)

Characteristics Characteristics of

• f Learning

Learning Ou Outcom tcomes es

Well-written learning outcomes should be

✓ Student-centered ✓ Measurable ✓ Concise and clear

Characterist Characteristics ics of

• f (PLO’s) … 1

Student Student-centered.

• centered. This means that the outcome

should state what the student will do upon exiting from the program, not what the instructor does. Example: Example: Upon Upon gra graduation duation fr from

• m the

the program, program, the the studen student wil will be be able able to to

able able to to analyze analyze Characteristics Characteristics of

• f (PLO’s) … 2

Meas Measura urable.

• ble. This means that the outcome can be

effectively evaluated and assessed. Vague and general outcomes are hard to assess. Be specific

• n what you want the student to be able to do.

Use “ Action Action ver verbs bs” preferably one

• ne per outcome

Example: Example: Upon Upon graduation graduation from from the the program, program, studen students wil will be be

Verbs to avoid when writing learning outcomes include

These verbs are unclear and subject to different interpretations in terms of what action they are specifying. Such verbs call for covert behavior which cannot be observed or measured.

Example: Example: Upon Upon gra graduation duation fr from

• m the

the program, program, the the studen student Characteristics Characteristics of

• f (PLO

(PLO’s) s) … 3 Clear Clear and and Concise Concise learning learning out

• utcomes
• comes. Only include

the information that will be measured.

Keep in mind this information/data could be collected from one or

more courses.

wil will be be able able to to analyze analyze the the imp impact act of

• f historical

historical and and contempo contemporary rary is issues sues on

• n civil

civil or

• r construction

construction enginee engineering. ring.

❖ Spe

pecifi ific: c: precisely states what the learner will be able to do

❖ Meas

easurab urable: le: can be observed or counted during or after the

program ❖ Act

ctio ion-or

• riente

iented: d: uses an active verb that represents a behavior change or acquisition

❖ Reas

easonab

• nable:

le: is appropriate to the time and scope

❖ Time

ime-bo bound: und: can be achieved by the end of the program Effective learning outcomes are SMART: Specific, Measurable, Action-oriented, Reasonable, and Time-bound.

SMART SMART PLO’s

Measurable

Action

Time-bound Oriented

Specific

Upon graduation from the program, the student will be able to explain the impact of historical and

contemporary issues on civil or construction engineering.

SMART Example:

Reasonable

REMEMBER: Verbs to avoid when writing learning

• utcomes include:

Correct Correct PLO PLO or

• r not?

not?? Why?? Why??

• The program will present the different theories of

personality development.

• By the end of the program, the student will be able

to apply theories of personality development on regular and special cases. Correct Program - centered Student Centered

Cont’d

• The student will know what affects historical

events Vague CORRECT

• At the end of the program, students will describe

how philosophical and political ideologies, economic structures, and cultural perceptions have framed and affected historical events and issues. Concise/clear

Cont’d

• The student will show an understanding of

computer science in the design of computer-based

• Upon completion of the program, the student will

be able to apply mathematical foundations, algorithmic principles, and computer science theory in the design of computer-based systems showing the tradeoffs in design choices. Measurable systems. Non- measurable Correct

What What is is key key to to developing developing PLO’s?

Bloom (1956) defined three different domains of performances which he set in hierarchies:

➢ Cogniti

Cognitive ve dom

• mai

ain (th

(thin inkin ing, g, know knowledge): ledge): student cognitive behavior is categorized into six levels ranging from simple (knowledge) to more complex behaviors (evaluation)

➢ Affective

Affective domai

• main (feel

feelin ing, g, att attitudes itudes): ): this domain ranges from receiving going up to internalizing.

➢ Psychomotor

Psychomotor domai domain (do

(doin ing, g, skills): ls): this taxonomy ranges from the simple act of perception to the highest level of behavior,

• rganization.

The cognitive domain is the most commonly used.

Developing Program PLO’s

Bloom Bloom’s Cognitiv Cognitive Taxo Taxonomy nomy

New Taxonomy Traditional Taxonomy

45

Lev evel els of

• f Knowledge

Knowledge

• Fac

actua tual Kn Know

• wledge

ledge - The basic elements students must know to be acquainted with a discipline or solve problems.

• Concept
• nceptua

ual Know nowledge ledge – The interrelationships among the basic elements within a larger structure that enable them to function together.

• Proced

rocedur ural Know nowledge ledge - How to do something, methods of inquiry, and criteria for using skills, algorithms, techniques, and methods.

• Me

Metac tacogn

• gnitiv

itive Kn Know

• wledge

ledge – Knowledge of cognition in general, as well as awareness and knowledge of one’s own cognition.

46

SLO LO Co Components mponents

• Geo

Geograp graphy hy: Indicates the various types of maps, charts, and

graphs in interpreting data on topography. → Factual

• History

istory: Analyzes theories which investigate the

implications of the past on present. → Conceptual

• Nu

Nursing rsing: Applies the scientific method steps to report

findings on patient care → Procedural

• Ed

Education ucation: Reflects on teaching practices and the

curriculum in language arts → Metacognitive

➢Each 3-year program could have 5 to to 8 learning

• utcomes. These represent what students should

be able to do when they have completed the program. ➢Each course in that program should address 1 to to 3

• f the program learning outcomes.

➢Capstone courses cover most or all PLO’s.

Que

Le

st

ar

ionning Outcome Template Who will do the learning? When will the learning be completed? What will they learn? How will you know that they learned it? Learning Outcome:

Lea Learning rning Ou Outco tcome me Tem Template late Qu Quest estio ion Who ho will ll do do the he lea learning rning? When wi will ll th the lear learnin ing be be co complete mpleted? What at wi will ll the they lea earn rn? How will ill you you kno now tha hat they ey lea learn rned it? it? Lea Learning rning Ou Outco tcome: me:

Yes No Not

Sure

• 1. Is students’ intended performance observable?
• 2. Can students’ observable performance be assessed /measured?
• 3. Is PLO aligned with the mission and goals of your program?
• 4. Can level of knowledge, attitude and skills be identified?
• 5. Does PLO statement have one action verb?
• 6. Is focus on the learning result (i.e. product or performance) that

the student will exhibit and not the learning process or your instruction?

• 7. Can PLO be measured by more than one assessment method?

Program Program Lea Learn rning ing outcomes

• utcomes checkli

checklist st

Your description of the ideal student turns into PLO’s as shown in the examples:

Examples of PLO’s by Discipline Arabic

By the end of the program, students will: 1. Describe the cultural traditions, belief systems, and social worlds of peoples of Arabic-speaking countries 2. Analyze most factual material in nontechnical prose as well as some discussions on concrete topics related to personal interests 3. Use linguistic context and real-world knowledge to make sensible guesses about unfamiliar material 4. Write routine social correspondence, daily situations, and/or current events 5. Apply the morphology of language and of the most frequently used syntactic structures http://www.fresnostate.edu/artshum/mesp/documents/learning_outcomes_persian_modern.pdf

Business

Introduction to Business At the end of the program, students should be able to:

• 1. Describe current domestic and international business trends
• 2. Explain how proper business management benefits consumers and employees
• 3. Define the basic rules related to human resources management
• 4. Compare the different types of business ownership
• 5. Evaluate various marketing strategies
• 6. Summarize how technology can help a business manage information

http://www.tltc.ttu.edu/content/asp/assessment/pages/pg09.pdf

Chemistry

By the end of the program, students will be able to:

• 1. Solve complex problems (with organic molecules);
• 2. Apply the concepts of electro negativity, VSEPR, hybridization, and the octet rule to draw Lewis

structures and predict physical and chemical properties;

• 3. Apply the concepts of (Bronsted and Lewis) acids and bases, electrophiles and nucleophiles to

predict characteristic organic reactions;

• 4. Analyze the relationship of structure and reactivity;
• 5. Predict and design synthetic pathways to create complex structures from simple compounds;
• 6. Apply chemical (and eventually spectroscopic) data to predict structure.

http://www.gwu.edu/~assess/Coursegoalsexamples.html

CVSP

Upon completion of the program, students can 1. Synthesize information about the religious, political, moral, and cultural contexts in which major works were created. 2. Conduct art historical research, interpreting art work and integrating appropriate secondary sources. http://wasc.ucr.edu/docs/Sample%20Learning%20Outcomes%20--%20Humanities.pdf

Upon completing the program, students should be able to:

• 1. Solve problems in mathematics through differential equations, probability and statistics, calculus-

based physics, general chemistry, and one additional area of science.

• 2. Analyze and solve problems in material science and engineering mechanics.
• 3. Explain the impact of historical and contemporary issues on civil or construction engineering.
• 4. Recommend solutions to well-defined project management problems within civil or construction

engineering.

• 5. Design a system or process in more than one program-relevant civil or construction engineering

specialty field to meet desired needs, including sustainability and within other realistic constraints such as economic, environmental, social, political, ethical, health and safety, and constructability.

At the end of the program, you will:

• 1. Integrate evidenced based concepts of nursing practice across the health care continuum to ensure

quality patient centered care.

• 2. Synthesize principles of the nursing process and critical thinking to assist individuals, families

and communities to achieve positive adaptation to change in health or a peaceful death.

• 3. Synthesize principles of holistic nursing practice when providing nursing care for clients at

various stages in their life span.

• 4. Empower individuals, families and the community to develop positive health behaviors through

health promotion and teaching.

• 5. Value a commitment to caring.

http://www.ohlone.edu/instr/nursing/outcome.html

Nursing Engineering

Assessment of PLO’s is assessment + Of learning + For learning + Is learning

Assessment Assessment Cycl Cycle

What What is is assessment assessment of

• f PLO’s?

Def Definit initio ion ➢Assessment of program learning outcomes is the systematic systematic an and ong

• ngoing
• ing method of gath

gatherin ering, analyzi analyzing ng and and usin using informa information tion from measured

• utcomes in order to imp

impro rove ve studen student learning learning.

➢

PLO Assessment is using data from vari various

• us

sources sources to obtain a clear understanding of what the program graduates know know, can can do do, and val value ue as a result of their learning experiences in the program.

360 360 Degre egrees es Ev Evaluat aluatio ion

➢ Feedback from all levels of the

• rganization is collected, organized, and

presented to the employee, essentially giving a full 360[degree] view of one's performance. ➢This type of evaluation, which can be used either as a performance appraisal tool or as a developmental tool, encourages a diverse view of strengths, barriers, and areas of improvement.

Selecting Selecting Assessment Assessment Methods ethods Tri Trian angula ulate te

Direct Direct Mea Measures sures look at the actual student work (assignments or exams) that can be used to directly measure students’ knowledge and skills Exam Example les: s:

✓ Pre-test, post-test, ✓ comprehensive subject matter

exam,

✓ licensure exam, standardized tests ✓ portfolio, ✓ thesis, ✓ Embedded assignments, ✓ Exit Exam, ✓ Capstone Assignment, ✓ Performance Demonstration

(recital, art exhibit, or science project)

✓ Case Studies

Indirect Indirect Mea Measu sures res refer to the type of data that can be used to infer students’ knowledge and skills Ex Exampl mples es: ✓ Surveys (Exit, alumni, employer, graduates of program etc.) ✓ Interviews, ✓ Focus Group Studies, ✓ Instructor Course Evaluation, ✓ Students’ Self- Reports/reflections ✓ Observations ✓ Transcript Analysis

ALIGNMENT

Figure 1. Aligning learning outcomes, learning and teaching activities and the assessment Adapted from Biggs(1999) p 27

What What is is the the process process for for ef effecti fective PLO’s ass assessment? essment?

PLO LO Asse Assessment ssment Ro Roadm admap

St Step ep 1: 1: Define Define the the mission mission of

• f th

the pr progra

• gram.

m.

Develop the program mission.

The mission provides a clear description of the purpose of the program and the learning environment. The mission should be aligned with the

Department, College, and University’s mission.

St Step ep 2: 2: De Defin ine the the goal goals of

• f the

he pr program.

• gram.

Articulate the goals of a program or unit. Program goals must align with those of the school or college, and ultimately with the goals of the institution. St Step ep 3: 3: De Define fine the the in intended tended student student le learn arnin ing outcomes

• utcomes
• f
• f the

he pr program.

• gram.

The learning

• utcomes
• f

a program describe the intended educational

• utcomes

in terms

• f specific

abilities, knowledge, values and attitudes that you want students in your program to possess.

Step Step 4: 4: Organize Organize for

• r assess

assessmen ent. t. Before assessment can begin, identify the co

commit mittees tees and and struc structures

• tures. One or

more persons may lead the program assessment process, but it is crucial for all

faculty to assume the responsibility for designing, implementing and carrying

• ut the assessment process including reviewing the results and implementing

improvements. Define the sc

scop

• pe of
• f the assess

assessmen ment plan and its timel

• eline. Determine what the

assessment will include, when and how. Identify when the assessment of eac each PLO will occur and possibly how often. Step Step 5: 5: Se Select lect assess assessment ment method methods. s. Typically, sev

evera eral met ethods hods are used to measure the outcomes of the program; this

requires at at lea least st two wo measu measures res for each learning outcome. Normally, stud

tudents ents

who are majoring in a given field of specialization are the ones to be included in their program learning outcomes assessment. Step Step 6: 6: Ident Identify ify assess assessmen ment tar targe gets. Determine what st

stand andards ards are re exp expecte ected from rom stu stude dent nts in your program. For some

intended outcomes, you may want 100% of graduates to achieve them, but realize that this expectation is unrealistic for other learning outcomes. You may want to determine what pro propo porti rtion

• n of
• f you
• ur students

students achieves a specific

• level. If you have pr

previ evious

• usly measu

easured red an an outco come, it is helpful to use this as

the baseline for setting your target for next year.

CTL / AU AUB PLO Plan lan

Ce Cent nter er for for Teac Teaching ing and and Learning Learning (C (CTL) TL) Learning Learning Ou Outco tcomes Pla Plan (LO (LOP): P): Imple lement ntat atio ion Propo

• posa

sal Fr From

• m Ama

Amal, l, for for di discussio cussion in in CT CTL meeting ting: Thur ursday, ay, Oc October tober 26, 26, 2006 2006 In order to ensure effective LOP implementation, seven areas should be considered:

• 1. LOP Coordinator
• 2. Forming a LOP Coordinating Committee
• 3. Using

LOP PLO Assessment Time Table

• 4. Using LOP/CTL handbooks
• 5. Recommending PLO Assessment Committee in Each Department
• 6. Training
• f

Faculty → Human Resources (TOT) + Equipment

• 7. Determining Compensation (incentives)Procedures for Participating

Faculty Members

Do Do This phase involves measuring the outcomes of your program. Ste tep 7: 7: Col Collect lect th the data. data. Determine

➢ how

• w the data will be collected → put steps 5 & 6 to action

➢ who

ho will collect the data →assign a committee, work with faculty

➢ where

re and how the data will be archived → In the Department,

Dean’s Office? CTL? Soft coy? Hard Copy? Use qualitative and quantitative data collection procedures.

Ch Check eck St Step ep 8: 8: Analyze nalyze the the resul results. ts. Summarize the results in a meaningful way so that the facul faculty ty can can review review them and determi determine what what ac action tions are needed to improve the program. St Step ep 9: 9: Provide Provide fe feedback. edback. Document results and information gained in a scientific report, dis distribu tributed ted to to th the fac faculty ulty and other appropriate parties to obtain their ideas on how to improve the program.

Ste tep 10 10: Impl Implement ement ch changes anges.

Act Act

Use the results of the assessment to id identify entify changes changes to to impro improve the program. These changes could be applied to the content of the curriculum, staffing, facilities, among others. At this point in the continuous improvement cycle, the planned ch changes anges should should be be implemente

• implemented. In some cases, the changes are easy to implement, while in other

instances the proposed changes will have to be implemented over a period of time

• r through a series of steps.

Ste tep 11 11: Deve Develop lop pla lan to to mon monito tor the change changes an and co compar mpare the res results. ts. Monito Monitor imp imple lemented mented change changes to determine whether or not the changes had the desired effect. One way of achieving this is to use the same assessment plan as used in the previous cycle and compare the actual results to the intended results. Any dis iscr crep epanc ncie ies sh should uld be be ca caref refull lly studi studied to determine the underlying cause. In

• ther situations, when the out
• utco

comes mes have ave bee een me met, the action might be to conti ntinue moni monito toring ing the outcome to ensure quality. Ste tep 12: 12: Review Review info informati mation. n. Review Review all of the information obtained from the asse ssessment ssment proces rocess and determine how how th this is will ll aff affect ct yo your ur nex next ass assess essment ment plan. This provides the starting point for the next iteration of the plan-do-check-act cycle to continuous improvement of the academic program.

Align ign course urses with with pro rogr gram lea learni rning ng outco

• utcomes

mes

Sources Sources of

• f Dat

ata Coll Collection ection

Wh Why avoid avoid using using co course urse final final grades rades in in PLO’s as asse sessm ssmen ent? t?

The use of grades in program assessment is highly discouraged for several reasons:

• 1. Grades indirectly represent the quality of student work as they often

reflect more more than just the students’ intellectual skills and abilities. For example, does a course have atte attenda ndanc nce re requi quiremen rements ts or

• r strict

strict du due date date poli policie ies?

• 2. Grades represent the perspective of a single

single mem ember er of

• f th

the facu facult lty based on work in a single course. Part of the goal of program level assessment is to develop agreement among faculty about what proficiency in a given skill or knowledge area looks like. Program learning outcomes and the levels of proficiency graduates will exhibit should should repr represent sent shar hared goals

• als of
• f th

the fa facult culty working toward an exceptional program that produces successful graduates.

Putting Putting PLO PLO Ass Assessment essment Plan Plan to to Action Action

PLO PLO Ass Assessment: essment: Action Action plan plan

• An action plan refers to: a) who

ho will do what at and by whe hen b) What cha change nge needs to happen, and c) the specific ac actio tion st steps eps necessary to bring about changes.

• Action plan steps are :

✓ Act ctio ion st step(s): What will happen → who will collect data & from who? ✓ Pers rson( n(s) s) re respo pons nsible le: Who will do what ✓ Dat ate to to be be co comp mplet eted ed: Timing of each action step ✓ Res Resou

• urces

rces re requi uire red: Resources and support (both what is needed and what's available ) ✓ Ba Barriers rriers or

• r resista

resistance nce, and a plan to overcome them! ✓ Collab abora rator

• rs: Who else should know about this action

Here are sam samples les of action steps, graphed out so you can easily follow the flow:

Sa Sample 1

The annual assessment report should contain the following components 1.

• 1. Pr

Prog

• gram

ram Learn arning Outco Outcomes mes identified for assessment

• 2. Summary of assessment

assessment met ethods/mea easure res to include brief

explanation of methodology, persons responsible for data collection and analysis, and a summary of assessment results.

• 3. A summary of change

changes or

• r im

impro rove vemen ents proposed based on assessment results.

• 4. A brief description of the pl

plan an and and time imeline line for for imp imple leme menting nting the

proposed change(s) and any resources needed to implement the proposed change (s), using existing resources already in place, reallocating existing resources from other areas, or allocation of new funds.

• 5. A description of the process used to dissem

disseminate inate assessment assessment re resu sult lts

to key stakeholders.

Actio ion Pla lan

Sample Sample 2

Sample 4

Samp Sample le 5 Cont’d

Sample Sample 5 Cont’d

AUB AUB PL PLO Te Templates mplates

Annual Program Learning Outcomes Assessment Report

Program Information Faculty Department

Academic Program

Degree

Academic Year Program Learning Outcomes (PLOs) Please list below all of the learning outcomes for this program:

1. 2. 3. 4. 5. 6. Etc. Assessment Committee

(Please indicate the contact person for the PLOs assessment process.)

Full Name E-Mail Address Extension Date Submitted:

AUB Te Template mplate Cont’d

Abstract/ Summary

(Describe briefly the PLOs assessment approach followed in your department, the assessment tools and methods used, and the assessment results. Explain required changes based on the findings. This section should not exceed 1 or 2 pages. )

Outcomes Assessed, Means of Assessment, and Improvement Plans

Fall Semester PLOs Assessed

[Please provide the PLOs assessed for this semester.]

Data Source

[Present source of data: courses, surveys, sampling procedures used (if any), etc. Please report on each PLO separately.]

Assessment Methods (Direct and/or Indirect Assessment)

[Describe in detail the assessment methods used (e.g., capstone project, course-embedded assessment, standardized instrument, focus groups, questionnaires, etc.). Please

report on each PLO separately.]

Means of Scoring

[Describe the assessment tools used for scoring. For example, did you use a rubric or answer key, or was it scored by OIRA or a testing company? Please report on each

PLO separately. ]

Assessment Results

[Please present the PLO assessment results in a form of table (if applicable) or otherwise, and report on each PLO separately.]

AUB Cont’d

Spring Semester PLOs Assessed

[Please provide the PLOs assessed for this semester.]

Data Source

[Present source of data: courses, surveys, sampling procedures used (if any), etc. Please report on each PLO separately.]

Assessment Methods (Direct and/or Indirect Assessment) [Describe in detail the assessment methods used (e.g., capstone project, course-embedded assessment, standardized instrument, focus groups, questionnaires, etc.). Please

report on each PLO separately.]

Means of Scoring

[Describe the assessment tools used for scoring. For example, did you use a rubric or answer key, or was it scored by OIRA or a testing company? Please report on each PLO separately. ]

Assessment Results

[Please present the PLO assessment results in a form of table (if applicable) or otherwise, and report on each PLO separately.]

AUB Cont’d

Findings Use and Dissemination

Fall Semester Use of Findings

[Please report on emerging patterns obtained from the results and accordingly the action that will be taken, for example: Changes to the Assessment Plan (Revision of intended learning outcomes; revision of assessment methods; changes in data collection methods; changes in sample…) Changes to the Curriculum (Changes in teaching techniques; revision of prerequisites; revision of course sequence; revision of course content; addition of courses; deletion of courses...) Changes to the Academic Process (Revision of admission criteria; revision of advising standards or processes; improvements in technology; changes in personnel; changes in frequency or scheduling of course offerings…) Spring Semester

Use of Findings

[Please report on emerging patterns obtained from the results and accordingly the action that will be taken, for example: Changes to the Assessment Plan (Revision of intended learning outcomes; revision of assessment methods; changes in data collection methods; changes in sample…) Changes to the Curriculum (Changes in teaching techniques; revision of prerequisites; revision of course sequence; revision of course content; addition of courses; deletion of courses...) Changes to the Academic Process (Revision of admission criteria; revision of advising standards or processes; improvements in technology; changes in personnel; changes in frequency or scheduling of course offerings…)

Assessment Feedback

AUB UB Cont’d

Appendix I – PLO Assessment Plan Appendix II – Other Data

(It can contain curriculum analysis matrix, relevant assignments and outcomes, data collection methods (ex: rubrics), or materials as appropriate)

Other Comments

[Provide other comments on the PLO assessment process.]

AUB 3-year year PLO LO Asse Assess ssme ment nt Plan lan Summa ummary ry

Program Learning Outcomes Assessment Three Year Summary Report

Program Information

Faculty Department

Academic Program

Degree

Academic Year Assessment Committee

(Please indicate the contact person for the PLOs assessment process.)

Full Name E-Mail Address Extension Date Submitted:

Summary

Describe briefly the PLOs assessment approach followed in your department, the assessment tools and methods used, and the assessment results collected during the three-year assessment cycle. Explain required changes based

• n

the findings. This section should not exceed 1 page.

Summary of Assessment Results/Improvement Plan

Department/Program Learning Outcomes Year and semester in

which the PLO was assessed Assessment Methods

Direct/Indirect

Data Sources

List the sources of data used for assessment like courses, projects, surveys, standard tests, etc.

Data Analysis

Compare the actual

• utcomes

to the intended

• target. Do the results

indicate that the PLO is achieved and students are learning what was intended?

Improvement Plan

How can we make it more likely that our students will achieve the intended learning outcome? 1. 2. 3. 4. 5. 6. 7.

How PLO Assessment Data will be shared?

[Describe how data, analysis, and findings are being shared and discussed with faculty and staff involved in the program/department.]

Comments on the PLO Assessment Process

[Reflect on the PLO Assessment Process: What went well? What didn’t go well? Is there anything related to assessment procedures your program would do differently next time?]

Appendix I – PLO Assessment Plan Appendix II – Other Data

It can contain curriculum analysis matrix, relevant assignments and

• utcomes,

data collection methods (ex: rubrics), charts,

• r

materials as appropriate.

Samp mple le 7: 7: Un Unive iversit rsity of

• f Cal

alifo ifornia rnia

ASSESSMENT PLAN – MASTERS AND/OR PhD Program Learning Outcomes (PLOs), Evidence, Timeline, and Process: For each PLO* indicate what kind of direct (student work) and indirect evidence (ex. surveys, focus groups) will be gathered and examined to assess student achievement of the PLO. Indicate the year the PLO will be assessed (ex. AY2010-2011). Who will participate? How will assessment be conducted, results shared, and the findings used to improve student learning? What are the desired targets (or benchmarks) for student performance/learning and other metrics? PLO #1 Direct Evidence: Indirect Evidence: Year to be Assessed: Participants: Process: PLO #2 Direct Evidence: Indirect Evidence: Year to be Assessed: Participants: Process: PLO #3 Direct Evidence: Indirect Evidence: Year to be Assessed: Participants: Process: PLO #4 Direct Evidence: Indirect Evidence: Year to be Assessed: Participants: Process: PLO #5 Direct Evidence: Indirect Evidence: Year to be Assessed: Participants: Process:

Recomme Recommended nded Action Action Plan Plan (1 (1) ➢Cr Critical ically exam examine ine the the res esults ults to determine what actions should be taken ➢In Invo volv lve relevant elevant cons consti titu tuenci encies es: relevant staff, faculty, division head ➢If you you meet eet improvement improvement target et, the goal is still improvement: you may want to raise raise your your targets. targets.

Recomme Recommended nded Action Action Plan Plan “Mays and and Ifs”

➢ If results indicate that your program is not

not meeting meeting the the targ targets ets →check whether your program is movi moving ng in in th the wro wrong ng direction direction, some action may need to be taken

➢ You may not have been able to meet your target

yet, but but ar are movi moving in in the the right right direction direction—continue to track improvement

➢ If data results are questionable—cha

change ge metho method of

• f

measurement measurement

Reporting Reporting Asses Assessment sment Resul esults

The first step in developing an assessment report is to answer the following questions: ➢Who is the audience for this report? ➢What do they want to know? At its most basic, report should have information to answer five basic questions: ➢What did you do? ➢Why did you do it? ➢What did you find? ➢ How will you use it? ➢What is your evaluation of the assessment plan itself?

Program Program Improve Improvement ment Pla Plan “Closing Closing the the LOOP LOOP”

Ut Utili ilizing zing Assessme Assessment nt Res Resul ults - What

What does

• es it

it mea mean to to “close the the loop?”

➢ Using

ng ass assessme essment nt re resul ults ts for

• r prog

rogra ram imp mpro rovem vement. 4.

• 4. Use

Use

3.

• 3. Interpr

Interpretati etation

• n

Impro mproveme vement nt

Compliance

1.

• 1. Le

Learni arning Outcomes utcomes

2.

• 2. Gather

Gathering ng ev evide idence nce

Wha What is is a program program impr improvement

• vement plan?

lan?

➢ A program improvement plan is intended to provide

programs a format for translating the recommendations made into actions for improvement or maintenance.

➢ The plan also identifies who is involved and when the action

steps are to be achieved.

Program Program Improve Improvement ment Plan Plan

Below is an example of a program improvement plan:

Recommendation:

Action step(s) What action steps must be completed to implement the recommendation? Estimated implementation date When does the program expect to begin to implement the action steps? Estimated completion date When does the program expect the recommendation to be fully implemented and/or achieved? Person(s) responsible Who will take responsibility for seeing that the actions steps are implemented? Expected Outcome What is the expected impact/outcome the recommendation will have on the program, the students, and the college, etc. if it is implemented? Estimated cost(s) What is the estimated cost of implementing the recommendation? This information will be useful for assisting the Planning & Budgeting for Institutional Effectiveness Committee. Status Update Document progress made towards achieving the recommendation.

Example Example Program Program Improve Improvement ment Plan Plan

Recomme Recommendations ndations 1.

• 1. Faculty

Faculty buy buy-in in 2.

• 2. Assessment

Assessment committees committees 3.

• 3. Train

ainin ing on

• n PLO

PLO Assessment Assessment 4.

• 4. Centralization

Centralization→ Review Review - Co Consist sistent ent Implement Implementation ation 5.

• 5. Co

Colla llabor boratio ation → no no apprais appraisal, al, rathe rather imp improve rovement ment 6.

• 6. Incentiv

Incentives

Most Common Mis iscon

• nceptio

tions about Assessin ing PLO’s

Mis Misco conception nception 1: 1: The results of assessment will be used to evaluate faculty performance. Mis Misco conception nception 2: 2: Our program is working well, our students are learning; we don’t need to bother with assessment. Misc Misconce ncepti tion 3: 3: We will assign a single faculty member to conduct the assessment. Too many opinions would only delay and hinder the process. Mis Misco conception nception 4: 4: The administration might use the results to eliminate some of the department’s programs. Mis Misco conception nception 5: 5: Assessment is a waste of time and does not benefit the students. Mis Misco conception nception 6: 6: We will come up with an assessment plan for this year and use it every year thereafter. Mis Misco conception nception 7: 7: Program assessment sounds like a good idea, but it is time- consuming and complex.

Boo

• oks

ks an and Handb andboo

• oks

ks

Som

• me of
• f th

the Consulted Consulted References References

• Banta, T., & Associates (2002). Building a Scholarship of Assessment. San Francisco, CA: Jossey-Bass.
• Banta, T. (Ed.). (2004). Hallmarks of Effective Outcomes Assessment. San Francisco, CA: Jossey-Bass.
• Center for Teaching and Learning, Assessment of Learning Outcomes Handbook (2008), American University of Beirut
• Nichols, J.O., & Nicols, K.W. (2000). The Departmental Guide and Record Book for Student Outcomes Assessment and Institutional Effectiveness.
• PACT Outcomes Assessment Handbook (1999) University of Wisconsin, Madison,
• Palomba, C., & Banta, T. (1999). Assessment Essentials: Planning Implementing and Improving Assessment in Higher Education. San Francisco, CA:

Jossey-Bass.

• Pascarella, E., & Terenzini, P. (2005). How College Affects Students, (Vol. 2). San Francisco, CA: Jossey-Bass.
• Program Assessment Tool Kit (1998) University of Massachusetts
• Program-Based Review and Assessment Western Carolina University Program Assessment Handbook
• Rudolph, Amanda and Scharff, Lauren. (2008). How to Develop Learning Outcomes for Your Courses. The Teaching Excellence Center. Stephen F. Austin

State University

• University of Wisconsin, Madison, Outcomes Assessment Manual (2000) California State University Bakersfield,
• Wiggins, Grant and McTighe, Jay. (2005). Understanding by Design. 2nd Edition. ASCD
• Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education, 13(2), 181-208
• (Coates and Richardson 2012).
• Mary (Beth Claus Tobin, 2008)
• I-Tech International Training and Education Center for Health, Guide 4

Ele lectr tronic

• nic Hand

andbooks books on

• n Ou

Outc tcomes

• mes Assessme

ment nt

• University of Wisconsin, Madison, Outcomes Assessment Manual (2000 California State University Bakersfield, PACT Outcomes Assessment Handbook

(1999) Ball State

• University, Assessment Workbook, 1999 PACT Outcomes Assessment Handbook (1999)
• University of Wisconsin, Madison, Program Assessment Tool Kit (1998)
• University of Massachusetts Program-Based Review and Assessment Western Carolina University Program Assessment Handbook
• http://oeas.ucf.edu/doc/acad_assess_handbook.pdf
• http://oeas.ucf.edu/doc/acad_assess_handbook.pdf
• http://www.tamu.edu/marshome/assess/oabooks.html) http://www2.acs.ncsu.edu.upa/assmt/resource.htm)
• http://www.eiu.edu/~assess/assessment_resources.htm)
• www.umass.edu/oapa/assessment/program_based.pdf http://oeas.ucf.edu/assess_process.htm
• http://universityevaluation.asu.edu/assess/assessguide.pdf
• http://www.cord.edu/dept/assessment/ahbcontents.html
• http://www.oecd.org/education/skills-beyond-school/AHELOFSReportVolume1.pdf

Le Learn arnin ing Outcoe utcoes Ass ssess essmen ent Li Links nks

• http://www2.santarosa.edu/media/project_learn/Pro_Learn_Handbook_FINAL_092807%5B1%5 D.pdf
• http://www.csufresno.edu/csalt/programs_resources/assessing_outcomes.shtml
• http://www.umass.edu/oapa/oapa/publications/online_handbooks/program_based.pdf
• http://www.umt.edu/provost/assessment/what_is.htm
• http://web.odu.edu/webroot/orgs/AO/assessment.nsf/pages/Terms_page
• http://cep.cl.k12.md.us:2000/resources/glossary.html
• http://www.aahe.ort/assessment/principal.htm
• http://www.ascd.org/publications/books/107042/chapters/developing-a-vision-and-a-mission.aspx
• http://www.wcu.edu/facctr/assessment.html
• www.wku.edu/sacs/assessmentmanual.htm
• http://www.assessment.uconn.edu/primer/goals1.html
• http://www.wcu.edu/stratplan/AssessRpts.htm
• https://www.ccl.org/articles/leading-effectively-articles/avoid-360-degree-feedback-pitfalls/
• https://eric.ed.gov/?id=EJ819002
• http://www.assessment.uconn.edu/primer/goals1.html
• asignorini@ucmerced.edu
• http://www.assessment.uconn.edu/primer/goals1.html
• http://www.ascd.org/publications/books/109004/chapters/What-Makes-a-Question-Essential%A2.aspx
• http://www.pima.edu/acad_services/slo/documents/Sample_Program_Student_Learning_Outcomes.pdf
• http://liad.gbrownc.on.ca/programs/InsAdult/currlo.html
• http://www.assessment.uconn.edu/primer/goals1.html
• http://www.uhh.hawaii.edu/uhh/genedfac/objectives.php
• http://www.primaedu/acad_services/slos/documents/Sample_Program_Student_Learning_Outcomes.pdf
• https://smeresources.wikispaces.com/Planning+4+-+Writing+Learning+Outcomes
• https://thestepforwardcpdblog.files.wordpress.com/2016/01/blooms-affective-domain.png
• http://www.academics.calpoly.edu/assessment/assessplanguide.htm#defining
• http://www.assessment.uconn.edu/primer/goals1.html
• https://www.researchgate.net/publication/332151395_Evidence-Based_Teaching_in_Higher_Education/download
• https://research.acer.edu.au/cgi/viewcontent.cgi?article=1013&context=research_conference_2005