Setting the context: The changing assessment and feedback eco system:
Exploring changing roles, practices, environments and expectations
- Dr. Ziene Mottiar, School of Hospitality, Tourism and Leisure
Setting the context: The changing assessment and feedback eco - - PowerPoint PPT Presentation
Setting the context: The changing assessment and feedback eco system: Exploring changing roles, practices, environments and expectations Dr. Ziene Mottiar, School of Hospitality, Tourism and Leisure Assessment and Feedback The power of
Setting the context: The changing assessment and feedback eco system:
Exploring changing roles, practices, environments and expectations
Assessment and Feedback
The power of assessment in shaping what and how students learn is well recognised in the literature on assessment (Brown and Knight, 1994; Ramsden, 1997; Price, Carroll, O’Donovan and Rust, 2011; Jackal, Pearce, Radloff and Edwards, 2017). As Sadler (2010, p. 536) asserts, ‘feedback is central to the development of effective learning, partly because assessment procedures play a key role in shaping learning’.
Students may find the academic terminology of feedback difficult to understand (Carless 2015), fail to act on feedback received (Pitt & Norton 2017)
Molloy 2013a). W ith Increased student numbers and an increasingly diverse student population (Carless 2017), the time and effort required by staff in the provision
As noted by Pitt & Norton (2017), emotional responses can have an important impact on how students react to feedback.
The literature identifies many challenges
Societal i issu ssues Inc ncreased no.
nts Educatio ional p polic icie ies Cha hang nging ng expectations
Cha hang nging ng rol
Graduate te A Attr ttribute tes
Assessment and Feedback Eco-system
Highe her e education m
Student nt w wor
life Technol hnolog
New ew t tea eaching p practices es Cha hang nging ng student nt p prof
Learners
The LEAF (Learning from and Engaging with Assessment and Feedback) Project
Be Inspired!
Peer Assessment Early Feedback Successive Assessment W eightings Feedback in place of assessment Feedback in place of grade Assessment calendars TESTA Feedback by video Video assessments Assessment rubrics Class based polling Co-creation in assessment
Gasta L Light htni ning ng Present ntations
nt and nd feedback practices es at t the e module e and programme e lev evel el
Speakers: Emma Robinson/ Jane Courtney
Peer Learning
Peer Learning: Who used it?
(W ebsite HERE)
○ 160 first year students (5 Lecturers) ○ Level 8, General Engineering
○ W eek 6-12 Semester 1 ○ Basic Requirements each week for engagement (1Q, 2A, 1R) ○ Final quiz using questions as a bank within webcourses (Further details here: https:/ / tudeep.home.blog/ 2019/ 02/ 07/ peer-learning-using-peerwise/ )
PeerWise
Feedback on PeerWise
Student Feedback on Peer Wise
Speaker: Jane Courtney
Peer & Self Assessment
Test 2: Cohort: Engineering second years ~ 35 students Work: Individually submitted Research Articles Method: Brightspace “Video Assignment” (Bongo) Test 1: Cohort: Multi-disciplinary (Computer Science / Engineering) final years ~ 30 students Work: Group Project Presentation and Individual Blogs Method: Google Forms Grading Rubric
Peer & Self Assessment
Outcomes: Blog: A record with a handy how-to guide for Google Forms:
https://tudeep.home.blog/2019/02/25/peer-self- assessment-dont-like-your-mark-grade-it-yourself/Feedback: Rich detailed feedback from all sides - self, lecturer and peers Grades: No grade queries!
Peer & Self Assessment
Outcomes: Blog: A record with a handy how-to guide for Google Forms:
https://tudeep.home.blog/2019/02/25/peer-self- assessment-dont-like-your-mark-grade-it-yourself/Feedback: Rich detailed feedback from all sides - self, lecturer and peers Grades: No grade queries!
Peer & Self Assessment
Outcomes: Blog: A record with a handy how-to guide for Google Forms:
https://tudeep.home.blog/2019/02/25/peer-self- assessment-dont-like-your-mark-grade-it-yourself/Feedback: Rich detailed feedback from all sides - self, lecturer and peers Grades: No grade queries!
SUCCESS!
Early Feedback
Lean anne Har arris - School hool of
nd Health h Scienc nces
What does this tool address?
Types of tools trialled:
Students discuss in groups how they marked previous assessments
Early Feedback
“… .students continually rate timely, useful feedback high amongst their preferences” (Li & De Luca, 2014)
Accounting & Finance
Mechanical & Design Engineering
Mechanical & Design Engineering
“Information only becomes feedback when it is used productively” (Carless 2005)
more captive audience
repetition of material
Early Feedback - Previous Assessments
Hospitality Management & Tourism
DT596B/2
Hospitality Management & Tourism
Biological & Health Sciences
DT229/2
Hospitality Management & Tourism
early feedback
previous year’s CAs
uploaded with common errors
previous year’s CAs
feedback
the feedback
grades
“The role of the student in the assessment and feedback process has become a focal point of sustainable feedback, with a move from the traditional ‘monologue’ model of feedback to a more ‘dialogue’ model” (Boud & Molloy, 2013a; Orsmond, Maw, Park, Gomez, & Crook, 2013)
“A dialogue model creates a more active, engaged role for students in the generation and use of feedback” (Carless, 2015)
Successive Assessment Weighting
Lean anne Har arris - School hool of
nd Health h Scienc nces
Biological and Health Sciences
Low weighted Lab Report 1, detailed feedback, and higher weighted Lab Report 2.
= 8% = 12% +
Biology Lab Report 1
INTRODUCTION Background information, explanation of terms, principle of the experiment. AIMS Identify the purpose of the investigation. MATERIALS & METHODS There is no need to rewrite the method. Simply writing “as per manual” is acceptable. However, you must note any changes or modifications to the method. RESULTS Any data collected during the experiment. This includes graphs, tables or drawings. Each must have a proper title. There should be no interpretation or discussion of the results, they are simply stated here. DISCUSSION This is where you explain the meaning of the results. Are they as expected? If not, can you postulate an explanation? Were there any problems? How would you overcome these in the future? Note: “I enjoyed this experiment” is not a valid scientific observation! CONCLUSION 1-2 lines summarising your main findings. REFERENCES If used this is a list of references numbered in order of 1+ of how you used them in the Introduction section and any further sections such as the Discussion.WHAT AT DOES ES TH THIS IS TOOL OOL ADDRESS? SS?
struggle to gauge the required ac acade demic stan andar dards and identifying how to improve.
feel the qua quality of
Successive Assessment Weighting
practical component.
12% of the marks.
comments from Report 1 to help improve Lab report 2 which is worth more marks.
Practical Assessment breakdown
“In my opinion we would benefit in more assignments with each carrying less marks” “I found it really advantageous as it meant I could better prepare for the second lab report and learn from mistakes I made in the first lab report. It gave us a better idea
what level of work/material they expect from us” “Gave time to work on the lab report technique between the first and last lab report” “I liked that they were weighted differently but it would be better had they weighted the first one even lower and maybe had a bit more instructions” “Could be weighted lower” “Less weighting” “It’s a good tool to revise the course” “Should be more direction in the first at to layout” “I think the lab reports were very beneficial to me as I learned from my mistakes” “Very advantages because the feedback from the 1st report helped a lot for the 2nd
“It’s a handy way to engage students to participate in class and also interactive learning is the best way to study”
Feedback before or in place of grade
Natalie Hop
ns ( (School hool of
nd Health S h Scienc nces) Mairea ead Stack, Hel elen en O’Brien en G Gatel ely, N Noel el O’Nei eill, a and L Louise e Bel ellew ew Why u hy use thi his method hod?
How
can n you you u use it?
Schools and courses where this was used
Feedback in place
Natalie Hopkins & Mairead Stack, Biological and Health Sciences. DT223 and DT225 yr 2 Noel O’Neill, Mechanical and Design Engineering. DT005 yr 3, DT7013 yr 2 Helen O’Brien Gately, Accounting and Finance. DT315 yr 1. Louise Bellew, Hospitality, Management and Tourism. DT458 yr1
Examples of Tool used
Feedback in place
Students complete group disease project following guidelines given. Students given feedback initially, percentage mark following later. Direct feedback (one to
submission of their Final Year Project
attempts at end of semester. Grade student receives for work across semester reflects their engagement with feedback Students submit report for feedback (one to one written and verbal). Allow to resubmit taking feedback into account for final mark.
Student Feedback
student
feedback.
What did the students think?
problem areas are, rather that grade alone
subsequent submissions.
specific to their submission, rather than general feedback.
feedback without risk of being penalised for mistakes.
risk appealed to them.
(verbal and written) very helpful.
Lecturers?
Lecturer Feedback
groups and group projects.
and engagement.
relationship between lecturer and student.
groups.
feedback on board.
feedback on board for improvement in subsequent submissions.
feedback was used.
feedback to students.
consuming than in class tests.
losing marks while learning.
group.
time programme - coming from industry.
Lo Louis ise Ly Lynch (School of Civil & Structural Engineering)
Feedback by audio or video
Issu ssues s to address: ss:
adequate prompt and detailed feedback.
if they are getting on ok, or falling behind.
available time restrict their ability to provide prompt and detailed feedback.
○ Screen capture with voice over through question, highlighting common mistakes.
○ Kaizena with Google docs - reuse common feedback phrases.
○ individual feedback with Voice Recorder app.
○ Brightspace - individual recording possible, “Insert Stuff” - allows reuse of common feedback statements.
○ unique group feedback with Voice Recorder app.
Feedback by audio or video
Feedback by audio or video
Advan antag ages:
aided and can be referred back to if you want to see it again.”
easily understood.”
easy to understand.”
Feedback by audio or video
Di Disa sadvantages: s:
in subsequent years could see time saving.
○ “...Too quiet” ○ “...Hard to understand at times”
visual support.”
where we went wrong.”
Feedback by audio or video
% - “I wou
his tool
n ot
her modules.” .”
% - “I feel eel that this will hel elp me e with future e asse ssessm ssments. s.”
77% - “I enj njoy
he tool
n thi his mod
72% - “The he tool
helped me und nderstand nd the he subject m t matte tter.”
% - “I felt it ha had no no bene nefit to
11% - “I fel elt the t e tool was not ef effec ectivel ely del eliver ered ed.”
Feedback by audio or video
% - “The e tool made m e me m e more en e engaged ed w with the e module. e.” [30% neutral]
% - “The he tool
me m mor
mot
.” [41% neutral]
○ Engagement, Interactive ○ Assessment (formative, summative), Feedback ○ Polling
○ Tool Survey - Agree, Strongly Agree ○ +‘I found it very useful and engaging’ ○ +‘Very good way of learning’ ○
used for too long’
○ Training required, portability
Technologies, TU Dublin (Dolores McManus)
Anne Hurley (School of Hospitality Management & Tourism)
Speaker: C Clíona
Met ethod u used ed:
Why hy use thi his method hod?
etc.
Assessments submitted by video
Assessments submitted by video
Fin indin ings:
Biological & & Hea ealth Scien ences es
student-focused and allowed flipping of content.
continue in subsequent years and videos can be used as future learning tools Civ ivil il E Engin ineerin ing
step to create the video.
Assessments submitted by video
Mar arketing
Musi sic
innovation and creativity.
Fut utur ure: Video submission tool available in Brightspace. Supporting mixed approaches.
Assessments submitted by video
Mai airead ad Stac ack (School of Biological and Health Sciences)
Natalie Hopkins Why hy u use t e thi his met etho hod?
eedback L Liter eracy.
eedback L Loop. How c can y n you use e it?
Using Rubrics in Assessment and Feedback
Assessment Description with Task Specific Criteria
Sections: Subsections Length Marks Introduction: DescriptionSchool of Biological and Health Sciences
Medical Physiology (2 stage assessment) Optometry, Clinical Measurement Science and Medical Physics and Bioengineering Biological Basis of disease (Group assessment) Biomedical Science, Biomolecular Diagnostics Pathophysiology (Group assessments) Human Nutrition and Dietetics, Public Health Nutrition DT225 (Programme Level) Food Studies (1) Nutrition Science (2) Nutritional Epidemiology (2) Pathophysiology (2) Applied Nutrition (3)
Level 8 Degree courses
What did students think of webcourses Rubric?
Rubric was set up with specific set percentages per topic of discussion. Students had a higher chance of a lower grade, skewing it towards students performing poorly. The feedback although useful for future projects didn’t help for the current module to improve my grades. I think use of the Rubric is very useful in most subjects. For this assignment it was helpful in guiding me to my final submission.
Mixed feelings on this tool!
I feel that it (feedback) will help me to be more successful in future assignments I want feedback as this mark will contribute to my degree classification
Lecturers - Webcourses Rubric?
Easy to use Efficient Paperless Marking scheme in the webcourses rubric somewhat restrictive.
Assignment and SafeAssign tools of VLE
the gradebook
I've found it very helpful in terms of ensuring consistency with grading. It's also easier than making notations For assignments with multiple parts, it got a little bit tedious setting up the rubric
Gre reg Byrn rne Provision of a detailed and up-to-date assessment calendar is essential to allow students to plan their studies Ideally, an assessment calendar should be…
A year long assessment calendar can also be a useful tool during programme reviews.
Assessment calendars
In my role as a Teaching Champion for the College of Sciences & Health I have been reviewing best practice and promoting the use of a Google calendar
bookmark
Assessment calendars
Assessment calendars
T ransforming the E xperience of S tudents T hrough A ssessment
TESTA
Louise Bellew, School of Hospitality Management & Tourism
understanding asse ssessm ssment patterns on whole degree programmes.
programme teams id identif ify enha nhanc ncement nts f for
nt learni ning ng based on evidence and assessment principles.
feedback
Why use TESTA?
Why use TESTA?
Tansy Jessop, DIT 2018
Why use TESTA?
Tansy Jessop, DIT 2018
TESTA
Programme Audit Assessment Experience Questionnaire (AED) Focus Group
TESTA Masterclass
Masterclass with Tansy Jessop, October 2018 - all staff
Geraldine Gorham), DT412 (Louise Bellew) and DT406A, DT408A (Ziene Mottiar)
Tansy Jessop, DIT 2018
Questionnaire is used to understand how students learn from assessment and feedback.
time, facilitated by staff.
AEQ
○ Type of assessment formative and summative ○ Breakdown in terms of assessment and exam ○ Example of the type / range of assessment ○ Type and quantity of feedback oral (in minutes) and written (in words) ○ Time to return feedback
Programme Audit
Next step
○ TESTA workshop with programme team, to be facilitated by Tansy Jessop May 2019 (AED findings & programme audit), May 2019 ○ Types of issues for programme team ■
Balance of summative formative
■
Review connections within/ across modules
■
Feedback as dialogue
College of Sciences and Health
CoCREATE Assessment and Feedback
Are students consumers or producers?
An age old question…..
Consuming?
Are our students….
Creating?
Biggs (2003): “Learning cannot be transmitted by direct instruction, but is created by the students learning activities”.
Creation as an alternative paradigm?
Neary (2009): “Real Life, Complex and Unstructured research-like activities”
Authe hent ntic A Assessment nt a and nd Learni ning ng Advanced topic, assessment student specific and student centred.
Students as Creators (of Assessment)
Peer Discussion Reflections on and as learning Student negotiated assessment
Authe hent ntic L Learni ning ng and nd Assessment nt
Students as Creators (of Learning)
Students as Creators (of Feedback)
Peer Feedback: ‘Two Stars and a Wish’
PAL and PAF: Student designed learning activities Authen entic L Lea earning, Asses essmen ent a and Feed eedback
Students as Creators….does it work?
Group-based learning “I really enjoyed the team
members give their ideas, taking them in and giving my
be the rea eal hi highl hlight ht of the project”. Authe hent ntic L Learni ning ng and nd Assessment nt
Students as Creators….does it work?
To effectively create, or re-design, as part of an student centred learning, assessment and feedback approach, a student must:
se and sy synthesi size ze several mu multimo modal sources on the subject content.
Brai ainstorm, conc
ate, implem emen ent and eval aluat ate their idea(s).
eflec ect and refin ine not only their design, but also their understanding.
Are these not some of the key skills lls we want every student to develop in every module?
Authen entic L Lea earning, Asses essmen ent a and Feed eedback
Col
nces and nd Health h Teachi hing ng Cha hampion
Students as Creators….how to?
TU D Dublin T n Team T Teachi hing ng Fellow
hip 2 2019-20 2020 20
Students as Creators….how to?
www.cocreatetudublin.wordpress.com @CTUDublin
Icon made by FreeIcons from www.flaticon.com‘Assignment Mapping’
Alan McDonnell M.Sc. (Eng.), Pg.Dip. (2018)
School of Mechanical & Design Engineering City Campus (Bolton St.) DIT/ IT/ TU TU D Dublin L Lecturer (since 2002) DT001 ( (Product Desi sign) Y Year 1 C Class ss Tutor (since 2016)
Developing a ‘colour-coded outline progression’
in a single semester.
76
Inspiration
I w was delivering ng cont
nt on
he prog
for
years… . “Wha hat c cou
do a
new) Year 1 C Class Tutor
new s student nts improv
the heir p planni nning ng a and nd t time m mana nagement nt?” Based on ‘practicality’, not ‘pedagogy’ or ‘research’ (..yet)
Inspiration
Gantt C tt Charts ts (I a am an en engineer er!) Based on ‘practicality’, not ‘pedagogy’ or ‘research’ (..yet)
Inspiration
No
her prog
n my School hool ha has develop
anular ar, visu sual str trate tegy f for stu tudents ts! Based on ‘practicality’, not ‘pedagogy’ or ‘research’ (..yet)
Inspiration
(Inactive) Map apMyProgram amme proje ject [ https:/ / sites.google.com/ site/ mapmyprogramme/ home ] Based on ‘practicality’, not ‘pedagogy’ or ‘research’ (..yet)
81
Sample
Rationale
Identify assignment 'pinch points‘, typically in Week 12 and Week 13 Collaborate to revise issue/ submission dates Offload responsibility of appreciation of assignment time frames… … to the students [PRE-ISSUE of map document] For the programme team...
Rationale
Appreciate assignment time frames (each has a start, middle and end) Identify 'pinch points' in assignment submissions Identify lecturing breaks + additional capacity for assignment completion Improve their general planning ability
83
[POST-ISSUE of map document] For the students...
Preparation &
Rol
Week 1 call Week 3 reminder Week 4 review Week 4 rollout
Current nt Implement ntation
Excel Spreadsheet Variety of tedious reformatting sessions Custom VBA macros (to develop the timeline colour-coding)
Development W orkload (p
(per-sem emes ester er)
Advan antag ages
Semester-wide assignment design & implementation Higher quality, semester-wide data visualisation for students Re-tasking for other assessment strategies (e.g. Year 2 Super- assessment)
Disad advan antag ages
Complete Programme Team buy-in ** Developer buy-in Self-imposed formatting + presentation standards Lack of automation in document development
85
Perception
Inc ncreased autom
hin n Excel) Alter ernate e database e dev evel elopmen ent s system em Pro rogra ramme-wid ide im implementatio ion Onl nline ne a access f for
team and nd student nts (via alter ernate e dev evel elopmen ent system em)
Development Potential
Any q quest stions… s… ?
alan.mcdonnell@ dit.ie
Thanks for listening… !