Enhancing student project selection and allocation in higher - - PowerPoint PPT Presentation

Enhancing student project selection and allocation in higher education programmes

Johann A. Briffa1 Simon Lygo-Baker2 26 September 2018

• 1Dept. of Comm. & Computer Engineering,

University of Malta, Msida MSD 2080, Malta

• 2Dept. of Higher Education,

University of Surrey, Guildford GU2 7XH, England 1

Outline

Overview

Introduction and motivation Proposed method Results Conclusions

2

Introduction and motivation

Introduction

Final year projects – motivation

• Most undergraduate degrees include a substantial project in

the final year

• Allows student to:
• Demonstrate command of subject
• Integrate material from taught modules
• Demonstrate higher order skills (e.g. application, synthesis,

evaluation)

• Transition towards self-sufficiency
• Often the first major piece of work for students
• Usually assessed through a report / dissertation

3

Introduction

Final year projects – mechanics

• Shift in emphasis towards a research-led approach
• Involves working on a specific topic for an extended

duration (beyond standard coursework deadlines)

• Under supervision of academic staff (or possibly graduate

students or post-doctoral fellows)

• Work is performed independently but supported by tutor
• Relationship between learner and tutor is potentially of

great significance

• (Much of this also applies to post-graduate degrees)

4

Introduction

Allocating supervisor and topic

• Need to allocate all students to a finite number of

supervisor, taking into account:

• Topics of interest to student
• Topics of expertise of supervisors (esp. at higher levels)
• Equitable distribution of students among academics
• This can be problematic, as students may have strong

preference for choice of supervisor, may be due to:

• perceived shared interest in area
• topics of interest considered ‘hot’ or ‘easy’
• perceived leniency or helpfulness of academic

5

Introduction

Allocating supervisor and topic – common approaches

• Coordinator-led method:
• 1. A list of titles is published, each connected with an academic
• 2. Students select a list of preferences
• 3. Coordinator manually allocates a project per student (maybe

with a defined order of allocation)

• Student-led method:
• Students meet potential supervisors and agree on a title
• Requires a lot of pre-allocation meeting time
• Can overwhelm popular supervisors
• First-come-first-served increases pressure on students
• Can favour some students (e.g. who live closer)

6

Introduction

Other complications

• Requests for change of topic / supervisor
• Often because supervisor is not ‘preferred’
• Or when the allocated topic does not match expectation
• Difficult to solve without repercussions
• Reallocation increases imbalance in teaching load
• Multiplier effect if students see this as an option
• Ideal is to avoid the need as much as possible
• In some cases it’s unavoidable (e.g. staff illness)
• Having a published approach helps

7

Introduction

Other complications

• Multiple students working on the same title
• Requires clear separation of work
• Consideration of dependencies
• Collaborative work with third party (e.g. industry)
• Often tied to specific academic (pre-established link)
• Can also be tied to student (e.g. previous placement)
• Formal agreement with University often required
• Contingency plan to ensure student completion if industry

partner pulls out

8

Introduction

Examining projects

• Project generally has a significant weighting, so typically is

assessed by two examiners

• Final mark obtained by some process of agreement (may be

simply a weighted average)

• Significant discrepancies need to be reconciled
• Supervisor may or may not be one of the examiners

(depends on institutional policies)

9

Introduction

Allocating examiners – constraints

• Examiners need to have necessary expertise
• Marking may need to be blind in the first instance
• May be preferable to avoid consistent examiner pairings

(limits potential for problems with agreement)

• May need to avoid certain pairings of examiners (e.g. to

avoid conflicts of interest, incompatible personalities, or having two harsh or two lenient markers)

• Allocation of duties needs to be equitable among staff
• If supervisor is not an examiner, need to consider three-way

constraints

10

Introduction

Administrative overhead

• Often handled by a projects coordinator; alternatively by

the board of studies or its delegate

• Collection of proposed titles from staff
• Collection of preferences from students
• Allocation of supervisor and examiner(s)
• Moderate grade agreement where necessary
• Consider complaints and other requests from students and

staff

11

Proposed method

Proposed method

Key features

• Separation of title selection from allocation of supervisor
• Title selection in discussion with supervisor after allocation
• Allocation of supervisor takes into account topic

preferences

• Students submit a prioritized list of preferred keywords
• Each keyword is associated with one or more academics
• Avoids selecting supervisors directly (popularity contest)
• Adds flexibility to final allocations

12

Proposed method

Key features

• Uses an interactive web application to collect preferences
• Gives immediate feedback on current keyword popularity
• Students with popular choices can make informed choices:
• 1. Either accept higher risk of not getting top preference
• 2. Or lower their risk by picking a less popular topic
• Allows students to partially self-select, simplifying allocation
• Makes process as transparent as possible
• Uses a global optimization algorithm to allocate supervisors
• Quicker and less laborious than manual process
• Easier to ensure that all constraints are satisfied
• Treats all students and staff equally, removing potential bias
• Likely to come closer to an optimal allocation

13

Proposed method

Global optimization algorithm

• We express the problem as a simulated annealing problem
• Energy function (which we seek to minimise):
• 1. For each supervisor, increases exponentially with the

discrepancy between allocated and nominal load

• 2. For each student, increases logarithmically as the matched

keyword goes down in priority

• 3. For each student, applies a high fixed penalty if the allocation

does not match any keyword

• Start with a random allocation and a high ‘temperature’
• Whole process is in the order of a few minutes

14

Proposed method

20 40 60 80 100 Percentage

Acceptance Improvement

10-2 10-1 100 101 102 103 104 105 106 107 Temperature 102 103 104 105 106 107 108 109 1010 1011 Energy

Example timeline Initial ‘temp.’ high enough to accept all state changes;

• ften improves

allocation

15

Proposed method

20 40 60 80 100 Percentage

Acceptance Improvement

10-2 10-1 100 101 102 103 104 105 106 107 Temperature 102 103 104 105 106 107 108 109 1010 1011 Energy

Example timeline At final ‘temp.’ few changes accepted; rarely improves allocation

15

Proposed method

20 40 60 80 100 Percentage

Acceptance Improvement

10-2 10-1 100 101 102 103 104 105 106 107 Temperature 102 103 104 105 106 107 108 109 1010 1011 Energy

Example timeline Allocation improves in stages

15

Results

Results

Overview

• We give results from two consecutive years
• 2012/2013:
• Students choose from a list of published titles
• Each title associated with an academic
• 2013/2014:
• Students choose from a list of published keywords
• Each keyword associated with at least one academic
• Allocation using simulated annealing in both cases

16

Allocation results 2012/2013

Summary for 2012/2013

• 110 students
• 19 academics; three reduced load, one increased load
• 100 project titles; some could be taken by more than one

student, some titles were restricted by programme of study

• Students could choose a ‘Title to be agreed with supervisor’
• All but one student submitted their preferences (prioritized

list of five titles from at least three academics)

• Six students had discussed a collaborative project; these

were given priority

• Four students indicated a particular concern about

supervisor; these were also given priority

17

Allocation results 2012/2013

Preference Number of students Rank 1 33 students Rank 2 12 students Rank 3 8 students Rank 4 10 students Rank 5 9 students Random (preferred) 2 students Random (not preferred) 36 students Notes

• Rank 1 includes 10 students allocated as a priority
• Random includes 1 student who did not submit preferences

18

Allocation results 2012/2013

Preference Number of students Rank 1 33 students Rank 2 12 students Rank 3 8 students Rank 4 10 students Rank 5 9 students Random (preferred) 2 students Random (not preferred) 36 students Notes

• Only 33/110 students allocated their first choice
• Supervisor not associated with any preference in 36/110
• Led to considerable student dissatisfaction

18

Popularity of academics 2012/2013

Supervisor Capacity n1 n2 n3 n4 n5 Index Academic 1 6.3 19 22 21 13 11 283 Academic 2 6.3 12 20 15 12 11 220 Academic 3 6.3 17 14 15 6 14 212 Academic 4 6.3 14 5 6 12 6 138 Academic 5 2.5 7 5 9 7 7 103 Academic 6 6.3 6 6 5 8 4 89 Academic 7 7.5 6 5 4 3 10 78 Academic 8 6.3 3 7 6 7 2 77 Academic 9 2 4 5 2 9 7 71 Academic 10 6.3 4 4 4 7 9 71 Academic 11 6.3 5 3 6 5 3 68 Academic 12 6.3 4 2 3 8 8 61 Academic 13 6.3 3 2 3 2 3 39 Academic 14 6.3 2 1 3 3 1 30 Academic 15 6.3 3 2 1 6 26 Academic 16 6.3 2 2 5 2 26 Academic 17 6.3 1 2 1 2 18 Academic 18 6.3 1 1 1 2 14 Academic 19 4 1 1 8

Notes

ni = number of students whose rank i choice is associated with that academic Index = 5n1 + 4n2 + 3n3 + 2n4 + n5

19

Popularity of academics 2012/2013

Supervisor Capacity n1 n2 n3 n4 n5 Index Academic 1 6.3 19 22 21 13 11 283 Academic 2 6.3 12 20 15 12 11 220 Academic 3 6.3 17 14 15 6 14 212 Academic 4 6.3 14 5 6 12 6 138 Academic 5 2.5 7 5 9 7 7 103 Academic 6 6.3 6 6 5 8 4 89 Academic 7 7.5 6 5 4 3 10 78 Academic 8 6.3 3 7 6 7 2 77 Academic 9 2 4 5 2 9 7 71 Academic 10 6.3 4 4 4 7 9 71 Academic 11 6.3 5 3 6 5 3 68 Academic 12 6.3 4 2 3 8 8 61 Academic 13 6.3 3 2 3 2 3 39 Academic 14 6.3 2 1 3 3 1 30 Academic 15 6.3 3 2 1 6 26 Academic 16 6.3 2 2 5 2 26 Academic 17 6.3 1 2 1 2 18 Academic 18 6.3 1 1 1 2 14 Academic 19 4 1 1 8

Notes

First four academics account for first choice

• f 62 students

Their combined capacity is only 25.1 Five academics were

• nly listed once as a

first choice, or not at all This makes a good allocation very difficult / impossible

19

Allocation results 2013/2014

Summary for 2013/2014

• 60 students
• 17 academics; one reduced load, six increased load
• 61 keywords; 30 associated with one academic, 17 with two,

14 with three or more

• Used web application to collect preferences (prioritized list
• f five keywords)
• All students submitted their preferences
• Three students had discussed a collaborative project; these

were given priority

20

Allocation results 2013/2014

Preference Number of students Rank 1 52 students Rank 2 6 students Rank 3 0 students Rank 4 0 students Rank 5 1 student Random 1 students Notes

• Rank 1 includes 1 student allocated as a priority
• Rank 2 includes 1 student allocated as a priority
• ‘Random’ allocation was due to collaborative project

21

Allocation results 2013/2014

Preference Number of students Rank 1 52 students Rank 2 6 students Rank 3 0 students Rank 4 0 students Rank 5 1 student Random 1 students Notes

• Most students allocated their first or second choice
• Effectively no random choices
• Considerably higher student satisfaction

21

Popularity of academics 2013/2014

Supervisor Capacity n1 n2 n3 n4 n5 Index Academic 15 3 22 26 24 21 18 346 Academic 2 3 29 21 21 17 8 334 Academic 10 3 29 17 15 22 14 316 Academic 3 3 16 17 17 19 17 254 Academic 4 4.6 17 18 12 10 5 218 Academic 19 3 15 15 13 11 5 201 Academic 13 3 14 7 13 10 6 163 Academic 18 6.1 13 7 7 15 11 155 Academic 11 4.6 11 12 8 8 6 149 Academic 7 4.6 7 12 11 10 13 149 Academic 16 3 6 12 9 12 12 141 Academic 14 4.6 6 7 4 6 7 89 Academic 8 3 3 5 5 3 11 67 Academic 1 3 5 3 4 4 7 64 Academic 17 3 4 2 6 7 3 63 Academic 6 6.1 5 3 5 3 4 62 Academic 9 2 3 2 3 2 7 43

Notes

ni = number of students whose rank i choice is associated with that academic Index = 5n1 + 4n2 + 3n3 + 2n4 + n5

22

Popularity of academics 2013/2014

Supervisor Capacity n1 n2 n3 n4 n5 Index Academic 15 3 22 26 24 21 18 346 Academic 2 3 29 21 21 17 8 334 Academic 10 3 29 17 15 22 14 316 Academic 3 3 16 17 17 19 17 254 Academic 4 4.6 17 18 12 10 5 218 Academic 19 3 15 15 13 11 5 201 Academic 13 3 14 7 13 10 6 163 Academic 18 6.1 13 7 7 15 11 155 Academic 11 4.6 11 12 8 8 6 149 Academic 7 4.6 7 12 11 10 13 149 Academic 16 3 6 12 9 12 12 141 Academic 14 4.6 6 7 4 6 7 89 Academic 8 3 3 5 5 3 11 67 Academic 1 3 5 3 4 4 7 64 Academic 17 3 4 2 6 7 3 63 Academic 6 6.1 5 3 5 3 4 62 Academic 9 2 3 2 3 2 7 43

Notes

Each student’s choice may be associated with more than one academic Popularity of academics is still very imbalanced However, less-popular academics still match with enough students This makes a good allocation considerably easier

22

Conclusions

Conclusions

Summary

• Project is an important aspect of an undergraduate degree
• Allows the student to engage in deeper learning
• Opportunity for the learner to develop independent

learning skills

• However, benefits can be constrained by a range of factors
• Some constraints are based on the resources available (e.g.

staff student ratios)

• Certain strategies can offset some constraints and limit

potential for complaints and dissatisfaction

23

Conclusions

Contributions

• We have shown that a ‘fairer’ and more balanced approach

to project allocation is possible

• We have identified (and shown) the problem of popularity
• To limit this, we place emphasis on the student to select the

‘subject’ from a list, each matched to one or more supervisors

• This does not ignore the importance of the relationship

between supervisor and student

• As an added bonus, the automated process is quicker and

reduces administrative overhead

24