Remote invigilation: a spotlight on assessment administration - - PowerPoint PPT Presentation

Remote invigilation: a spotlight on assessment administration

Nicola Mellor, Qualifications Director, CII Liam Simington, Assessment Manager, CII

• Standards. Professionalism. Trust.

• Standards. Professionalism. Trust.

Existing research

• Key themes and names
• Ref to reading list

Bibliography available

• Standards. Professionalism. Trust.

Existing research weaknesses

"keen to protect their own reputations for secure and safe assessment systems" (Jefferies, 2017)

Reputational risk Commercial sensitivity Operational focus

Lack of published research

• Standards. Professionalism. Trust.

Existing research weaknesses

Lack of published research Research issues (method and focus)

Small sample questionnaires Technical variation in RI Exam room as gold standard Reputational risk Commercial sensitivity Operational focus

• Standards. Professionalism. Trust.

Pilot

2 providers 27 candidates 2 settings (office and home) 4 countries

• Standards. Professionalism. Trust.

Pilot survey

1 4 4 2 4 6

• 3. IN THE ROOM
• 2. NEUTRAL
• 1. REMOTE

Where do you prefer your invigilator to be …

2 7 2 4 6 8

• 5. VERY INCONVENIENT
• 4. INCONVENIENT
• 3. NEUTRAL
• 2. CONVENIENT
• 1. VERY CONVENIENT

How would you rate RI for convenience?

1 7 1 2 4 6 8 ..VERY EARLY? ..EARLY? ..ON TIME? ..SLIGHTLY LATE? VERY LATE?

Did your exam start..

Did your exam start..

• Standards. Professionalism. Trust.

Post pilot questionnaire

442, 99% 5, 1%

Non-UK preference

RI Centre 2775, 85% 505, 15%

UK preference

RI Centre 2598, 79% 472, 14% 210, 7%

UK- fee will pay

Below proposed Propsed Above proposed 36, 8% 48, 10% 383, 82%

Non-UK fee will pay

Above proposed Proposed Below proposed

Further research wish list

• 1. Human interactions
• 2. Environmental

factors and risk perception

• 3. Cheating behaviours

• Standards. Professionalism. Trust.

The watching eye

• Standards. Professionalism. Trust.

The neglected situation

‘Managing the Quality of Data Collection in Large Scale Assessments’. (Maddox, B, 2017, p8), OECD Headquarters, Paris. 11- 12 May 2017. Accessed

• nline https://www.oecd.
• rg/skills/piaac/Interview

er- Respondent%20Interacti

• ns%20in%20PIAAC_Br

yan%20Maddox.pdf 25. 10.18

• Standards. Professionalism. Trust.

Dynamics of control vs. validity

The sweet spot: High confidence Authenticity of candidate performance Authenticity of task to construct L H H

The Authenticity Square

• Standards. Professionalism. Trust.

Dynamics of control vs. validity

The sweet spot: High confidence Authenticity of candidate performance Authenticity of task to construct L H H

The Authenticity Square

Trend: the greater the control, the more the task authenticity is challenged

• Standards. Professionalism. Trust.

Who manages the risks and controls?

• - Travel
• - Conduct
• Invigilation
• Rules
• Environment (physical)
• Environment (technical – equipment,

internet and platform)

• Who is in control?

How does this affect the way we perceive risk?

Candidate Centre Awarding organisation

Factors: anxiety disorders; access arrangements; unconscious biases in risk perception

• Standards. Professionalism. Trust.

Who manages the risks and controls?

• - Travel
• - Conduct
• Invigilation
• Rules
• Environment (physical)
• Environment (technical – equipment,

internet and platform)

• Conduct
• Environment (physical)
• Environment (technical – internet

and equipment)

• Invigilation
• Rules
• Environment (technical – platform)

Who is in control? How does this affect the way we perceive risk?

Candidate R.I. Centre Awarding organisation

Factors: anxiety disorders; access arrangements; unconscious biases in risk perception

• Standards. Professionalism. Trust.

The interaction between cheating and RI

The 'fraud triangle’. (Jefferies, 2017,after Ramos, 2003)

• Usual exam

nerves

• Pressure of being

watched Factors:

• Cultural context
• Communications

and marketing

• Adverse publicity
• Choice and

communication of model

• Penalties and

disciplinary processes

• Standards. Professionalism. Trust.

The interaction between cheating and RI

• Hide unauthorised

materials or web access

• Assume an invisible

invigilator is not watching / assume the invigilator is watching closely at all times

• Usual exam nerves
• Pressure of being

watched Factors:

• Cultural context
• Communications

and marketing

• Adverse publicity
• Choice and

communication of model

• Penalties and

disciplinary processes

• Standards. Professionalism. Trust.

The interaction between cheating and RI

• Hide unauthorised

materials or web access

• Assume an invisible

invigilator is not watching / assume the invigilator is watching closely at all times

• Usual exam nerves
• Pressure of being

watched

• Publicity of online cheating
• But cannot see it happening

directly Factors:

• Cultural context
• Communications

and marketing

• Adverse publicity
• Choice and

communication of model

• Penalties and

disciplinary processes

• Standards. Professionalism. Trust.

Detection and prevention

This Photo by Unknown Author is licensed under CC BY-SA

• Standards. Professionalism. Trust.

Vision of the R.I. future

This Photo by Unknown Author is licensed under CC BY-SA-NC This Photo by Unknown Author is licensed under CC BY-SA