Scribes or Independent with ICT? Can technology replace human - - PowerPoint PPT Presentation

Reliant on Readers, Stuck with Scribes or Independent with ICT? Can technology replace human support in examinations?

9th European e-Accessibility Forum Cité des sciences et de l'industrie - Universcience – Paris 8 June 2015

CALL Scotland

• Communication, Access, Literacy

and Learning

• Funded primarily by Scottish

Government

• National Assistive and

Communication Technology

• Service, research and development

www.CALLScotland.org.uk

Scottish School Assessments

• Formal examinations sat

by most students in Scotland.

• Traditional paper

format.

• 675,361 examinations

sat by 132,588 candidates in 2014.

• Administered by the

Scottish Qualifications Authority (SQA).

“Assessment arrangements allow candidates who are

disabled and/or who have been identified as having additional support needs appropriate arrangements to access the assessment without compromising its integrity.”

Introduction to Assessment Arrangements for Schools and Colleges SQA January 2010, revised July 2014

SQA Assessment Arrangements

Number of Candidates requesting AA 14,214 % of total candidates 10.7% Number of requests for AA 43,344 % of total examination entries 6.42%

SQA Assessment Arrangements 2005

Total number of requests 41,454 Assessment Arrangement Number of requests which included the specific arrangement Percentage of requests which included the specific arrangement Extra Time 33,146 79.96% Reader 15,740 37.97% Scribe 14,505 34.99% PA Referral 3,658 8.82% Use of ICT 2,388 5.76% Transcription with correction 1,178 2.84% Coloured Paper 1,108 2.67% Calculator 893 2.15% Transcription without correction 742 1.79% Enlarged Print 737 1.78% Question Paper signed to candidate 92 0.22% Candidate Signs Responses 38 0.09% Use of tape recorder for responses 26 0.06% Braille 19 0.05%

• Visually identical

to hard copy papers

• Read questions

with text-to- speech

• Use on-screen

drawing tools

• Type and choose

answers on screen

Digital Question Papers

• Zoom in and out
• Papers (without

answer boxes) can be re-flowed

• Set page and text

colours

• Navigate with

keyboard/mouse

• Read with text-to-

speech

Visual impairment / dyslexia

• Type into answer

boxes

• Draw and annotate
• Spellchecker
• Word prediction
• On-screen

keyboards

• Switch access
• Speech recognition

Writing and answering

“many of the students commented that the synthetic voice was of poor quality and was difficult to understand” “The most common suggestion for improvement to the PDF examinations was to improve the quality of the synthetic voice.”

Nisbet, P.D., Aitken, S., Shearer, N. (2004) Trial of External Papers in Accessible PDF for Candidates with Additional Support Needs. http://www.adapteddigitalexams.org.uk/Downloads/Reports/

Text-to-speech voice quality

2008: ‘Heather’ licensed for Scottish schools 2011: ‘Stuart’ developed and licensed > €2.7 million cost saving compared to schools buying high quality TTS Schools get the voices from: http://www.thescottishvoice.org.uk

..so we licensed high quality Scottish voices from CereProc in Edinburgh

www.CereProc.co.uk

Low cost -> wider access

All schools and students have access at zero cost to:

• Adobe Reader
• Text reader

software

• High quality

voices

• Past papers

from SQA’s web site

“Simplify, then add lightness” Colin Chapman, Lotus

Can technology replace human support?

5 000 10 000 15 000 20 000 25 000

2008 2009 2010 2011 2012 2013 2014

Reader Scribe ICT or Digital Paper

Number

• f AA

requests

Assessment arrangements 2014

Extra Time 34,494 Transcription with correction 863 Separate Accommodation 24,277 Transcription without correction 421 Reader 9,399 Calculator 341 Scribe 7,105 Modified Content (e.g. text description of images) 308 ICT / word processor 6,786 Referral of script to the Principal Assessor 192 Digital Question Papers 3,540 Adapted Certificate (e.g. in Large Print / Braille) 68 Rest Period 2,232 Question Paper signed to candidate 32 Coloured Paper 1,793 Taped transcription-Live Presentation 26 Prompter / Practical Helper 1,783 Braille 18 Enlarged or Adapted Print Question Papers 1,064 Candidate Signs Responses 11

Problems and limitations

• 1. “Integrating the question papers and answer

booklets into one.”

• 2. “Pronunciation of some of the longer and

geographical terms was poor.”

• 3. “takes up a lot of paper/ink when printing”
• 4. “Two machines crashed during Higher English

exam.”

• 5. “give symbols for Science/Maths in order to do

more straight on to the digital paper”

Accommodation and costs

• 9,399 reader requests; 7,105

scribe requests

• ~ 18,000 individual exams
• ~ 27,000 hours
• Say average €28/hour for

reader/scribe?

= €756,000

• Say €14/hour for invigilator?

= €378,000

TOTAL = €1,134,000 in 2014

(Although it was €2.26 m in 2013)

• “The pupils' experience of using the digital exams was a

very positive one. Pupils adapted to the technology with ease and each found their own preferred way of using it - some pupils used the digital reader only for text that they struggled with in the paper copy whereas

• thers did not look at the paper exam and worked

exclusively on the computer. It was this flexibility of use that impressed me.”

• “I think the digital exam system is an excellent option

for pupils with ASN and gives them more independence.”

• Increases candidates’ feelings of independence and

control over their speed of progress. Helps with confidence to face next step, e.g. college, work.

Independence

Conclusions – PDF Digital papers are:

• Accessible for many candidates who require Assessment

Arrangements, but they are not a universal solution for all candidates and all assessments.

• Low cost to produce and use.
• Reliable, familiar and secure.
• Accessible on different platforms.
• A pragmatic solution to the problem of providing digital

versions of assessments.

• A good solution for paper-based assessment systems.

A viable alternative to human readers and scribes.