Science Accessibility for Students with Visual Impairments - - PowerPoint PPT Presentation

Science Accessibility for Students with Visual Impairments

Presenters: Linda M. Hooper James P. Rice Danielle Guzman-Orth Discussant: Cary A. Supalo

National Conference on Student Assessment San Diego, California June 27, 2018

Presentations and Discussion

• California Science Test (CAST) Overview
• CAST: Applying Twinning for Students with

Visual Impairments

• CAST Cognitive Lab Study for Students with

Visual Impairments

• Multisensory Learning for Students Who Are

Blind or Visually Impaired in the 21st Century

California Department of Education

2

3

California Department of Education

CAST Overview

CAST: Summarized

• The CAST is part of the California Assessment of

Student Performance and Progress (CAASPP) System.

• The CAST is administered to all students in grades

five and eight and once in high school (i.e., grades ten, eleven, or twelve).

• Once a high school student takes the CAST, that

student may not take the test again.

• CAST field testing began April 2, 2018. As of June 20,

approximately 1.5 million students had completed the assessment.

California Department of Education

4

CAST Key Events: How We Got Here

5

California Department of Education March 2017–November 2018 K–8 Science Instructional Materials Review January–February CA NGSS Framework Focus Groups & Public Comment March–May Science Leadership Team develops State Implementation Plan for CA NGSS September California State Board

• f Education

(SBE) adopts CA NGSS March–July CAST Pilot Test April–July CAST Field Test January–July CAST Operational Test July Begin Science Assessment Stakeholder Meetings November SBE adopts the Science Curriculum Framework March SBE adopts the CAST

We are here 2013 2014 2015 2016 2018 2017 2019

Planning

Legend:

Board Action Testing

CAST Background: Blueprint

• The CAST blueprint was approved by the State Board
• f Education (SBE) on November 8, 2017.
• The blueprint reflects the complexity of the California

Next Generation Science Standards (CA NGSS) and their multidimensional aspects.

• The CAST includes three science domains (Physical

Science, Life Science, and Earth and Space Science).

• Engineering, Technology, and Application of

Science domain will be incorporated into the three science domains.

California Department of Education

6

CAST Background: Reporting

• The CAST student score reporting structure was approved

by the SBE in November 2017 and includes an overall scale score and the three-dimensional (3-D) domains of Life Science, Physical Science, and Earth and Space Science.

3-D Overall Scale Score four achievement levels 3-D Life Science 3-D Physical Science 3-D Earth and Space Science

California Department of Education

7

CAST Background: General Achievement Levels

• The CAST general achievement level

descriptors (ALDs) provide generic descriptions of student performance level expectations across four levels.

• The general ALDs will be used to set grade-

level-specific ALDs and will be used to develop reporting ALDs that describe levels of performance to educators, students, and parents.

California Department of Education

8

CAST General ALDs (1)

Level Description 4 The student has exceeded the performance expectations of the CA NGSS by demonstrating a thorough understanding of the CA NGSS and a thorough ability to apply his or her knowledge and skills through the CA NGSS Science and Engineering Practices, Disciplinary Core Ideas, and Crosscutting Concepts. 3 The student has met the performance expectations of the CA NGSS by demonstrating an adequate understanding of the CA NGSS and an adequate ability to apply his or her knowledge and skills through the CA NGSS Science and Engineering Practices, Disciplinary Core Ideas, and Crosscutting Concepts.

California Department of Education

9

CAST General ALDs (2)

Level Description 2 The student has nearly met the performance expectations of the CA NGSS by demonstrating a partial understanding of the CA NGSS and a partial ability to apply his or her knowledge and skills through the CA NGSS Science and Engineering Practices, Disciplinary Core Ideas, and Crosscutting Concepts. 1 The student has not met the performance expectations of the CA NGSS by demonstrating minimal understanding of the CA NGSS and a minimal ability to apply his or her knowledge and skills through the CA NGSS Science and Engineering Practices, Disciplinary Core Ideas, and Crosscutting Concepts.

California Department of Education

10

CAST High-level Test Design

California Department of Education

11

CAST Design Goals

• Promote improvements to teaching and learning.
• Incentivize science instruction in every grade.
• Measure the range and depth of CA NGSS.
• Provide models of high-quality assessment items that reflect

the fidelity to the CA NGSS.

• Minimize testing time and costs.
• Accessibility for all California students.

California Department of Education

12

CAST Operational Test Design (1)

Assessment Characteristics Assessment Segment A Assessment Segment B Assessment Segment C Performance expectations (PEs) measured by grade/ grade span Grade 5: G3–5 PEs* Grade 8: G6–8 PEs Grades 10, 11, or 12: G9–12 PEs Grade 5: G3–5 PEs* Grade 8: G6–8 PEs Grades 10, 11, or 12: G9–12 PEs Grade 5: G3–5 PEs* Grade 8: G6–8 PEs Grades 10, 11, or 12: G9–12 PEs

Scope and depth of measurement Measures a broad sample of CA NGSS PEs Deep measurement

• f a targeted sample
• f a few PEs provided

in item sets Broad and deep—full range of measurement

• f PEs for each grade

span. Reporting level Contributes to student and group scores Contributes to student and group scores Contributes to group scores *Includes the foundational concepts introduced in K–grade 2

California Department of Education

13

CAST Operational Test Design (2)

Assessment Characteristics Assessment Segment A Assessment Segment B Assessment Segment C Type of items Includes selected- response, technology- enhanced, machine- scorable items Two Performance Tasks (PTs) — includes selected- response, technology- enhanced, and constructed response items Includes selected- response, technology-enhanced items (as with segment A) or a single PT (as with segment B) Number of items 32 to 34 discrete (stand- alone) items Two PTs with four to six items each One PT with four to seven items or six to seven discrete (stand-alone) items

California Department of Education

14

CAST Operational: Segment A

Segment A is a two-stage adaptive segment.

• Segment A combines technology-enhanced, machine-scorable,

short answer, and selected response items that cover a very broad range of the CA NGSS performance expectations.

• Segment A contributes to student and group scores.

Stage 2: Second set of items Stage 1: First set of items

Mix of Difficulty Levels Easy Difficulty Level Medium Difficulty Level Hard Difficulty Level

California Department of Education

15

CAST Operational: Transition from Segment A to Segment B

• Performance in Segment

A can guide the selection

• f science domains

presented in Segment B.

• The assignment of the

science domains in Segment B will be random unless performance on a particular science domain in Segment A is significantly weak.

Segment B: Selected science domains via PTs

Earth and Space Science Physical Science Engineering, Technology, and Applications

• f Science

Life Science

Screener Segment A: All science domains via discrete items

Transition Example

California Department of Education

16

CAST Operational: Segment B

Segment B includes PTs that require students to solve a series of complex problems set in domain- specific contexts, which deeply measure a student’s command of selected CA NGSS performance expectations.

• PTs consist of four to six items within a

science domain.

• Segment B contributes to student and group

scores.

California Department of Education

17

CAST Operational: Segment C

Segment C includes a range of items that, collectively, broadly and deeply measure the CA NGSS performance expectations associated with the tested grade span.

• Encourages the teaching of science at all

grade levels

• Contributes to group-level scores

California Department of Education

18

CAST Student Accessibility

California Department of Education

19

How to Get Involved in the Development of the CAST

Opportunities for California educator involvement may include:

• Item writing
• Item review
• Scoring
• Data review
• Range finding

To get involved, complete the content reviewer application at http://caaspp.org/reviewers.html.

California Department of Education 20

For Further Information

CDE CAASPP Office 916-445-8765 caaspp@cde.ca.gov California Department of Education CAASPP Web page https://www.cde.ca.gov/ta/tg/ca/ CalTAC Assistance 800-955-2954 caltac@ets.org Join the CAASPP listserv for weekly updates! Send a blank e-mail to subscribe-caaspp@mlist.cde.ca.gov.

California Department of Education

21

California Science Test (CAST): Applying Twinning for Students with Visual Impairments

James Rice Presentation for the Council of Chief State School Officers National Conference on Student Assessment June 2018

22

Overview (1)

• Background
• Introduce twinning
• Application
• Next steps

23

Promoting Access in Assessments

• Guidelines and best practices (AERA, APA, &

NCME, 2014; APH, n.d.; BANA, 2011)

• Universal Design (CAST, 2014; Thompson,

Johnstone, & Thurlow, 2002)

• Omitting test items/domains
• i.e., selective participation (Christensen, Albus,

Liu, Thurlow, & Kincaid, 2013)

• Multitiered conceptual models for accessibility

resources and accommodations (PARCC, 2016;

Smarter Balanced, 2016)

24

Science Is Visual (1)

• One example—genetics
• Goal: to demonstrate understanding using the

Punnett square

• Challenge: pea color (visual attribute)
• Opportunity: texture (visual and physical

attribute)

25

Science Is Visual (2)

• Technology can

allow for a more authentic assessment than a multiple-choice item.

• In this example,

students can make a choice of “Water Velocity,” which influences how the item will score.

26

Twinning—Conceptual Underpinnings

• Context
• The target of twinning is students with visual

impairments (e.g., students who are blind and require a braille accommodation).

• Challenge
• Items are not “born accessible.”
• Braille alone is not a sufficient support for all

students and all technology-enhanced items.

• “Twinning” is a broad continuum for accessibility

solutions applied to an existing item pool.

Least-Intensive Solutions Most-Intensive Solutions

27

Twinning Process (1)

(Guzman-Orth, Pooler, & Sova, in progress)

• Item type
• Review item type

characteristics.

− Design − Content − Functionality − Response format

28

Item Type Action Steps Twin Type Create Twin Comparison

Twinning Process (2)

(Guzman-Orth, Pooler, & Sova, in progress)

• Action steps
• Evaluate item

characteristic needs.

29

Item Type Action Steps Twin Type Create Twin Comparison

Twinning Process (3)

(Guzman-Orth, Pooler, & Sova, in progress)

• Twin type
• Review of

accessible item

• ptions
• Crosswalks

30

Item Type Action Steps Twin Type Create Twin Comparison

Twinning Process (4)

(Guzman-Orth, Pooler, & Sova, in progress)

• Create twin
• Revise or redesign

inaccessible item to the accessible twin version.

31

Item Type Action Steps Twin Type Create Twin Comparison

Twinning Process (5)

(Guzman-Orth, Pooler, & Sova, in progress)

• Comparison
• Evaluate and

compare the original (inaccessible) item with the twinned (accessible) version.

32

Item Type Action Steps Twin Type Create Twin Comparison

Applying Twinning

Review Aspects of the Item (High School) (1)

34

Review Aspects of the Item (High School) (2)

• Item types: Custom interaction plus Match

interaction

• Specific item characteristics:
• Select the radii to create a pie chart

representing the percentages of each genotype in the F2 generation AND add the allele pair labels to the pie chart sections.

• Item aligns to performance expectation (PE)

HS-LS4-3.

35

Evaluate the Needs of the Item (High School)

• The item aligns to the PE by requiring that a

student:

• Recognize patterns in the Punnett square

data and use the patterns to deduce the inheritance.

• Represent that inheritance as pairs of alleles

(i.e., genotypes) that make up certain proportions of the F2 generation.

36

Review Twinning Options (High School)

• Neither this Custom interaction nor this Match

interaction are accessible for students with visual impairments.

• Twinning to a different item type (e.g.,

multiple choice or grid or inline choice list) can make the content assessed in the item accessible.

• Twinning should maintain the construct and

as much of the rigor as possible.

37

Create the Twinned Item (High School)

• Each menu has “75 | 50 | 33 | 25 | 0”
• Test takers select one value from each

menu

38

Compare the Parent and Twin (High School)

• Both items require students to translate data

in a Punnett square into percentages of genotypes that represent proportions of the F2 generation.

39

Review Aspects of the Item (Grade 8) (1)

40

• Item type: Custom (scored simulation)
• Specific item characteristics:
• Select and move the sliders to a number on

the slider bar to indicate a number of particles to add to one or both sides of the model.

• Aligns to PE MS-PS1-5.

41

Review Aspects of the Item (Grade 8) (2)

Evaluate the Needs of the Item (Grade 8)

• The item aligns to the PE by requiring that a

student:

• Use knowledge of conservation of mass to

evaluate the model by using the sliders to indicate how many more particles to add to

• ne or both sides of the model.

42

Review Twinning Options (Grade 8)

• This scored simulation Custom interaction is

not accessible for students with visual impairments.

• Twinning to a different item type (e.g.,

multiple choice or grid or inline choice list) makes the content assessed in the item accessible.

• Twinning should maintain the construct and

as much of the rigor as possible.

43

Create the Twinned Item (Grade 8)

• Select the

correct answer among the four options.

44

Compare the Parent and Twin (Grade 8)

• Both items have students evaluate a model by

using his or her knowledge of the conservation of mass.

45

Future Research

• Validity evidence
• Prototyping twins

− Cognitive Lab Study spring 2018

• Opportunities to explore future

accessible design

46

California Science Test (CAST) Cognitive Lab Study for Students with Visual Impairments

Danielle Guzman-Orth, Cary Supalo, Teresa King Presentation for the National Conference on Student Assessment June 2018

7/25/2018

47

Overview (2)

• Purpose
• Context
• Research Questions
• Study Design
• Findings
• Implications
• Recommendations for Future Research

48

Overview of Cognitive Labs for CAST

• Participants are asked to complete a task.
• Participants are interviewed to elicit

explanations about the cognitive processes that he or she engages in while completing the task.

• Additional information may be elicited

(e.g., sources of ease, difficulty, confusion) depending on the study’s focus.

49

Purpose and Rationale for CAST Cognitive Labs

• Examine interactions to determine how

students with visual impairments access the tested content to show what they know and are able to do.

• New technology-enhanced item (TEI) types
• Twins
• Provide recommendations to improve students

with disabilities’ access to content.

• Item design
• Accommodations
• Directions for administration

50

Accessibility Defined

• The interaction between characteristics of

the test taker and the characteristics of the item (or overall test) (Beddow, 2012; Ketterlin-

Geller, 2008)

• Goal: Maintain reliability and validity of test

score inferences (Beddow, 2012; Beddow, Kurz,

& Fray, 2011; Ketterlin-Geller, 2008)

51

Accessibility Challenge (1)

• “Visual impairment including blindness

means an impairment in vision that, even with correction, adversely affects a child’s educational performance. The term includes both partial sight and blindness.” (Individuals with Disabilities Education Act, 2004)

• There is a functional difference between

visual impairment and total blindness.

• Students may have a limited understanding
• f the physical world.

52

Accessibility Challenge (2)

• Assessment design characteristics
• Animations
• Simulations
• Visual-based representations of concepts

53

Accessibility Challenge (3)

• Accommodation
• Familiarity
• Usefulness
• Limitations

54

Accessibility Opportunities

• Multifaceted accessibility solutions
• Combinations of support strategies to

promote access

• Example: Twinning

55

Twinning Process

(Guzman-Orth, Pooler, & Sova, in progress)

56

Item Type Action Steps Twin Type Create Twin Comparison

Research Questions

• 1. Do the selected TEIs introduce barriers to

allowing students with visual disabilities to respond appropriately to the items?

• 2. Which approaches to overcoming these

barriers show the most promise?

57

Participants

• Targeted sample
• Four students in grades five, eight, and high

school (grades ten, eleven, and twelve)

• All students must be pre-registered to receive

the braille form of the CAST field test

• Obtained sample
• Grade five = 3 students
• Grade eight = 1 student*
• High school = 4 students

58

Preliminary Participant Background Characteristics (1)

• Three students were female and five students

were male

• Two students were also English learners
• All eight students reported using hard copy

braille to access content

• All eight students reported familiarity reading

contracted Unified English Braille (UEB)

59

Preliminary Participant Background Characteristics (2)

• Four students regularly use assistive

technologies (e.g., BrailleNote)

• Two students use Job Access With Speech

(JAWS) to access content

• Two students reported preferring other screen

reading software to JAWS (VoiceOver, NonVisual Desktop Access [NVDA], ChromeVox)

60

Cognitive Lab Items

The following criteria was used to guide the selection

• f items for the cognitive lab forms:
• 1. A minimum of one twinned task type
• 2. A minimum of one PT
• 3. A minimum of one task type where the original

version included an animation/simulation

• 4. A minimum of one task type where the original

version included interactivity (i.e., student choice)

• 5. A minimum of one constructed response
• 6. A minimum of one tactile graphic

61

Cognitive Lab Forms

Task type Grade 5 Grade 8 High school

Discrete items 2 2 1 Items within performance tasks (PTs) 10 (2 PTs) 12 (2 PTs) 12 (2 PTs)

62

Data Collection Procedures

• Students take Segment A block with their test

administrator

• B and C blocks were the research component
• 1:1 observation and interview
• 60–90 minutes
• Data sources
• observation notes (student and test administrator)
• interview notes
• post-study survey
• log files
• audio recordings

63

Data Analysis

• Grounded theory
• Initial coding with closed codes (usability,

accommodation familiarity, technology)

• Detailed open coding (braille literacy,

graphics, assistive technology, TEIs)

64

Study Findings

Overall Findings: Braille literacy

• Braille code
• Standard braille symbols
• Presentation
• Interaction with content
• Initials, abbreviations (names)
• Specialized content (chemical compounds, scientific

symbols, mathematical symbols)

66

Overall Findings: Graphics

• Illustrations, tables, charts, grids
• Computer presentation
• Tactile presentation
• Interpretation
• Timing, exploration, clarity, comprehensibility

67

Overall Findings: Assistive Technology (AT)

• Availability, usability
• TEIs
• Perception
• Positive, encouraging

68

Study Recommendations

69

Recommendations: Braille literacy

• Consider how to work with the field of

teachers of the visually impaired (TVIs) to ensure that they introduce the standard braille conventions during the school year so that the braille transcriptions in the test are not new to the students

• Continue allowing for multiple modes of

presentation (refreshable braille, embossed

• n demand)

70

Recommendations: Graphics

• Continue using industry standards to transcribe

the test content

• Consider simplifying illustrations to increase the

clarity and comprehensibility

• Consider using “Keys” to help students orient to

multiple components in the illustrations

71

Recommendations: Assistive Technology (AT)

• Consider ways to work with the field of TVIs to

ensure that all students have access to, and

• pportunity to learn, standard

accommodations that are used for the testing platform before they sit down to take the test

72

Recommendations: TEIs

• Continue using item types that are the most

intuitive and present the fewest usability challenges for students with visual impairments, unless there is a justifiable rationale to do otherwise

73

Implications and Future Research (1)

• Overall, students enjoyed working

through test despite some usability challenges

• Opportunities exist for ongoing

collaboration with students and TVIs to learn how to support the goal of students having an opportunity to learn standard braille conventions and AT

74

Implications and Future Research (2)

• Opportunities for continued monitoring and

support to continue increasing the accessibility

• f the CAST while establishing a body of

validity evidence so that students with visual impairments have every chance to show what they know and are able to do

75

Thank you! For more information, please contact: dguzman-orth@ets.org

Multisensory Learning for Students Who Are Blind or Visually Impaired in the 21st Century

Cary Supalo Presentation for the National Conference on Student Assessment June 2018

7/25/2018

77

78

January 2011: Mark Riccobono drove on the road course section of the track at the Daytona International Speedway.1

79

August 2013: Dan Parker competed at the Bonneville time trials in Utah. He is a blind motorcyclist.2

80

In September

• f 2017,

Jake Olson made his debut as the University

• f Southern

California’s blind long

• snapper. 3

September 2017: Jake Olson made his debut. He is the University of Southern California’s blind long snapper.3

81

Pole vaulter Charlotte Brown currently attends Purdue University. She is legally blind.4

Inclusion is an increasing trend in spaces where it was

• nce thought

impossible.

82

Consider: Today’s blind and visually impaired (BVI) population expects mainstream, out-of-the-box accessibility.

• BVI populations can:
• watch TV, play music, and listen to

radio and audio-described movies

• get driving directions, navigate

surroundings, call for a ride, get real-time transit information, make airline/ train reservations

• send and receive text messages,

make phone calls, take photographs, read the printed word

• pay bills, shop online

83

The process of inclusion for BVI learners may require interventions.

• use of modified or new methodologies:
• e.g., Daniel Kish uses echolocation to ride a

bicycle.5

• use of support teams
• e.g., Scientists and engineers are designing

user interfaces for nonvisual access to driving.

• use of modified or new technologies
• e.g., Three-wheeled motorbike design helps to

maintain balance.

84

PhET, an award-winning free STEM simulation site, has many examples of interventions.5

85

Static electricity example: John Travoltage

86

Accessible John Travoltage

87

Accommodations Are Individualized to the Learner

88

Various Technologies to Facilitate Multimodal Access to Simulations (1)

• Tactile supports
• e.g., refreshable braille display technologies, using

dots 7–8 to show dynamic shifts in a visual process

• hands-on manipulations, haptics
• Auditory supports
• e.g., data sonification tools (audio graphing

scientific calculator)

• marking progress in a visual process with music

89

Various Technologies to Facilitate Multimodal Access to Simulations (2)

• Visual supports
• e.g., screen magnification software
• Navigation conventions
• e.g., CTRL+TAB, ALT+F4, ALT+W, INS
• gesture finger navigation
• Text-based support
• e.g., audio-described video
• careful descriptions of visual information that do not

give away the construct

90

Various Technologies to Facilitate Multimodal Access to Simulations (3)

91

• Graphiti Graphics Display6

Benefits with Accessible Science Simulations

• Dynamic presentation and interpretation of

data sets

• Information available in real time
• Equitable access to scientific information

92

Challenges with Accessible Science Simulations

• Additional R&D needed for development of

more accessible interfaces

• Efficacy studies
• Adoption of accessible interfaces in K–12

space

• Overabundance of authors publishing

inaccessible science simulations

• e.g., book publishers, science faculty, and

science teachers

93

Why Is All of This Important?

• Accessible science simulations leverage

multisensory science learning.

• Today’s teachers are challenged to make

teaching more interactive and engaging.

• Multimodal presentation is a driving vehicle to

that end.

94

Multisensory Interactions

Multisensory Interactions

Keyboard Navigation Sonification (i.e., audio graphics) Audio Description Physical Tactile Graphic Manipulatives

95

Bringing It All Together

96

Twinning Student

Comprehension

Reliable Access Technology 21st Century Digital Skills

In Summary:

• Item twinning is the current innovative

intervention to make inaccessible science test items (like simulations) accessible.

• This research documents its potential and

promise, and surfaces recommendations to promote more inclusive science assessments.

• In conjunction with current and future 21st

century digital technologies, this approach will lead to an increasingly inclusive science assessment landscape for all learners.

97

References (1)

• 1. Mark Anthony Riccobono's blind lap of Daytona. (2013,

August 8). Retrieved October 20, 2017, from https://www.autoblog.com/photos/mark-anthony-riccobonos- blind-lap-of-daytona/

• 2. Riccobono, M. (2013, October). The Blind Driver

Challenge™, the Quest for the Salt, and the Continued Journey of Innovation and Adventure. Retrieved October 20, 2017, from https://nfb.org/images/nfb/TheBlindDriverChallengeandtheQ uestfortheSalt

• 3. Blind USC football player on helping lead his team to victory:

'It was emotional'. (2017, September 22). Retrieved October 20, 2017, from http://abcnews.go.com/Lifestyle/blind-usc- football-player-helping-lead-team-victory/

98

References (2)

99

• 4. Blind high school pole vaulter Charlotte Brown wins

medal in Texas. (2015, May 16). Retrieved October 20, 2017, from http://www.nydailynews.com/sports/blind-high-school- pole-vaulter-wins-medal-texas

• 5. Blindness No Obstacle To Those With Sharp Ears.

(2011, March 13). Retrieved October 23, 2017, from http://www.npr.org/2011/human-echolocation-using- sound-to-see

• 6. PhET Interactive Simulations. (n.d.). Retrieved

October 20, 2017, from https://phet.colorado.edu/interactive-simulations

• 7. Graphiti Graphics Display. Retrieved May 11, 2018,

from http://www.aph.org/graphiti/product-information

Questions?

Contact: csupalo@ets.org

10