Texas Tackles the Creation of an Alternate English Language - - PowerPoint PPT Presentation

Equity for Special Populations: Texas Tackles the Creation of an Alternate English Language Proficiency Assessment

June 26, 2019 National Conference on Student Assessment Kim Brannan, Esmeralda Cavazos, Heather Roeters-Solano, Bob Schwartz, Martha Thurlow

Purpose and Agenda

This session will focus on assessment design and implementation for English learners with the most significant cognitive disabilities. We will look at the:

• background of the current general English language proficiency assessment, TELPAS
• design and creation of the TELPAS Alternate assessment
• teacher training needed to administer such an assessment with fidelity given the format

and the population of students being assessed

• validity and reliability evidence that can be gathered for this type of assessment
• national outlook on lessons learned and best practices for other states going through

this journey

2019 NCSA Texas Education Agency

2019 NCSA Texas Education Agency

Background Information

Texas English Language Proficiency Assessment System (TELPAS)

• Based on English Language Proficiency Standards (ELPS) adopted

by Texas State Board of Education in 2007

• First administered in 2007 with online-administered reading test

and holistically rated listening, speaking, and writing with extensive teacher training component

• Reborn in 2018 with online-administered reading, listening, and

machine-scored speaking tests; writing test still holistically rated based on collection of student work

2019 NCSA Texas Education Agency

ELPS and Proficiency Level Descriptors

• In Texas, the ELPS are published with the State-required curriculum for

all grades and content areas.

• There are three instructional components of the ELPS:

○ Cross-curricular second language acquisition essential knowledge and skills ○ Proficiency level descriptors (PLDs) defining characteristics of each proficiency level ○ Linguistic accommodations

• Teachers use the PLDs to interpret their English learners’ summative

TELPAS results and for ongoing formative assessment during the year.

2019 NCSA Texas Education Agency

Key Features of Each Proficiency Level

2019 NCSA Texas Education Agency

Beginning

Little or no English ability

Intermediate

Limited ability, simple language structures, high-frequency vocabulary, routine contexts

Advanced

Grade appropriate, with second language acquisition support

Advanced High

Grade appropriate, with minimal second language acquisition support

TELPAS by the Numbers

2019 NCSA Texas Education Agency

972,000 English learners eligible to take TELPAS 88,000 English learners eligible to take TELPAS receive special education services 4,600 English learners eligible to take TELPAS are receiving special education services and taking alternate assessment (2,100 not tested in all domains, 2,500 not tested in some domains)

Not Tested Due to ARD Discussion

• An Admission, Review, and Dismissal, or ARD, committee (i.e., IEP team),

in conjunction with the Language Proficiency Assessment Committee (LPAC), may determine that an English learner receiving special education services should not be assessed in one or more domains with TELPAS for reasons associated with the student's disability.

• Examples may include but are not limited to

○ A student who is deaf and does not participate in the listening or speaking domain assessments ○ A student who is blind and does not participate in the reading or writing domain assessments

2019 NCSA Texas Education Agency

Theory of Action, the “If”

2019 NCSA Texas Education Agency

the agency creates a holistic,

• bservational

inventory to assess the English language proficiency of English learners with significant cognitive disabilities in grades 2-12, and the agency communicates TELPAS Alternate policy and administration information in a timely manner so that regions, districts, and campuses can appropriately train test administrators, and the agency supports cross- training

• pportunities

and professional development by providing tools and resources to support special education teachers, bilingual/ESL teachers, administrators, and parents, and the agency provides timely and relevant data regarding the performance

• f English

learners with significant cognitive disabilities on TELPAS Alternate, and the agency develops entrance and exit procedures appropriate for this population

IF IF IF IF IF

Theory of Action, the “Then”

2019 NCSA Texas Education Agency

teachers, administrators, and parents will be equipped with meaningful data to inform instruction for ELs with SCD as they progress through the stages

• f language acquisition and

proficiency at school and at home, which will result in the demonstration of growth in English language acquisition such that it limits the barrier to academic achievement for ELs with SCD.

THEN

Guiding Principles for TELPAS Alternate Development

2019 NCSA Texas Education Agency

Limit additional burden on the field Accessible to student population Familiar form of evaluation used by special educators

What is TELPAS Alternate?

• A holistic inventory that assesses the language domains of listening, speaking,

reading, and writing for students with significant cognitive disabilities in grades 2-12

• Aligned to the Texas ELPS
• Based on alternate PLDs created to address the specific access needs of this

population

2019 NCSA Texas Education Agency

2019 NCSA Texas Education Agency

TELPAS Alternate Development

Development Timeline

2019 NCSA Texas Education Agency

TEA presented information on AELPA requirement to Advisory Committee, collected feedback August 2017 Advisory Committee reviewed draft materials and provided feedback, determined assessable ELPS November 2017 TEA and Pearson conducted cognitive lab and pilot Spring 2018 TEA posted operational Observable Behaviors, classroom examples, and training materials October–December 2018 TEA sought guidance from TTAC

• n approach for

AELPA development September 2017 Educator Committee assisted in drafting of Observable Behaviors January 2018 Educator Committee wrote classroom examples for Observable Behaviors September 2018

State Definition

Students taking TELPAS Alternate are English learners in grades 2-12 who have significant cognitive disabilities and who are in the process of acquiring English proficiency in listening, speaking, reading, and writing. These students have one or more disabilities that significantly limit their intellectual functioning, as shown by their ability to plan, comprehend, and reason, and their adaptive behavior, as shown by their ability to apply social and practical skills.

2019 NCSA Texas Education Agency

Kindergarten-Grade 1

• Based on recommendation from educator advisory committee, English

learners with significant cognitive disabilities in grades K–1 will take the TELPAS K–1 holistically rated assessment in all four language domains. No eligibility determination is needed.

• Why not TELPAS Alternate?

○ Some students at K-1 have no identified disability yet (i.e., non- categorical) ○ Reluctance to assign labels to young students that would make them eligible for an alternate assessment ○ Students can access the general holistically rated assessment

2019 NCSA Texas Education Agency

Participation Requirements: Grades 2-12

• For grades 3-12, a student must be an EL and

eligible for the State’s academic alternate assessment.

• For grade 2, the student must meet 6

eligibility criteria to be eligible. ○ The student is identified as an EL. ○ The student has a significant cognitive disability as determined by the ARD using a full and individual evaluation. ○ The student requires specialized, extensive supports for instruction. ○ The student requires intensive, individualized instruction. ○ The student accesses grade-level curriculum through prerequisite skills. ○ The decision is based solely on the disability and EL status and not on any extenuating factors.

2019 NCSA Texas Education Agency

Language Domain Definitions

2019 NCSA Texas Education Agency

Domain TELPAS Definition TELPAS Alternate Definition

Listening The ability to understand spoken language, comprehend and extract information, and follow social and instructional discourse through which information is provided. The ability to understand spoken or signed language, comprehend and extract information, and follow social and instructional discourse through which information is provided. Speaking The ability to use spoken language appropriately and effectively in learning activities and social interactions. The ability to use spoken language or alternative communication appropriately and effectively in learning activities and social interactions. Reading The ability to comprehend and interpret written text at the grade-appropriate level. The ability to comprehend and interpret written text, including braille, at a modified level. Writing The ability to produce written text with content and format to fulfill grade- appropriate classroom assignments. The ability to produce written text or alternative communication with content and format to fulfill classroom and community-based assignments.

TELPAS Alternate Proficiency Level Labels

2019 NCSA Texas Education Agency

Awareness Imitation Early Independence Developing Independence Basic Fluency Beginning Intermediate Advanced Advanced High

TELPAS Alternate proficiency levels TELPAS proficiency levels

TELPAS Alternate Proficiency Level Labels and Global Definitions

2019 NCSA Texas Education Agency

Proficiency Level Label Global Definition Awareness

Students who receive this rating may be aware of English sounds or print; however, they have little or no functional ability to participate in communication activities in English.

Imitation

Students who receive this rating match, imitate, or approximate some English in their environment; however, they are not able to independently understand or produce English. They participate in routine communication activities in a familiar environment when the activities are significantly linguistically accommodated.

Early Independence

Students who receive this rating understand short, simple messages and produce messages of one or two high-need, high-frequency words (e.g., book, cafeteria, teacher). They are starting to participate in linguistically accommodated communication activities in English in familiar environments.

Developing Independence

Students who receive this rating understand longer messages of multiple sentences in English and produce simple, descriptive, original messages by combining two or more words (e.g., new red bike, big fast truck). They participate meaningfully in linguistically accommodated communication activities in English in familiar environments.

Basic Fluency

Students who receive this rating understand and produce more detailed, complex, and elaborate messages with multiple sentences in English. These students participate independently in communication activities in English in familiar environments.

Alternate Proficiency Level Descriptors

• Lists major characteristics that

define how well English learners at each proficiency level can understand and use English in social and academic settings for all 4 domains

• Shows progression of second

language acquisition across proficiency levels

• Serves as a road map to help

teachers tailor instruction to linguistic needs of English learners with significant cognitive disabilities

• Provides a common framework

for understanding language acquisition skills needed to be classified within each proficiency level

2019 NCSA Texas Education Agency

Assessable ELPS

A committee of Texas educators analyzed the Texas ELPS to determine those which would be eligible for inclusion on an alternate ELP assessment. An example from the listening standards is shown below:

2019 NCSA Texas Education Agency

Notice the text determined to be ineligible contains language about grade- appropriate levels and/or classroom activities that would not be appropriate for summative assessment.

TELPAS Alternate Test Blueprint Development

• TELPAS organizes the majority of the ELPS student expectations under

several reporting categories, or skill areas, on the test blueprints.

• These student expectations that were determined by an educator

committee to be assessable were organized under the same reporting categories as TELPAS; however, the blueprints are not identical.

○ TELPAS contains more test questions than TELPAS Alternate. ○ Some student expectations were deemed not accessible and therefore not assessable for students with significant cognitive disabilities.

• TELPAS Alternate test blueprints illustrate the number of Observable

Behaviors assessed in each reporting category with a total test length of 10 Observable Behaviors per domain.

2019 NCSA Texas Education Agency

TELPAS Alternate Test Blueprints

All blueprints can be found at: https://tea.texas.gov/student.assessment/telpasalt/

2019 NCSA Texas Education Agency

What are Observable Behaviors?

• The “questions” the test administrator answers about a student.
• Measures one skill that is aligned to the ELPS.
• The skill can be found on the left. The boxes contain descriptions of

characteristics that students learning English are likely to demonstrate over time. The descriptors show the progression of second language acquisition from one proficiency level to the next and are aligned to alternate PLDs.

2019 NCSA Texas Education Agency

Observable Behaviors

• Texas educators helped to draft the Observable Behaviors.
• The TELPAS Alternate instrument contains 10 Observable Behaviors for

each language domain.

• Test administrators, based on their knowledge of a student’s English

language skills over a period of time, mark the level that best describes the student at the time of the assessment. A determination will be made for each of the 10 Observable Behaviors in every domain.

2019 NCSA Texas Education Agency

TELPAS Alternate Pilot Test and Cognitive Lab

• Pilot Test

○ Administered over a 10-day window from April 23-May 4, 2018, to students in grades 2 through 12 who were eligible to participate ○ A preliminary study intended to evaluate the effectiveness and feasibility of this new assessment ○ Approximately 2,112 students tested

• Cognitive Lab

○ Conducted April 30-May 4, 2018 ○ Visited 20 school districts across the state and interviewed 41 teachers who used the instrument with 101 students

2019 NCSA Texas Education Agency

Pilot Results

2019 NCSA Texas Education Agency

Texas educators administered the TELPAS Alternate Assessment to 2,112 students in grades 2 through 12. All assessed students participated in limited English proficient (LEP) and special education programs.

Pilot Results

2019 NCSA Texas Education Agency

Table 1 presents the rates of occurrence of repeated rating patterns within each domain. For example, 5 percent of the Listening ratings had the pattern “1111111111.” All other non-repeating rating patterns made up 88 percent of the rating patterns for Listening.

Pilot Results

2019 NCSA Texas Education Agency

Raw score distributions by domain:

• Listening (upper left)
• Reading (upper right)
• Speaking (lower left)
• and Writing (lower right)

Pilot Results

• High reliability (coefficient alpha) on all 4 domains
• Very high pt-polyserial values for all items
• No items showed DIF between

○ Male/Female ○ Hispanic/non-Hispanic

2019 NCSA Texas Education Agency

Listening Speaking Reading Writing .97 .98 .98 .97

Cognitive Lab Feedback: Teachers Wanted...

2019 NCSA Texas Education Agency

help interpreting some of the language used in the Observable Behaviors a version of the Observable Behaviors to have note- taking space additional training on classroom activities that aligned with the Observable Behaviors materials to be posted in the fall semester so they could have time to become familiar with the instrument and incorporate appropriate activities into lessons to continue administering TELPAS to students in grades K-1

Observable Behaviors with Classroom Examples

2019 NCSA Texas Education Agency

• Texas teachers developed classroom examples to help test administrators better

understand the descriptions of student performance for each Observable Behavior.

• Elementary and secondary examples describe one way that students could

demonstrate each skill across the five levels of proficiency.

All Observable Behaviors can be found at: https://tea.texas.gov/student.assessment/telpasalt/

Using the Classroom Examples

• There are many other classroom activities that could be used as

examples for the Observable Behaviors.

• These examples are not intended to be used as test questions or

performance tasks for teachers to replicate, although using them for this purpose is acceptable if needed.

• Teachers are encouraged to use their own activities in the regular

classroom setting when determining a student’s ability to understand and use English.

2019 NCSA Texas Education Agency

TELPAS Alternate Training Resources

2019 NCSA Texas Education Agency

Resource Purpose Audience

Educator Guide to TELPAS and TELPAS Alternate Provides an overview of TELPAS and TELPAS Alternate and serves to support effective implementation of the ELPS Administrators, Coordinators, Teachers Participation Requirements Used by ARD committees in conjunction with the LPAC to make decisions about TELPAS Alternate Members of ARD committees and LPACs Observable Behaviors Measures the student’s use of English and contain a notes section that can be used to become accustomed to TELPAS Alternate prior to the assessment window Teachers

TELPAS Alternate Training Resources

2019 NCSA Texas Education Agency

Resource Purpose Audience

Parent Brochure Communicates basic information about TELPAS Alternate to parents Parents Test Administration Manual Contains instructions covering the responsibilities of test administrators and the Observable Behaviors used to assess students Required to be read carefully and followed as written Administrators, Coordinators, Teachers serving as test administrators Training PowerPoints (see list on next slide) Provides training on a variety of topics, including authentic classroom activities for each domain that explain how to rate students with the Observable Behaviors Designed as short PowerPoints that can be viewed in 30 minutes or less Are optional though highly recommended Administrators, Coordinators, Teachers serving as test administrators

TELPAS Alternate Training PowerPoints

• Introduction to TELPAS Alternate
• TELPAS Alternate Student Eligibility
• TELPAS Alternate Accessibility
• TELPAS Alternate Listening Domain
• TELPAS Alternate Speaking Domain
• TELPAS Alternate Reading Domain
• TELPAS Alternate Writing Domain
• TELPAS Alternate Test Administration

2019 NCSA Texas Education Agency

2019 NCSA Texas Education Agency

TELPAS Alternate Administration

TELPAS Alternate Administration Timeline

2019 NCSA Texas Education Agency

TELPAS Alternate materials and trainings posted, TETN presentations Fall 2018 Required TELPAS Alternate training sessions January–February 2019 TELPAS Alternate administration window February 25– April 5, 2019 Final TELPAS Alternate reports posted August 16, 2019 Districts receive TELPAS Alternate manuals and materials January 7–11, 2019 TELPAS Alternate webinar February 1, 2019 TELPAS Alternate standard setting June 10–12, 2019

Response Modes

• Some English learners may use sign

language, braille, or another method of communication instead of traditional English in one or more domains.

• Test administrators allow students to

use one or more alternate response modes if the students regularly use the response mode(s) during instruction and in accordance with the Individualized Education Program (IEP).

• Alternate response modes are only

intended for students who cannot listen, speak, read, or write in a traditional way. They are intended to address the communication needs of students based

• n their disability.

2019 NCSA Texas Education Agency

Alternate Response Mode Examples

• Gaze at
• Touch
• Gesture toward
• Form responses using a

communication device

• Use adaptive writing

equipment

• Nod
• Draw
• Picture cards and boards

2019 NCSA Texas Education Agency

“Do Not Score” Designations

• If a student does not participate in TELPAS Alternate, a “do not score”

designation must be documented. It will apply to all domains.

• “Do not score” designations:

○ M = Medical Exception ○ N = No Authentic Academic Response (NAAR) ○ E = Extenuating Circumstances

2019 NCSA Texas Education Agency

“Do Not Score” Designation - Medical Exception

• In rare situations, a student may be unable to participate meaningfully in

TELPAS Alternate on the basis of the student’s medical condition, as determined by the ARD committee and documented in the student’s IEP.

• Students who are medically fragile and cannot attend to or tolerate any

academic interaction can qualify for a medical exception for the following circumstances:

○ The student is in the final stages of a chronic, terminal, or degenerative illness. ○ The student is receiving extensive short-term medical treatment due to a medical emergency or serious injury in an accident. ○ The student is unable to interact with peers or staff without risk of infection or contamination to him/herself or others. ○ The student is unable to receive sufficient or consistent homebound services due to medical issues.

2019 NCSA Texas Education Agency

“Do Not Score” Designation – No Authentic Academic Response

• In rare situations, a student may be unable to participate meaningfully in

the TELPAS Alternate assessment on the basis of the student’s disability, resulting in the inability to make an authentic academic response to stimuli, as determined by the ARD committee and documented in the student’s IEP.

• Students who are not able to respond authentically to any verbal, visual,
• r tactile stimuli during academic instruction due to level of cognition

rather than a medical condition can qualify for this exception if:

○ The student is unable to demonstrate any observable reaction to a specific stimulus. ○ The student exhibits only startle responses. ○ The student tracks or fixates on objects at random and not for a purpose. ○ The student moves or responds only to internal stimuli. ○ The student vocalizes intermittently regardless of changes in the environment around him or her.

2019 NCSA Texas Education Agency

“Do Not Score” Designation - Extenuating Circumstances

• In rare situations, a student may not begin or finish TELPAS Alternate

during the testing window for reasons other than those listed for medical exception and No Authentic Academic Response.

• Extenuating circumstances may include the following; however, this list

is not exhaustive. ○

The student withdraws during the testing window prior to completing the assessment.

○

The student enrolls in the school from another state or country, from a private school, etc. and the test administrator does not have time, nor enough information to administer the assessment.

○

The student was not able to finish the assessment due to factors such as illness, injury, hospitalization, etc.

2019 NCSA Texas Education Agency

2019 TELPAS Alternate Operational Administration Numbers

2019 NCSA Texas Education Agency

• 6,465 total students

○ 6,341 complete scored tests for all domains

• 64% male and 36%

female

• 89% Hispanic and 6%

Asian

• 88% economically

disadvantaged

2019 Operational Administration: Raw Score Distribution by Domain

2019 NCSA Texas Education Agency

Standard Setting: June 10-12, 2019

• Standards set for all 4 domains and rules established for reporting
• verall TELPAS Alternate Composite Rating
• Used test-centered, criterion-referenced method to guide and support

panelists as they made their cut score recommendations

○ Extended Modified Yes/No Angoff method ○ Similar methodology to what was used for STAAR Alternate 2 standard setting in 2015 and TELPAS standard setting in 2018

• Texas educators brought content knowledge and experience with English

learners with significant cognitive disabilities to the standard setting process.

○ Panelists included blingual/ESL educators and special educators with expertise in low incidence disabilities at the campus, district, and regional level

2019 NCSA Texas Education Agency

Standard Setting Results: Recommended Cut Scores

2019 NCSA Texas Education Agency

Domain

Proficiency Level

Imitation Early Independence Developing Independence Basic Fluency Listening 17 26 36 45 Speaking 16 26 35 44 Reading 18 24 33 42 Writing 16 24 33 41 Note: The range of scores on each domain-specific assessment is from 10 to 50, inclusive.

Standard Setting Results

2019 NCSA Texas Education Agency

Validity and Reliability

• 2019 TELPAS Alternate Correlations between domains

2019 NCSA Texas Education Agency

Listening Speaking Reading Speaking 0.88 Reading 0.87 0.81 Writing 0.82 0.78 0.92

• 2019 Cronbach’s alpha reliability

Listening Speaking Reading Writing 0.97 0.98 0.98 0.97

Validity and Reliability

• Validity studies between TELPAS Alternate and STAAR Alternate 2

Reading/Writing/English tests

○ Higher mean STAAR Alternate scale scores and higher pass rates/performance-level classifications as TELPAS Alternate proficiency- level classifications increased

• Feasibility of an Interrater Reliability study was explored.

2019 NCSA Texas Education Agency

2019 NCSA Texas Education Agency

Lessons Learned

TELPAS Alternate Survey

• Administered throughout TELPAS Alternate window

○ 488 respondents

■ 86% special education teachers ■ 3% bilingual/ESL teachers ■ 10% testing coordinators ■ 1% other roles

2019 NCSA Texas Education Agency

TELPAS Alternate Survey: Primary Disability Categories

2019 NCSA Texas Education Agency

TELPAS Alternate Survey: Bilingual/ESL Services

2019 NCSA Texas Education Agency

• “My students are nonverbal and untestable.”
• “I have a bilingual teaching assistant.”
• Parent denials
• “Bilingual/ESL services are offered at another campus.”
• “Students are not eligible for bilingual/ESL services.”
• Student already receives speech and special education services and does not

need bilingual/ESL services as well.”

TELPAS Alternate Survey: Collaboration

2019 NCSA Texas Education Agency

• Only about 20% of test administrators collaborated with general education or

bilingual/ESL educators.

• The remaining 24% collaborated with paraprofessionals or other special

education teachers.

TELPAS Alternate Survey: Change in Services Because of New Assessment

2019 NCSA Texas Education Agency

• Almost 10% of teachers reported that students began receiving bilingual/ESL

services because of TELPAS Alternate.

TELPAS Alternate Survey: Improved Communication Because of New Assessment

2019 NCSA Texas Education Agency

• About 50% of teachers reported an increase in communication because of

TELPAS Alternate.

TELPAS Alternate Survey: Additional Opportunities

2019 NCSA Texas Education Agency

• Almost 40% of teachers reported they had additional opportunities for ESL

training.

Outstanding Questions for Future Administrations

2019 NCSA Texas Education Agency

• Holistic inventory versus standardized, item-based

assessment

• What to do about K-1
• How to communicate bilingual/ESL requirements for

ALL English learners

• Continue to make administration-specific trainings
• ptional or required
• Continue to force test administrators to score or not

score for all domains or make domain specific

• How to expand communication about reclassification

requirements for students with significant cognitive disabilities

• Keep testing window the same as TELPAS or move to

different time

Take Aways from Year One

2019 NCSA Texas Education Agency

• This is the best assessment for this particular moment in time to
• pen the conversation between special education and English

learner specialists.

• Equity and access for English learners with significant cognitive

disabilities is paramount in this endeavor.

• Collaboration between experts in the field, both locally and

nationally, is critical when creating and revising the assessment.

• There is always room for improvement.

2019 NCSA Texas Education Agency

National Perspective

2019 NCSA Texas Education Agency

TEA’s Student Assessment Division 512-463-9536 assessment.specialpopulations@tea.te xas.gov TELPAS Alternate Resources https://tea.texas.gov/student.assessme nt/telpasalt/ Technical Digest https://tea.texas.gov/student.assessme nt/techdigest/

Photograph Disclaimer

2019 NCSA Texas Education Agency

Most of the students shown in the photographs in this presentation are Texas students. Only TEA has parental permission to use these photographs for training purposes.