When Students Copy: Examining state policies, practices, and - - PowerPoint PPT Presentation

When Students Copy:

Examining state policies, practices, and consequences of copied text in evidence based writing NCSA 2018

1

Presenters and Moderator

Sally Rhodes (Presenter) English Language Arts Coordinator, Test Development Center, Florida Department of Education Jan Reyes Ed.D. (Presenter) Director of Assessment Development, Georgia Department of Education Diane Simaska/Rebekah Baum-Leaman Ed.D. (Presenters) Education Associates, Division of Instructional Quality, Pennsylvania Department of Education Shiloh Cox-Fouch (Presenter) Senior Director Performance Assessment Services, DRC Holly Baker Ph.D. (Moderator) Vice President Education Solutions, DRC

Context and Goals

• Copied text versus plagiarism

– Using provided sources to support students original words and ideas

• Requires a full set of critical reading skills, not just

comprehension

– Characterized as an “inability to read well enough to understand, analyze, synthesize, and evaluate sources and then use those sources in support of an argument.” Horning, 2011

• Goals

– Explore state policies for identifying copied text, collecting information on copied text and reporting – Discuss state initiatives to improve writing instruction to address copied text – Discuss lessons learned and ideas for future development

3

www.FLDOE.org

4

Florida’s Observations

• f Copied Text

Florida Standards Assessment English Language Arts Sally Rhodes FSA ELA Coordinator

www.FLDOE.org

5

Florida Standards Assessment English Language Arts

FSA ELA is administered in grades 3-10 with approximately 200,000 students per grade.

• Reading Component – two sessions
• Writing Component –
• Grades 4 – 10
• Text-based writing prompt
• Opinion/Argumentation or Informational
• One session – 120 minutes
• Students receive a combined ELA scaled score for

reading and writing.

www.FLDOE.org

6

FSA Writing – Rubric Domains

• Clear claim/controlling idea
• Effective organization
• Logical progression of ideas

Purpose, Focus, Organization (4)

• Integrated, relevant evidence from sources
• Thorough, convincing support through elaboration
• Effective expression of ideas

Evidence and Elaboration (4)

• Adequate command of basic conventions

Conventions (2)

www.FLDOE.org

7

What Constitutes a “Copy” Score?

To qualify as scorable, a student must give a controlling idea, a reason, and “a little bit more.”

• The response demonstrates no original writing.
• Often the introduction is the prompt followed by a

three-pronged thesis of copied phrases from text.

• Each paragraph may have transition words, but

followed only by direct copy from texts.

• A few words may be substituted but usually only

basic synonyms are used (i.e., “wrong” used instead of “incorrect”).

www.FLDOE.org

8

Condition Code - Copy

• Between 1% and 9% of operational responses per

grade have been coded as copy since 2015.

• Student receives 0 of possible 10 raw score points
• n writing portion of assessment.
• Factors into overall FSA ELA score
• Students scoring 0 on writing rarely receive a

satisfactory Level 3 score (out of 5 levels).

• Reported out to students/schools/districts.

www.FLDOE.org

9

Percentage of Responses Receiving Copied Text Condition Code

1 2 3 4 5 6 7 8 9 10 2015 2016 2017 2018 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

www.FLDOE.org

10

Mode Effects – Informational vs Argumentation

1 2 3 4 5 6 7 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

% of responses coded as copied text Writing by Grade and Mode

Informational Argumentation

www.FLDOE.org

11

Approach to Handscoring

• Train scorers with a training set including multiple

examples of copied text and an example of a “barely scoreable” response.

• Scorers do many read-throughs of text, including in

small group discussions.

• A response must go through a minimum of three

levels of review before any condition code can be applied.

• Keep a file of “decision papers” to be used from

year to year for consistency.

www.FLDOE.org

12

Automated Scoring Engine

• Grades 8 and 9 are given one score from an

automated scoring engine and one human score.

• The engine is trained to pick up on several

dimensions including prompt copy, duplicate text, not enough text, etc.

• If identified as problematic by the engine, a

response automatically receives two human scores.

www.FLDOE.org

13

Steps Taken by FDOE

• Provided an example of an annotated “copied text”

condition code in Writing Samplers posted online.

• Emphasized the importance of analysis and

synthesis of texts as related to writing prompts during statewide meetings.

• Enhanced student score reports to show scores by

domain and define condition codes. https://fsassessments.org/resources/

www.FLDOE.org

14

Lessons Learned

In future writing development

• Prompts should encourage students to analyze and

synthesize information from multiple sources.

• Make purpose crystal clear –

argumentation/opinion, or expository/informational.

• Concept must be appropriate for the grade-level

Florida student with clear connection between the topic, text set, and prompt.

• Avoid overly technical articles/texts.

www.FLDOE.org

15

Questions?

Sally Rhodes FSA ELA Coordinator sally.rhodes@fldoe.org (850) 922-2584 ext. 2228

Richard Woods, Georgia’s School Superintendent

“Educating Georgia’s Future” gadoe.org

Georgia’s Experience wi with th Co Copi pied ed Text xt

Jan Reyes, Ed.D. Director of Assessment Development

7/6/2018 16

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Hi Hist stor

• ry

y of

• f Writ

iting ing Ass ssess ssment ent in in Geor

• rgia

gia

• Evidence-based writing was introduced into

Georgia’s student assessment program in the 2014-2015 school year with the transition to the Georgia Milestones Assessment System.

• Prior to Georgia Milestones, writing was assessed

through separate performance-based writing assessments in Grades 3, 5, 8, and 11, as required by Georgia law.

7/6/2018 17

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Former rmer Writ iting ing Ass ssessment essment Prog

• gram

am

• Student writing was assessed in four domains:
• Ideas (40%)
• Organization (20%)
• Style (20%)
• Conventions (20%)
• Performance Levels:
• Does not meet
• Meets
• Exceeds
• A performance level of Meets or Exceeds on the

Georgia High School Writing Test was a requirement for high school graduation.

7/6/2018 18

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Former rmer Writ iting ing Ass ssessment essment Prog

• gram

am

Grade Level Types of Writing Assessed Type of Assessment Time Allowed 3 Narrative Informational Persuasive Teacher evaluation; includes all types of writing Year-long 5 Narrative Informational Persuasive Response to assigned prompt for one type of writing 120 minutes 8 Expository Persuasive Response to assigned prompt for one type of writing 100 minutes 11 Persuasive Response to assigned prompt 100 minutes 7/6/2018 19

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Current nt Writ iting ing Ass ssessment essment

• Georgia Milestones is a comprehensive summative

assessment program spanning grades 3 through high school.

• Students in grades 3 through 8 take end-of-grade

(EOG) assessments and high school students take end-of-course (EOC) assessments.

• Georgia Milestones measures how well students

have learned the knowledge and skills outlined in the state-adopted content standards in English language arts, mathematics, science, and social studies.

7/6/2018 20

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Geor eorgia gia Mile ilestones stones

• Features of the Georgia Milestones Assessment

System include:

• constructed-response items in English Language Arts and

mathematics (all grades and courses);

• a writing component (in response to text) at every grade

level and course within the English Language Arts assessment;

• norm-referenced items in all content areas and courses, to

complement the criterion-referenced information and to provide a national comparison; and

• transition to online administration over time.

7/6/2018 21

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

En End-of

• f-Gr

Grade ade EL ELA A Ass ssessments essments

• Georgia Milestones English Language Arts (ELA) End-of-

Grade (EOG) assessments measure student achievement in grades 3 through 8.

• Georgia law mandates that student achievement in

reading, as measured as a component of the ELA EOG assessment, be utilized in promotion and retention decisions for students in grades 3, 5, and 8.

• Students who fail to demonstrate grade-level

achievement on these measures must receive remediation and be offered an opportunity for a retest prior to consideration for promotion to the next grade.

7/6/2018 22

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

En End-of

• f-Cour

Course se EL ELA A Ass ssessments essments

• Georgia Milestones End-of-Course (EOC)

assessments for English language arts measure student achievement in the following courses: Ninth Grade Literature and Composition and American Literature and Composition.

• The EOC assessment counts as 20% of a student’s

final grade in the course.

7/6/2018 23

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Reading eading & E & Evi vidence dence- Based ased Writi iting ng

• The extended writing prompt appears in the

Reading & Evidence-Based Writing section of the ELA assessment (Section 1).

• This section consists of:
• Two passages
• Three selected-response questions
• One short constructed-response question
• One extended writing prompt (either

informational/explanatory OR opinion/argumentative)

• Students are allowed 90 minutes to complete this

section of the test.

7/6/2018 24

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Na Narrati tive Writ iting ing

• The narrative writing prompt is embedded within

either Section 2 or Section 3 of the ELA assessment.

• Each of these sections consist of about 26

questions, and students are allowed 60-75 minutes to complete each section.

• Students develop a narrative response based on a

text as stimulus.

7/6/2018 25

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Writ iting ing Rubrics brics

Informational/ Explanatory

• Idea Development,

Organization, and Cohesion (4 points)

• Language Usage and

Conventions (3 points)

• Any Condition Code

applies to both traits

Opinion/ Argumentative

• Idea Development,

Organization, and Cohesion (4 points)

• Language Usage and

Conventions (3 points)

• Any Condition Code

applies to both traits

Narrative

• One trait
• 4 points
• Errors in usage and/or

conventions impact a student’s score only to the extent that they interfere with meaning 7/6/2018 26

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

ELA A Poin int t Dis istrib tributio ution

7/6/2018 27 # Item Type # Points 23 Selected Response 23 1 Evidence-Based Selected Response 2 2 Constructed Response (2 points each) 4 Total 29

Reading and Vocabulary

Writing and Languag

Writing and Language

# Item Type # Points 15 Selected Response 15 1 Extended Constructed Response Narrative 4 1 Extended Writing Prompt 7 Total 26

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Condition dition Codes des (N (Non-Scor Score) e)

7/6/2018 28

Non-Score (Code) Non-Score (Code) Description B Blank C Copied I Too Limited to Score F Non-English/Foreign Language T Off Topic/Off Task U Unreadable/Illegible/Incomprehensible S Offensive

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Georgia’s Scoring Ph Phil ilos

• sophy
• phy for
• r Co

Copied pied Text xt

• Condition codes are used only for extended writing

items.

• The nonscore/condition code C, for “copied,”

should only be applied if the entire response consists of text copied from the passage and/or prompt.

• If a response consists of a mix of original student

work and copied text, the original student work is scored.

7/6/2018 29

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Georgia’s Scoring Ph Phil ilos

• sophy
• phy for
• r Co

Copied pied Text xt

• For constructed-response items aligned to reading

comprehension standards, it is possible that a copied portion of the text could answer some part

• f the question, in which case the response could

receive a score.

7/6/2018 30

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Average Percent of “C” Co Cond ndition ition Co Codes es

201 016-2018 2018

Grade/Course Informational/ Explanatory Opinion/ Argumentative Narrative Grade 3 7.0 3.7 8.4 Grade 4 5.0 1.8 4.4 Grade 5 1.3 2.3 3.8 Grade 6 1.7 3.7 4.4 Grade 7 2.5 1.8 1.8 Grade 8 2.0 1.7 2.6 Ninth Grade Literature 1.8 2.0 5.0 American Literature 1.0 0.3 2.0

7/6/2018 31

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Per ercent cent of

• f Res

esponses ponses Rece eceiv iving ing Co Copied ied Text xt Con

• ndition

dition Cod

• de

7/6/2018 32

1 2 3 4 5 6 7 8 9

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 9th Grade Literature American Literature Informational/ Explanatory Opinion/ Argumentative Narrative

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Lesson essons s Learned earned

• The wording of extended writing prompt must be

clear and not leading for students (especially

• pinion/argumentative prompts).
• The purpose of narrative prompt must be clear.
• The short constructed-response item that precedes

the extended writing prompt must provide scaffolding, but not be too similar to the prompt itself.

7/6/2018 33

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Original iginal Pr Prompt mpt

In the world of athletics, some people believe that stretching before an activity is extremely important. Others, though, disagree that it should be done. Write an essay arguing whether stretching has a positive or negative effect on athletes. Think about both sides of the discussion, and then write an argumentative essay supporting either side. In your essay, you will convince the reader whether stretching has a positive or negative effect on athletes.

7/6/2018 34

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Rewor

• rded

ded Pr Prompt

• mpt

In the world of athletics, there is much debate about whether stretching before an activity is beneficial. Consider the claims on BOTH sides of the debate. Then write an argumentative essay in your own words arguing whether stretching has a positive or negative effect on athletes.

7/6/2018 35

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Na Narrati tive Writ iting ing

• Reports from the field indicate less improvement in

narrative writing compared to other writing types.

• Narrative writing in response to text is new to

Georgia students.

• Georgia plans to increase development of narrative

writing items to allow more flexibility in test form construction.

• A narrative focus group will be held in conjunction

with Item Review by Georgia educators to inform future narrative item development as well as creation of additional resources.

7/6/2018 36

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Resou esources es

• Assessment & Study Guides
• Structure and content of the assessment
• Sample items and exemplar responses
• Item Samplers
• Released items, including short constructed-response

items, narrative writing items, and extended writing items

• Scored and annotated student responses
• Professional Learning

37 7/6/2018

Richard Woods Georgia’s School Superintendent “Educating Georgia’s Future” gadoe.org

Questions? estions?

Jan Reyes, Ed.D. Director of Assessment Development Georgia Department of Education jreyes@doe.k12.ga.us 404-463-6665

38 7/6/2018

When Students Copy: Examining State Policies, Practices, and Consequences of Copied Text in Evidence Based Writing

PENNSYLVANIA WRITING ASSESSMENT HISTORY

NEW TEST DESIGN 2018

PENNSYLVANIA COPIED TEXT CODES

C – Copied item/directions Responses consisting of text copied from the item and/or test directions. N – Non scoreable For short answers and TDAs: responses are completely copied from first to last word or merely rearranged.

HAND SCORING

Grade 3 short answer

• Holistic 3 point scale
• Reading comprehension

Grades 4-8 text dependent analysis essay

• Holistic 4 point scale—weighted 4x
• Reading comprehension, analysis, essay writing

Copy verbatim from reading passage

• Permitted as text evidence
• Citation/punctuation considered part of holistic score
• Mostly copied text but relevant to prompt receives a 1

Copy paste

• Test engine locks down student’s device to prevent copy/paste

Write an essay analyzing how the author’s organization of the passage helps the reader to understand the tasks. Use information from the passage to support your response.

The other puts evry idea in there to do furst. “Be sure to ask an adult for permission before staring this project.”

Grade 4 Score Point 1 -- Student Response

AVERAGE 2015-2018

TEXT DEPENDENT ANALYSIS PROMPT

Change over time

• Move to ELA PA Core Standards
• More rigorous curriculum, instruction and assessment

Created new test design

• Initiated ongoing research with Center for Assessment
• Field tested TDA test items for two years prior to operational
• Developed and revised scoring guidelines

IMPACT OF COPIED TEXT

Challenge to teaching and learning

• Components of TDA
• Curriculum

Professional development

• Learning Progression
• Proof of Concept Study
• Regional consultants
• District leadership

ANALYTICAL WRITING AS A MODE OF DISCOURSE

CONTACTS

Diane A. Simaska Bureau of Curriculum, Assessment and Instruction Division of Instructional Quality Pennsylvania Department of Education Phone: 717.346.8064 dsimaska@pa.gov Rebekah Baum-Leaman, Ed.D. Bureau of Curriculum, Assessment and Instruction Division of Instructional Quality Pennsylvania Department of Education Phone: 717.783.6538 rbaumleama@pa.gov

Lessons Learned

• Challenges in applying copied text policies when scoring

– What constitutes copied text – Communication of policy – Amount of time spent

• Gathering responses to review at client meetings
• Meetings – determining lines
• Throughout scoring window
• With AI-scored responses

Lessons Learned

• Considerations for item development to minimize instances of

students relying solely on source text

– Analysis and synthesis information from multiple sources must be emphasized – Purpose must be clear (arg/opinion; info/expos) – Balance topic, prompt, and text concepts

• Concept must be grade-level appropriate
• Clear connection between topic, text/passage set, and prompt

– Commonly copied pieces:

• Analysis of a quotation copy of that piece of text/passage
• Beginning of passage
• Unique/interesting/colorful quotations or lines

Educating the Field

• Continued professional development for field
• Provide examples of what state’s “copy” looks

like

• Appropriate score reports to assist in

understanding