[PPT] - Assessing Student Writing Across Programs and Time: PowerPoint Presentation

SLIDE 1

Assessing Student Writing Across Programs and Time: (Inter)disciplinary and Programmatic Perspectives

Presented by Ann M. Blakeslee, Ph.D., Director, Office of Campus and Community Writing

W. Douglas Baker, Ph.D., Associate Dean, College of Arts and Sciences

SLIDE 2

Session Parts

1. Challenges, potential, and importance of assessing student

writing across time and programs

2. EMU’s context and history
3. Our response and the results and outcomes
Faculty investment
Use of campus LMSs and other resources
4. Plans and next steps

SLIDE 3

First, why are campus-wide writing programs and assessments of those programs and student writing important?

SLIDE 4

Importance of Campus-wide Writing and Assessment of Writing

Proven importance of writing to help students improve as

writers and to support their learning and critical thinking (Anderson et al., 2015; Peer Review 2017)

First-year, intermediate, and advanced [writing in the disciplines

(WID)] offerings at many universities (Morris, 2017; Rutz and Grawe, 2017)

Deliberate/non-deliberate scaffolding of writing across the curriculum
Importance—and difficulty—of assessing these initiatives

(NCTE, 2014; Sparks et al., 2014; Wardle and Roozen, 2012)

SLIDE 5

Challenges in and Potential for Assessing Student Writing

Faculty perceptions of responsibility
Disciplinary differences
Programmatic vs. individual
Past/current experiences with “assessment”
Example: “It turns out that the assessment program your college

imposed on you was probably never going to improve anything. A new article by an assessment insider [David Eubanks] explains why this is so and suggests that assessors have known for sometime now that assessment does not work.” - The Chronicle of Higher Education, January 12, 2018

SLIDE 6

EMU Assessment Context and History

General Education Reformed (approved 2006-07)
Plans to assess Gen Ed learning outcomes (2007-12)
Centralized approach: Office of Institutional Effectiveness and

Accountability (2009-2012)

Decentralized: Programmatic assessment of student learning to

colleges and General Education Program (2012-current)

Developing local capacity and leadership
Soliciting participation (e.g., College of Arts and Sciences,

approximately 70% of 134 programs contribute)

SLIDE 7

Our response to the challenges and to our context and history…

SLIDE 8

Small Group Share

What is the situation on your campus with student writing and the assessment of student writing programs? Discuss in a small group of 3-4.

SLIDE 9

Gen Ed/WI history…

1. General Education reform – 2002/03; Implementation – 2006/07
Education for Participation in a Global Community
Effective Communication – first-year writing and speech
Upper-level writing-intensive course requirement
2. FYWP assessment – 2013-16 (three parts)
3. WI assessment – starting in 2015/16; ongoing

SLIDE 10

Background…

1. Assessment of WI was planned from the outset
2. WI classes are taught and approached in a variety of ways
3. Writing Across the Curriculum (WAC) has supported the

development and teaching of WI classes since their inception

4. Colleges, departments, schools, and programs all have different

needs and interests and face different challenges and opportunities with WI classes

SLIDE 11

WI Outcomes

Literacy Strategies that Support Inquiry: Develop and employ

successful and flexible reading and writing strategies that support sustained inquiry in a discipline

Rhetorical Awareness: Use writing strategies that achieve the

purposes for writing and address expectations of audiences in a disciplinary context

Research Strategies: Formulate research questions and employ

strategies for researching and responding to those questions

Genre Awareness: Use discipline-specific genres to communicate info
Disciplinary Conventions: Understand conventions for

communicating, disseminating, and interpreting information in a discipline

SLIDE 12

Our Initiative

Began with WAC/UWC impact survey in 2016
WI assessment pilot was carried out in 2016
Dietetics, social work, history
Used AACU value rubric for written communication
Most recent assessment started in winter 2017
Participation was voluntary – sent invitation to faculty on

record for teaching a WI class that semester

37 initial faculty volunteers
Meetings to explain and orient volunteers to assessment

SLIDE 13

SLIDE 14

Components and Rationales

Faculty questionnaire:
Writing assigned, genres taught, strategies used
Understanding of outcomes
Perceptions of effectiveness in teaching and achieving outcomes
Perception of student success in developing proficiency with outcomes
Ease in teaching, allocation/use of class time
Collection and review of syllabi:
Presence/absence of required rationale
Presence/absence of WI/any writing outcomes
Uses of drafts, peer review, instructor feedback, revision, other strategies
Student questionnaire:
Students’ perceptions of proficiency with the outcomes

SLIDE 15

Components and Rationales

Rubric in Canvas:
Faculty assessment of student success with outcomes on a particular assignment
Post-assessment questionnaire:
Any changes they would make as a result of the assessment
Use of class time after vs. before assessment
Understanding of outcomes after assessment
Perception, after assessment, of
Success and effectiveness in teaching and achieving outcomes
Student success in becoming proficient with outcomes
Ease or difficulty of teaching outcomes
Focus groups:
Students’ experiences with the classes
Faculty members’ experiences with the classes and perceptions…

SLIDE 16

Approach

Collected course section numbers
Developed and loaded rubric into Canvas
Held information sessions
Developed and distributed questionnaires
Provided written instructions for using the rubric
Provided individualized instruction as needed

SLIDE 17

Small Group Share

What have you, or others on your campus, done (or what would you like to do) to assess writing programs and/or student writing? Why? What are your reasons/rationales? Discuss in a small group of 3-4.

SLIDE 18

Faculty Questionnaires…

SLIDE 19

Demographics/Information

Most tenured (17), tenured track (8)
Disciplines: STEM, humanities, social sciences,

business/professional

Majority face-to-face (76%)
21% hybrid/web-enhanced
Majority had taught class more than twice
46% had taught it 6+ times
Enrollments ranging from 15 to 45
Most 20-25

SLIDE 20

Writing

Mix of formal/informal
Genres: research papers (66%), lit reviews, essays,

abstracts, summaries, multi-genre projects, other (48%)

Approaches to assignments – 86% scaffold
Drafts (76%)
Revision (68%)
Peer review (64%)
Project proposal, annotated bib (52%)
Rubrics (79%)
Feedback on drafts (83%)
Opportunities for revision (86%); Require revision (60%)

SLIDE 21

Writing Feedback

I'm developing Canvas discussions in which students explore ideas in their formative stages, and students respond to each others' Canvas writings in this way, and I also comment on and grade these postings. For each of the two assigned "analytical essays" (no research), I comment on essays and require/recommend resubmission with revisions. For the final "research" essay, I require that students defend their research questions orally and then they bring in rough drafts for peer review or individual consultation. I also require an "annotated bibliography" for this assignment in the weeks that precede the draft, so that I can help students to find the best research for their ideas (and so that I can assess -- and intervene when necessary -- how well they understand and are able to use their sources). (high touch approach)

SLIDE 22

Writing Feedback

My feedback is typically at the higher and middle levels of concern with an emphasis on what worked/did not work for this assignment and questions to answer or things to be aware of when preparing for the next assignment. To allow students time to review and benefit from the feedback, I work to return feedback within one week of the assignment due date and make sure that major writing assignments are reasonably spaced throughout the term.

SLIDE 23

Outcomes – Understanding of

Excellent Good Average Limited

Wtd. Avg.

Literacy 38% 52% 7% 3% 1.76 Rhetorical 52% 38% 7% 3% 1.62 Research 61% 25% 14% 1.54 Genre 55% 34% 10% 1.55 Conventions 69% 28% 3% 1.34

SLIDE 24

Outcomes – Perception of Effectiveness in Teaching

Highly Effective Effective Somewhat Effective Not Effective

Wtd. Avg.

Literacy 28% 45% 21% 7% 2.07 Rhetorical 17% 66% 17% 2.00 Research 22% 44% 30% 4% 2.15 Genre 34% 41% 24% 1.90 Conventions 29% 43% 25% 4% 2.04

SLIDE 25

Outcomes – Perceptions of Student Success in Achieving Proficiency with

Highly Successful (exceeds expectations) Successful (meets expectations) Somewhat Successful (approaches expectations) Unsuccessful (does not achieve expectations)

Wtd. Avg.

Literacy 7% 66% 24% 3% 2.24 Rhetorical 3% 72% 24% 2.21 Research 19% 33% 33% 15% 2.44 Genre 14% 69% 17% 2.03 Conventions 10% 62% 28% 2.17

SLIDE 26

Student Questionnaires…

SLIDE 27

Student Questionnaires

Disciplines - STEM, humanities, social sciences,

professional

Majority were native English speakers
Self-rated proficiency levels:

1) Literacy: highly = 42%; proficient = 57% 2) Rhetorical: highly = 48%; proficient = 50% 3) Research: highly = 42%; proficient = 45%; minimally proficient = 12% 4) Genre: highly = 46%; proficient = 46%; minimally proficient = 8% 5) Conventions: highly = 47%; proficient = 50%

Moving forward – start and end-of-semester surveys

SLIDE 28