AAC&U General Education and Assessment Conference, 2018 Missy - - PowerPoint PPT Presentation

AAC&U General Education and Assessment Conference, 2018

¡ Missy Schen ¡ Denise Bostdorff § Civic Engagement Rubric ¡ Michelle Johnson § Quantitative Literacy Rubric ¡ Rohini Singh § Problem Solving Rubric

¡ Small liberal arts college in rural Ohio § Approximately 2,000 students § 100% Complete Senior Independent Study (“IS”) § Part of the Great Lakes Colleges Association

(GLCA)

¡ Student learning assessment is primarily

department-based

§ Direct and indirect assessments, often on IS § No signature assignments

¡ Sherman Fairchild Foundation funded 2-year

project to help GLCA schools use VALUE rubrics for direct assessment of student learning

¡ Wooster focused on testing rubrics for

course-embedded assessment at multiple levels

§ 13 departments, 36 faculty, 7 VALUE rubrics

¡ Aligning assignments with the VALUE rubrics ¡ Grading assignments vs. assessing with

rubrics

¡ Taking a look at assignments ¡ Data interpretation and action ¡ Wrap up and questions

¡ Each row = Dimension ¡ Each column = Levels ¡ Each cell = Performance descriptor ¡ Read the performance descriptors from left

to right, determining if the student meets each level as you go

¡ We will consider the rubric as is ¡ Definitions on back

¡ Used for two courses: § 1st course: Applied rubric to existing assignment § 2nd course: Adjusted rubric by cutting two

dimensions, made assignment more explicit

¡ Group project: Policy proposal ¡ Individual component: reflection ¡ Rubric adapted partially § Did not include “Evaluate Outcomes” and

“Identify Strategies” in the assignment or grading

§ Adapted “Implement Solutions” ¡ Included ideas from rubric in designing

assignment and writing instructions

¡ Final research project for Quantitative

Research Methods

¡ Did not specifically align assignment with

rubric

¡ Using rubric framing language and definitions

to help guide language used in assignment and class

§ Need to be explicit in assignments to reflect rubric

dimensions

§ Not every dimension in a rubric is appropriate for

an assignment, course, or discipline

§ Take care not to turn the rubric into an

assignment sheet

¡ Rubrics aren’t equally applicable

¡ Don’t need to perfectly overlap rubric and

grading scheme

§ Pro – make sure students are fulfilling goals of

class/discipline not related to rubric (e.g. writing skills and working in collaboration)

§ Con – time required to both grade and then assess

with rubric; need broader scale in grading

¡ Rubric organization can help to think about

grading scheme

¡ Can use the rubric as part of grade

¡ Using one of your own assignments or the

samples provided, consider:

§ How well does the assignment align with the

VALUE rubric? How would you change the assignment to better fit?

§ How would you incorporate the VALUE rubric in

the grading? Would you change the grading scheme to better fit the rubric?

Capstone 4 Milestone 3 Milestone 2 Benchmark 1 Mean Explanation of issues 11 21 33 7 2.50 Evidence 7 21 36 6 2 2.35 Influence of context and assumptions 8 31 27 4 2 2.54 Student's position 4 21 26 21 2.11 Conclusions and related outcomes 5 17 34 16 2.15

¡ Critical Thinking VALUE Rubric ¡ 73 First year seminar student final papers

¡ When interpreting data, consider:

¡ Course goals ¡ Pedagogical choices ¡ Student demographics/development

¡ Data helps you evaluate how well you have reached

goals and their particular aspects

¡ In looking at results and making course

adjustments:

¡ Pro – being more intentional in teaching and assignments ¡ Con – can over-emphasize criteria where students did

poorly, at expense of other criteria

¡ Right now, findings are focused on refining

individual courses

¡ Limited in comparing across classes/students

without repeated measures

¡ AAC&U ¡ GLCA ¡ Sherman Fairchild Foundation

Handout for: Schen, M., Bostdorff, D., Johnson, M. & Singh, R. (2018, February). Lessons learned from using the VALUE rubrics for course-embedded departmental assessment. Session presented at AAC&U General Education and Assessment Annual Conference, Philadelphia, PA. COMM 250 Final Exam Synthesis and Reflection Paper Your final exam reflection paper will be a 6-8 page, double-spaced paper that asks you to pull together material from the course to write a deep reflection on how rhetoric intersects with civic life and how your own sense of civic engagement has evolved throughout the semester. It should cite course readings using appropriate MLA in-text citation with author last name and page number and should be accompanied by a Works Cited in proper MLA format. This essay is due to me, both in hard copy and on Moodle, by Wednesday, December 13, at 2:00 pm. No late work will be accepted. This final assignment is worth 50 points toward your final grade. Reminder of Course Objectives As you prepare, please keep in mind the course objectives, which are that students completing this course should be able to:

• 1. explain rhetorical concepts covered in the course;
• 2. identify and analyze the three constituent elements of a rhetorical situation;
• 3. write an analysis of a rhetorical message in light of rhetorical situations and concepts;
• 4. construct a message about an issue that applies course concepts in light of a particular situation for civic

ends. Writing Prompts Overall, your paper should attempt to answer the question: How does rhetoric intersect with civic life and how has your own sense of civic engagement evolved over the course of the semester? Be sure to define rhetoric and civic engagement. The goal is to answer this overarching question and also to demonstrate to me your knowledge of the course material and your ability to think critically as you apply it. As you answer the overall question, make sure that you attend to each of the following topics:

• a. the diversity of communities and cultures;
• b. the application of communication knowledge and concepts to understanding the ways in which people

participate in civic life;

• c. factors that influence civic identity and what you have learned about your own civic identity;
• d. civic communication, including listening, adapting one’s ideas based on listening, and tailoring

messages to audiences.

CIVIC ENGAGEMENT ASSIGNMENT

Course Readings and Application/Analysis Synthesis and reflection papers should:

• incorporate the breadth of material we have covered this semester which, broadly speaking, includes the

below topics, although you should refer far more specifically to particular concepts within these categories;

• refer to and cite specific readings and page numbers—that is, you should be consistently referring to

course readings (and citing appropriately in accordance with MLA;

• explain and apply concepts accurately and synthesize. Avoid the temptation of only repeating what the

readings have said. Instead, dig deep to integrate ideas and show how they connect.

• reflect deeply. Instead of providing clichéd and/or superficial comments, challenge yourself as you

reflect on your sense of civic engagement and its evolution. Rhetoric and symbolic action Audiences Rhetors Media and public issues Rhetors and the postmodern world “Fake” news Unsupportive and supportive responses of majority Rhetorical situations group members Public issue letters/emails Political imagination Phone calls to officials Archival messages about World War II, 1950s anti- civil rights documents, 1980s anti-nuke messages Publics and counterpublics Publics in the digital age Language (e.g., narrative, metaphor doublespeak, etc.) President Trump & Twitter Visual rhetoric Internet rhetoric of the KKK AIDS Memorial Quilt #BlackLivesMatter Evaluation In evaluating your essay, I will apply the following the criteria:

• 1. Have you fully responded to all parts of the writing prompts?
• 2. Does your essay have an overall thesis that answers the overarching question and that ties the various

parts of your essay together?

• 3. Does your essay draw on a wide range of course readings, both the text and the ERES and online

readings, in responding to the writing prompts?

• 4. Does your essay demonstrate accuracy in both explaining and applying concepts from the readings?
• 5. Does your essay synthesize course concepts rather than simply repeating what the readings say?
• 6. Does your essay demonstrate deep thought and self-reflection in its analysis?
• 7. Does your essay make use of brief examples to illustrate your points?
• 8. Does your essay cite readings internally by author last name or organizational author name and, when

page numbers exist (as they do for almost all readings) by page number?

• 9. Does your essay have attached to it a Works Cited with the list of readings that you actually referred to,

in proper MLA?

• 10. Is your essay written well? Are the ideas presented clearly, and does the essay follow the writing

guidelines for the class?

COMM 250 Synthesis and Reflection Take-Home Final Exam ___Did not respond to all parts of the writing prompt: ___ Did not discuss how rhetoric intersects with civic life ___ Did not discuss how your own sense of civic engagement evolved over the course of the semester ___ Did not define rhetoric and civic engagement and explain their relationship ___ Did not discuss: ___ diversity of communities and cultures ___ application of communication knowledge and concepts to understanding how people participate in civic life ___ factors that influence civic identity and what you learned about your own civic identity ___ civic communication, including listening, adapting one’s ideas based on listening, and tailoring messages to audiences ___ Did not have an overall thesis that answers the overarching question and that ties the various parts of your essay together. ___ Did not draw on a wide range of course readings, both the text and the ERES and online readings, in responding to the writing prompts. ___ Did not demonstrate accuracy in both explaining and applying concepts from the readings. ___ Did not demonstrate deep thought and self-reflection in its analysis. ___ Did not make use of brief examples to illustrate your points. ___ Did not cite readings internally by author last name or organizational author name and, when page numbers exist (as they do for almost all readings) by page number? ___ Did not have attached Works Cited with the list of readings that you actually referred to, in proper MLA. ___ Essay was not well written: the ideas were not presented clearly, or the essay did not follow writing guidelines for the course, or was poorly proofed, etc.

CIVIC ENGAGEMENT GRADING SHEET

COMM 20008W: COMMUNICATING PUBLIC POLICY (FALL 2017) GROUP PROJECT ASSIGNMENT GUIDELINES Overview This assignment has 2 objectives:

• 1. To give you experience in working collaboratively with others to analyze a problem and design a fitting
• solution. Here, I will evaluate the end-product – in this case, your group’s policy proposal.
• 2. To have you reflect on the process of problem-solving and decision making in groups – which is how most

policy making is conducted. Here, I will evaluate process – in this case, your reflection on how your group worked together to solve a problem. Instructions This assignment has 3 components. Here are the instructions for each component:

• 1. Group Proposal
• a. For this part of the assignment, you will work as a group to solve a pressing local problem. You should

present your analysis of the problem and your proposed solutions and implementation plan in an 8 page group proposal. Each person in the group is responsible for writing 2-3 pages. The proposal should include the following sections:

• i. Define the Problem (2 pages): In this section, you should identify a problem, describe its scope

and scale (how the problem is and what groups it affects), and analyze its causes. You must give a sense of the current context within which this problem operates, and show why it is urgent that action be taken on this issue. Note: This problem must be specific to Wayne County, Ohio. Why? Because much of the policymaking that affects our daily lives occurs at the local level – so let's begin there.

• ii. Propose Solutions (2 pages – 1 page per solution): Then, propose 2 possible solutions to the
• problem. For each solution, you should tell your readers (i) what your proposed solution is

(give a brief description); (ii) your reasons for taking this approach and (iii) details on how it will work. Here, you should provide as many specific details about your proposed solution, what resources it will need (and where you will get these resources), an estimate of how much it will cost, how quickly it can be implemented, whether there will be incentives to participate Components: 4-5 page draft proposal; 8 page final version (plus cover sheet and Works Cited list); 2-3 page individual write up; 10 minute presentation. Point Value: 50 [Draft (5 -shared grade); Final Version (20 -shared grade); Individual Write-up (20 – individual grade); Class Presentation (5 -individual grade)] Due Dates

• Wed Sep 13: Groups assigned
• Mon Sep 18: Topic approval due by 4pm
• Fri Sep 29: Drafts due in class for peer review
• Fri Oct 6: Final Version of Group Project, Class Presentation, and Individual Write-Up due

PROBLEM SOLVING ASSIGNMENT

(or disincentives/punishments for not participating), etc. Note: Part of the grade for this assignment is based on how well you fit the solution to the problem and how realistic your solution is. Avoid proposing an "off the shelf" solution, which is a simplistic "typical" solution not tailored to the specific problem. An example

• f an "off the shelf" solution is to hold a bake sale to save an underfunded local school – the solution does not

really fit the nature of the problem, but is an easy and common idea that people are familiar with, so it gets proposed a lot.1

• iii. Evaluate Solutions (2 pages): Next, compare and contrast both solutions using the criteria of

validity, feasibility, and ethicality, and end by selecting 1 solution as being the better of the two, explaining your decision. The comparison should take up at least 1.5 of the 2 pages.

• iv. Implementation Details (1 page): Now that you’ve decided which solution to go with, you

need to explain how you would implement this solution. In this section, you should be as specific as you can about the steps you will now take to carry out your proposed policy and make it a reality (this could include not just how you will implement the proposal but how you will get support for the proposal from key stakeholders and affected parties. However, your implementation details cannot be only about gathering public/political support – you should focus on offering details of how you will go about carrying out the plan. You should also explain how you plan to evaluate whether the proposal has been successful). Finally, tell us how you will message this solution – the kind of language you will use, and the kinds of images and public messages you will choose to accompany the implementation of this policy.

• v. Public Messaging/Material (1 page). Here, you should provide 1 communication 'text' to

accompany the implementation of your policy. Some examples are: a campaign poster, a write- up for the local paper, a mock interview with residents/groups affected by this policy, a brief speech given by a relevant public figure, recruitment material related to your proposal, a mock- up of a website created to help implement your proposal, etc.

• b. On Friday Sep 29, each person in the group should bring a 4-5 page hard copy draft to class for peer

review (the drafts should be identical). This 4-5 page hard copy draft is worth 5 points – everyone in the group will share the grade received for the draft. If the draft is less than 4-5 pages, the group will not receive the full 5 points. And there will be no points given at all if any person does not bring a copy of the draft.

• c. On Fri Oct 6, each group should submit 1 hard copy, stapled, 8 page, revised version of their proposal,

incorporating feedback from the peer review session. Include a cover page in front and an MLA- formatted Works Cited list at the end (these do not count toward the 8 page requirement). On the cover page, make sure to include (in any order you deem fit): all group members' names, a title for your project that describes the problem, the name of the course, the date, and an image relevant to your project. The entire 8 page proposal is worth 20 points – everyone in the group will receive the same grade. Thus, although you will in all likelihood split up the writing for this assignment, you should make sure that it flows together well and that there are no significant jumps in the quality of the writing and research because I will grade the proposal as a whole.

• 2. Individual Write-up
• a. For this component of the assignment, I would like each group member to reflect on the features and

dynamics of working together to solve a problem. Your group has two problems to solve in this project: one is the social problem you’re supposed to address for the assignment, and the other is the “problem” or task of working together in a group. Therefore, your individual write-up should include

1 I took the concept of an "off the shelf" solution and the example of holding a bake sale to save an underfunded public institution from

the Glossary of the Problem Solving Value Rubric created by the Association of American Colleges and Universities.

your expectations, observations, and reflections on how your group handled both these tasks. This component of the assignment is graded individually and is worth 20 points. Your document must be written in the form of an essay containing the following elements:

• i. Pre-project expectations: Read the assignment guidelines and give yourself some time to think

about the project before meeting with your group. We will allocate time in class on Wed Sep 13 for you to think and make notes individually before you meet as a group. As you sit and think by yourself, make notes about your expectations for this project. How do you think your group will handle the two tasks of solving a social problem in Wayne County, Ohio and of working together in a group? What are you expecting will happen? What are some concerns you have about this project? What are some challenges you foresee? What are some elements

• f this process that you are looking forward to?
• ii. Observations: In the 2-3 weeks that your group works together on this project, make notes on

how the group approached this assignment. Some things you may want to include in your

• bservation are: how did the group identify a problem to focus on? What kinds of questions

did people ask about the problem, and how did they go about designing solutions? What were some assumptions people made, if any, in choosing specific problems to focus on, and in designing solutions to those problems? How did the group come up with an implementation plan and communication message for the solution they chose? Remember that you should make observations on the group’s problem-solving process both in terms of the public policy problem it had to solve, as well as how the group handled the dynamics of collaboration with

• thers in a small group setting.
• iii. Reflections. Finally, look back to the beginning of this project and at your own pre-project

expectations and the observations you made during the project. What has this project taught you about the positive aspects as well as the possible pitfalls of public policymaking in a group? What has this experience taught you about some of the features of the policymaking process and of working in small groups? How has this experience changed how you think about policymaking and about group work?

• 3. Class Presentation
• a. On Friday 6 October, each group will make a 10 minute presentation of their project. This

presentation will consist of a 5 minute overview of the project and a 5 minute “press conference” in which the audience will ask the group questions about their project. In your group, assign 2 people to do the presentation and 1-2 people to do the press conference. You will each be graded individually

• n your performance in the presentation.
• i. 5 minute overview: In this overview, 2 people should divide equally the task of presenting your
• project. Here, you should focus on describing the problem your group addressed, and telling

us details about your chosen solution. You do not need to talk about both your proposed solutions, only the one you selected as being the better of the two. Finally, you should tell the audience about your plans to implement this solution, including the communication messaging

• r material you will use.
• ii. 5 minute press conference: Here, 1-2 members of the group will take questions from the class

about their project. I suggest that the group prepare ahead of time to anticipate possible questions so that the person/people handling the press conference can prepare some answers. Note: I will be looking closely at the audience to see who asks questions, and also what quality of questions you ask, as I will factor the audience’s participation into each person’s overall class engagement score.

COMM 20008W: COMMUNICATING PUBLIC POLICY (FALL 2017) GROUP PROJECT GRADING SHEET Content: What you say Breadth & Depth

• Does each part of the project contain all the components/sections/parts it is supposed to have, i.e.:
• Cover Page: Group members, project title, date, course, image.
• Define the Problem; Scope, scale, context, urgency
• Propose 2 Solutions: (i) describe each solution (ii) Give reasons (iii) Give details
• Evaluate solutions: (i) Compare & contrast ethicality, validity, feasibility (ii) Pick one
• Implementation Details: Steps, gather support, messaging strategy, criteria for success
• Public Messaging/Material
• Works Cited
• Is enough space dedicated to each component?

Validity and Quality:

• Is the project well-conceived?
• Does it identify a problem local to Wayne county?
• Do the proposed solutions actually fit the problem, or they just “off-the-shelf” solutions?
• Does the evaluation section correctly apply the criteria of validity, ethicality, and feasibility?
• Are the implementation details realistic?
• Does the project draw on sound research and solid evidence?
• Does the Public Messaging/Material fit well with the nature of the problem and the solution?

Expression: How well you say it Writing Quality:

• Distinguishing your voice from that of others
• Making your stance clear
• Using transitions
• Framing quotations
• Summarizing without listing
• Work should be free of copious errors of spelling and grammar.
• Although you will in all likelihood split up the writing for this assignment, you should make sure that it

flows together well. Format:

• MLA in-text citation
• MLA Works Cited list
• Page numbers

PROBLEM SOLVING GRADING SHEET

COMM 353 Research Paper Assignment 100 Points

• A. State and explain the research question(s) that guided your study. One paragraph.
• B. Describe the method. Include the following. 1-2 pages.
• 1. A description of your instrument (the survey). Provide information regarding the number of

questions, the kinds of questions, and the focus of the questions. Include also some examples of questions, or paraphrases of questions. Organize your description around focus/content of the itmes, NOT type of items, and not a blow-by-blow from the start of the survey to the end (unless your questions were organized into blocks of similar focus/content). A reader should have a very good idea of what you measured and how from this description. If you took questions from other scholars’ surveys, this section should clearly identify which items came from which sources (cite the sources) and whether you used previous items exactly or modified versions of previous items. In this section, refer the reader to Appendix A, which will be a copy of your codebook. (Normally, you would just attach a clean survey, but for this class, I need your codebook.)

• 2. Describe the procedures you followed to collect data. Include a clear identification of the sampling

method(s) and a description/definition of the sampling method(s) used. Provide the kind of detail that would allow another researcher to do exactly what you did. Think of this as a recipe.

• 3. A description of your sample. Include the total number of participants, a breakdown of the various

demographics measured, and any other background information gathered for the purposes of sample

• description. Report the appropriate statistics.
• C. Provide the Results of your statistical analyses and Discuss the findings. 5-8 pages.
• 1. Organize this section in a meaningful fashion based on your variables of interest. In other words,
• rganize around concepts, NOT around statistics. For example, do not have a t-tests section, have

instead a Gender Comparison section.

• 2. Be sure to run and report ALL analyses necessary to fully answer your research question(s). Part of

this may include the running and reporting of reliability analyses. Most, if not all, of you have multiple items that measure the same concept. As such, most, if not all, of you should be running reliabilities (don't forget to recode first) and computing total scores to use in the analyses.

• 3. Standard practice within sections is to report univariate analyses first and multivariate analyses

second.

• 4. When writing this section do not use cryptic spss terms for your variables/items/factors. Use full

descriptive language that an outside reader will understand.

• 5. For the purposes of this assignment, you must include at least two statistics tables in the results
• section. You should include additional tables as necessary. DO NOT simply import SPSS tables into

your paper.

QUANTITATIVE LITERACY ASSIGNMENT

• 6. After each result or set of related results, explain WHY you found what you found.
• Explain why differences or relationships were significant or not significant.
• This section should follow the format of results-discussion, results-discussion, etc.
• The discussion section should not include numbers and should avoid direct reference to statistical

analyses by name (t-test, correlation, etc.).

• The use/citation of at least one scholarly journal article is expected here. Discuss your

findings in relation to previous findings. Do your results support or contradict previous findings?

• Failure to provide thoughtful and critical explanations for your findings will negatively impact the

paper grade. Other Information/Requirements

• 1. Include a title page that provides the title for your paper and your name. Your name should NOT appear
• n any other page. The title should not be your research question.
• 2. Number ALL pages in the upper right corner and double space all text.
• 3. Use 3rd person and write in past tense (The results revealed, found, included, etc.).
• 4. Use one inch margins on all four sides of the page and Times New Roman 12 point font.
• 5. Include a copy of your codebook. This codebook will serve as Appendix A.
• 6. Include a copy of your SPSS output for the statistics USED/REPORTED in the paper.
• a. Highlight on the printouts the specific information you used and make sure the variables are

easily identifiable on the printouts.

• b. Organize the SPSS printouts so that they follow your results section.
• c. Number the SPSS pages clearly in the upper right hand corner with a marker.
• 7. Include headings and subheadings. You must use the following 6 center headings:

(2) Purpose (3) Instrument (4) Procedures (5) Sample (6) Results and Discussion (a) within this section will be unique subheadings for stats and discussions (b) these subheadings should be on the left margin (7) Limitations

• 8. Proofread carefully so as to eliminate all grammatical, structural, and typographical errors.
• 9. Include a properly formatted reference page. Use APA formatting style. To save paper, you do not

need to start your references page on a new page.

• 10. Turn everything in together in a folder or small binder.

COMM 353 Research Paper Grading Rubric Clearly state research question(s) ______/5 Clearly and completely describe the sample ______/5 Clearly and completely describe the instrument ______/10 Clearly and completely describe the procedures ______/5 Provide appropriate, clear, and properly formatted results to answer your research question. (including requirement for two tables) ______/40 Discuss the findings (incorporate at least one scholarly source) ______/15 Formatting (spacing, font, page numbers, title page, margins, etc.), codebook, APA ______/10 SPSS printouts (pages numbered, organized, properly highlighted, etc.) ______/10

QUANTITATIVE LITERACY GRADING SHEET

2/21/18, 2:09 PM Edit Survey | Qualtrics Survey Software Page 1 of 2 https://wooster.co1.qualtrics.com/ControlPanel/?

Projects Contacts Library Help

• Communication: Civic Engagement V…

Survey Actions Distributions Data & Analysis Reports

Communication: Civic Engagement VALUE Rubric

This survey is currently LOCKED to prevent invalidation of collected responses! Please unlock your survey to make changes.

Default Question Block Block Options

Q1

• Civic Engagement VALUE Rubric

Q2

• Course Number

Q3

• Semester

Year Fall Spring Course Information

Q4

• Student class year:

2017 2018 2019 2020

Q5

• Student gender:

Male Female Unknown

2/21/18, 2:09 PM Edit Survey | Qualtrics Survey Software Page 2 of 2 https://wooster.co1.qualtrics.com/ControlPanel/? Add Block

Q6

• Student major/minor status:

Major: Communication Major: CSD × Neither major/minor

Q7

• Evaluators are encouraged to assign a zero to any work sample or collection of work that does

not meet benchmark (cell one) level performance.

Capstone (4) Milestone (3) Milestone (2) Benchmark (1) Does not meet Benchmark (0) × N/A Diversity of Communities and Cultures Analysis of Knowledge Civic Identity and Commitment Civic Communication Civic Action and Reflection Civic Contexts/Structures Qualtrics.com Contact Information Legal