FETC 2006- FETC 2006 -2007 2007 Jannette Jannette Finch, North - PDF document

FETC 2006- FETC 2006 -2007 2007 � Jannette Jannette Finch, North Campus, Library Finch, North Campus, Library � Online Course Evaluation Online Course Evaluation � James Neff, Physics and Astronomy James Neff, Physics and Astronomy � System: A Pilot Study System: A Pilot Study � Myra Seaman, English, Fall 2006 Myra Seaman, English, Fall 2006 � � Jared Jared Seay Seay, Library , Library � Faculty Educational Technology Faculty Educational Technology � Whit Whit Schonbein Schonbein, Philosophy, Spring 2007 , Philosophy, Spring 2007 � Committee (FETC) Committee (FETC) � Christopher Starr, Computer Science Christopher Starr, Computer Science � Mark W. Hurd, Ph.D., MCSE, Chair Mark W. Hurd, Ph.D., MCSE, Chair � Mark Hurd, Psychology, Chair Mark Hurd, Psychology, Chair � 2006- 2006 -2007 2007 History History Overview Overview � The paper The paper- -based course evaluation form / based course evaluation form / � � In 2005, the FETC was charged with studying instrument instrument the feasibility of implementing an online � General issues General issues � evaluation system at the CofC. � Time Time � � In 2006, a number of concerns were raised in � � Labor Labor the Senate with regard to implementing such a � Financial considerations Financial considerations � � An online evaluation system An online evaluation system system and with the language of the proposal. � � Advantages Advantages � � In 2006-2007, the FETC attempted to address � Concerns Concerns � the concerns that were raised and move � Challenges Challenges � forward to assess the feasibility of an online � Current status Current status � course evaluation system. General issues with the current General issues with the current system system � The current procedure relies on a manual multi- � Lack of security. Evaluation forms may be lost, step delivery system of paper instruments for misplaced, misused, incorrectly administered dissemination. (e.g. during the final exam), or never administered. � Multiple parties are involved including: � C Confidentiality and privacy may be compromised � � Information Technology given the number of hands that pass these � Accountability, Accreditation, Planning & Assessment (AAPA) Office paper instruments. � Individual departments � Open-ended (comments) questions are hand written � Faculty, staff, and students and compromise the student’s anonymity. � The Library

General issues with the current Issues with the current system - system Time � Data are difficult to analyze. Data are difficult to analyze. � Slow feedback: Approximately 15 weeks � are required to prepare, deliver, return and � Results are presented in a paper format. This format is difficult for faculty, chairs and tabulate the paper forms. administration to use effectively. � Class time is also used for in-class � Data cannot be easily extracted from the Data cannot be easily extracted from the � evaluations. paper- paper -based forms. based forms. Issues with the current system - Issues with the current system – – Issues with the current system Labor Financial considerations Financial considerations � Pre Pre- -evaluation labor (IT): evaluation labor (IT): � � The process is technically expensive � 10 hours to setup forms 10 hours to setup forms � (paper and printer) � 63 hours to print the forms 63 hours to print the forms � � ~ 70,000 evaluation forms per year not � 77 hours to sort for delivery to departments 77 hours to sort for delivery to departments � � 54 hours to deliver to the departments 54 hours to deliver to the departments � including paper copies issued later to faculty. � Total: ~ 205 hours + Administrative Total: ~ 205 hours + Administrative � Assistants time in each department (~8 hrs) Assistants time in each department (~8 hrs) � And the forms still have to be returned for additional processing/scanning by IT and AAPA once they’ve been completed. They also have be copied at the departmental level. Issues with the current system – Issues with the current system – Issues with the current system - - Issues with the current system Financial considerations Financial considerations � Printing: Printing: � � Current evaluation rate: Current evaluation rate: 67% 67% � � Purchasing forms: $3000 / year Purchasing forms: $3000 / year � � The current system is static and inflexible The current system is static and inflexible � � Printing forms: $3600 / year Printing forms: $3600 / year � � Errors in printing: $800 / year � Errors in printing: $800 / year � Potential for fines from the Council on Potential for fines from the Council on � � Total = ~ $7500 Total = ~ $7500 � Higher Education (CHE) if we are non- - Higher Education (CHE) if we are non � Hardware: Hardware: � compliant compliant � PC: $1500 PC: $1500 � � Scanner: $9000 Scanner: $9000 � � Total = ~ $10,500 Total = ~ $10,500 � � Grand Total = ~ $18,000 + labor costs (~205 Grand Total = ~ $18,000 + labor costs (~205 � hours ++) hours ++)

Overview Overview Advantages � The paper The paper- -based course evaluation form / based course evaluation form / � Rapid feedback. Student comments are � instrument instrument returned immediately after the semester ends � General issues General issues � for formative use before the next semester � Time Time � starts. � Labor Labor � � Anonymity. Student comments are typed to � Financial considerations Financial considerations � better preserve anonymity. � An online evaluation system An online evaluation system � � Analysis . Results are returned electronically in a � Advantages � Advantages form more suitable for analysis and graphing. � Concerns Concerns � � Challenges Challenges � � Current status Current status � Advantages Advantages � Less expensive to administer. Business � Enhanced security. processes are simplified. � Data may be encrypted. � Requires less class time to administer. � Faculty do not have to depend on one student � It is not necessary to remember to take the to execute the process in class and return the forms to class, multiple copies need not be forms. made). � Student’s comments will be anonymous. � No “double-bubbling” / unusable forms. � More opportunities to evaluate. Faculty will not miss access to student feedback due to student absences. Advantages Additional Features � Data integrity is increased � The number of days and the hours per day, during which the evaluations are available, can � Enhanced data security and student be adapted to alleviate faculty concerns. anonymity. � Comment boxes can be adjusted with respect to � Data reliability is increased. the amount of text allowed. � Simplified administration.

Additional Features Additional Features � Data are stored electronically allowing � Faculty may add questions to the instructors and administrators to analyze evaluation specific to a course or section. trends over time. � Individual departments and school could � A system could remind and encourage add questions to the evaluation. students to complete their evaluations. � Program assessment questions can be � A report of the class response rate can be included for learning/accreditation sent to instructors during the evaluation standards. period to allow instructors to encourage student participation. Concerns with an online system Concerns with an online system Challenges Challenges � Response Rate � The student body must be convinced that � The literature shows that student response rates may their information is of value. decrease (although response rates of over 80% and � Incentives must be built into the system to in some cases 100% have been reported using online course evaluation systems). encourage student participation. � The current response rate for paper-and-pencil forms � Faculty must “buy” into this system and is 67%. “sell” this system in their classes. � Response rates generally recover over a period of time, generally thee to four years. � The system must be marketed / advertised � Response rate can be encouraged with p ositive intensively by the college to make it work. reinforcement incentives (e.g. extra credit). Current status Current status Current status Current status � The campus IT department is The campus IT department is unable unable to to � Four vendors have been contacted and/or � assist in the full scale implementation of evaluated including: assist in the full scale implementation of an online system an online system � Equation Research � The FETC in collaboration with the The FETC in collaboration with the � OnlineCourseEvaluations.com � Accountability, Accreditation, Planning & A � WebCT / Blackboard Assessment (AAPA) office has � Digital Measures investigated third party vendors as third party vendors as alternatives to an in house system. alternatives to an in house system. � CourseEval