Monday Matters
Recently I've received a number of questions about the online course evaluation system that we have implemented for student evaluation of courses in the College of Education. I've been jotting down these questions, so now I'm going to take the opportunity to answer these in this column. I'm presenting this in a question and answer format using the questions I have written down as well as a few I've added that I thought you would be interested in.
What is the online course evaluation system and how does it work?
The online course evaluation system is a replacement for the paper and pencil system that we have been using to ask students to evaluate courses and instructors. The questions and possible responses are the same as they were with the paper system, but now students log in via the Web to evaluate a course. When a student logs into the system, they are presented with all of the courses that they are enrolled in so that they can choose to evaluate all of the courses in one session. The entered data is then stored in a database. Once the evaluation period has expired, the data can be directly submitted to an analysis program that generates statistical summaries and provides individual course reports. These reports are sent to instructors, department chairs, and the Dean.
When was the system implemented?
Major development and testing of the online course evaluation system occurred in 2002. In the fall of that year the timeline for implementation was set by Administrative Council. The system was used for distance education courses in the spring of 2003 and all College courses beginning in the summer of 2003. Prior to implementation e-mail messages were sent to all members of the College faculty announcing the timeline.
Who developed the system?
The system was developed staff in the Office of Program Evaluation under contract with the College of Education.
Why not use an existing survey system such as "Flashlight"?
Although existing systems may be tried and tested for certain common functions, there are special functions that we will be building into the survey and analysis phase of our online system that are not present on existing systems. Further, when we subscribe to an existing system we are dependent on an external entity for setting the cost. Although the University currently has a subscription to the "Flashlight" system, there is no guarantee that this will always be the case. Finally, and most importantly, to enable student login into the evaluation system we must put student identification and course information into the database. To put this information on an externally controlled system would be problematic due to privacy laws and would require complex contracts that involve the University Registrar's Office and Legal Department, as well as the external agency.
Is the system secure?
There are two major security concerns with any online system. The first concern has to do with who can access the system and the second has to do with protection of information once it has been entered. To address the issue of access, only students who are registered for a course can enter the system to evaluate that course. They do so by using their name and the last four digits of their Social Security number. This can only be done once for each course. If a student tries to enter the system for that course a second time, they are given a message that they have already filled out the course evaluation. This also helps prevent someone from impersonating the student. If the student logs on and is told that they have already evaluated the course, but they know that they have not, they can report this as potential fraud. A link is provided to a form to file this report at the time the student is blocked from accessing an evaluation form. Also, after making this report the previously entered information from that students name and ID is deleted so that the real student can fill in the evaluation form. To date, this problem has never occurred.
To further enhance security, the system is being altered so that students are required to fill in an email address before completing the online evaluation. Once the evaluation form is completed an e-mail is sent to the student to confirm that the data have been entered in the system. This not only confirms entry of the data but also serves as a check that it was the correct individual that entered the data. System administrators can conduct periodic scans of the database to verify that legitimate email addresses are entered.
The system for keeping these data is actually more secure than the paper-based system. Only authorized computer administrators can access the database. Furthermore, the data are backed up on a regular basis to prevent the loss of data in case of server failure.
Can the Office of Program Evaluation change the questions that are used?
No, each question used on the College form is required by the College, the University administration, or the Board of Trustees. The Office of Program Evaluation is only responsible for administering the evaluation but is not authorized to change the content.
What are the advantages of the online system?
The most obvious and immediate advantage is a cost savings. The paper course evaluation system cost between $13,000 and $15,000 per year. Now that colleges and departments have to cover tuition subsidies, the cost would probably be near $20,000 per year. The College is currently paying about half that amount during the development phase of the online system and will pay even less once the system has been completed. The cost of the paper system was high because of the time involved in cleaning up evaluation sheets to make them readable to scanning system. In addition there were scanning costs and the cost of having the forms printed.
Another advantage to an online evaluation system is the speed with which data can be analyzed so that summative reports can be given to instructors. Once the system is fully developed instructors should have their reports within a few days after the end of the evaluation period. To illustrate the efficiency of the system, consider the course evaluation of this past spring and summer courses. We did not use the system for all of the spring courses but did use it for all summer courses. As a result, reports were actually given to instructors for summer courses some weeks before they were completed for spring courses. In fact, reports for summer courses were distributed soon after summer courses ended.
A third advantage of the online system is that it is fast and user-friendly. We have conducted surveys of students who have used the system. To date, 100% of the responses have been positive. We have asked students to report problems and weaknesses of the system. Other than hearing about a few bugs that were subsequently fixed, we have had no negative reactions to the online system.
Other advantages to an online course evaluation system are that it does not detract from class time and it can actually increase participation in the course evaluation process. (See the question and answer below on participation rate for more information.) I believe that the main advantages of an online course evaluation system are those that are yet to be realized in future versions of the system. I discuss these in answer to another question, below.
What are the disadvantages of the online system?
The primary disadvantage of the system is that the participation rate can be low unless the instructor explicitly implements procedures to increase participation. When the instructor does this the participation rate actually can increase beyond what it was with the paper and pencil method. Another disadvantage of the system is that students have to access the Web in order to complete the online course evaluation. The members of Administrative Council viewed this as a minor concern because students enrolled in the College of Education should acquire a certain level of computer competency, including the knowledge of how to navigate the Web. Even students who do not own a computer can use computers provided by the College and University to complete the online evaluation.
How can the participation rate in the evaluation process be increased?
When Administrative Council endorsed the switch to an online course evaluation system it also endorsed the use of incentives for participation in the evaluation process. Typical incentives include providing bonus points for filling out the evaluation or making the evaluation a requirement of the course. In my summer course I gave students two additional points for completing a course evaluation. This course had 83 students yet the participation rate was 100%. I have never had 100% participation in the course evaluation process for that course, even though I've taught the course the past 12 summers.
For all summer courses the participation rate was low when incentives were not used but was over 90% when incentives were used. The typical incentive of a small number of points awarded for completing the evaluation is a small price to pay for higher participation and validity than is typically achieved with paper and pencil evaluations.
How can an instructor know if students are participating?
There is a feature of the online evaluation system that allows instructors to logon and see a list of students who have filled out the course evaluation. The instructor only sees the names of the students and not the details of the evaluation. This list can then be used to award students points for filling out the course evaluation or to send reminders to students who have not yet filled out an evaluation.
What should a student or instructor do if they have a problem with the system?
There is a phone number provided on the online course evaluation page for reporting problems. Reported problems are given immediate and individual attention. During the testing phase there were a few problems encountered. These were quickly remedied and always to the satisfaction of the student.
Can faculty members choose to use the paper system instead?
No, the paper system has been eliminated. The course evaluation forms were ordered in bulk and that is very costly. We have stopped ordering these and no longer have forms available.
What future developments are planned for the system?
To me, this is the exciting part. Currently the course evaluation form begins with the questions that are required by the College and continues with questions that are required by the Board of Trustees. If the course is a distance education course, questions that the University requires for distance education courses are automatically added to the form. In future developments, this same system of automatic addition will be available to departments and individual instructors. That is, the department could mandate specific questions for all of its courses and these would automatically be included on the course evaluation form based on course prefix. An instructor could ask to have certain questions included on all of the instructor's evaluation forms and these would automatically be added by the system. These questions could be traditional Likert style questions but could also include open-ended questions. Evaluation reports will include an analysis of all the numerically scored questions as well as the qualitative responses for open-ended questions.
Future development is also being planned to make report generation faster, provide automated system backups, and to enhance security even further. All of these developments take time because the College can only invest limited funding to the development of the system each semester. Nonetheless, system improvements will continue until we have met all of our goals.
How can I view the online evaluation system?
There are two ways to see the online course evaluation. First, browse over to the course evaluation system at:
http://edpsych.ed.sc.edu/CourseEval
You can view the online evaluation by clicking on "Survey Samples."
To introduce you to the system I asked the Office of Program Evaluation to create a fake course that included every faculty and staff member. This will allow you to play the role of a student and conduct an evaluation of the fake course. To do this, browse over to the evaluation system and click on "Student Login." (Note: Do not click on "Faculty Login" for this exercise. That will not take you to the evaluation form. For this demonstration you are playing the part of a student.) Just follow the prompts. The fake course is EDEV J900, Section 620.
If you have any questions or comments about the online course evaluation system, feel free to send these to me. If appropriate, I will pass these along to the Office of Program Evaluation staff. Also, tomorrow there is a meeting of the Faculty Affairs Committee to discuss the online course evaluation system. Feel free to give comments or suggestions to your Faculty Affairs Committee representative.
Until next week,
Mike