This tried-and-true process has been a paper-based enterprise for decades, the most cutting-edge advance being the introduction of Scantron forms to automate the collection of quantitative data. Even still, the process is labor intensive - it can take months to complete and requires significant staff time to perform.

Now many universities are wondering if course evaluations - like plenty of other academic and administrative endeavors -- wouldn't be better handled in pixels instead of on paper. This approach would have students complete evaluations on the web, on their own time, freeing up the class time usually taken for the evaluation process. Attractive to administrators are the potential efficiencies in staff resources to collect and disseminate the evaluation data. Instructors appreciate the rapid reporting of results so that, if necessary, they can make changes to their courses immediately.

Despite these advantages, the adoption of web-based evaluations has been slow. A national study conducted in the fall of 2002 found that 90% of the nation's "most wired" colleges still used a paper-based process. However, the 10% using online systems represented an 8% increase from 2000, thus there appears to be a gradual movement toward the online process.

Reasons often cited for not adopting the online process include concern about low response rates, and, as a result, potentially biased data (based on the assumption that only those students who had strong feelings, particularly negative, would be motivated to complete the evaluations on their own time.)

Recent research, as well as localized experience on a few UC campuses, provides some reassurance that online evaluations can work, although they must be implemented differently than their paper counterparts. First and foremost, because online evaluations are not done in a classroom where there is a captive student audience, an incentive must be offered to ensure high participation rates.

Examples of incentives that instructors have given include: providing extra credit; entering students in raffles for prizes, such as Palm Pilots; communicating with students early and often about the evaluation process and how the data will be used; and withholding grades until students submit their online evaluations.

"The first time we did evaluations online we did not offer an incentive and we had a very, very poor response with only a few students responding," says Rudi Berkelhamer, a professor in the Department of Ecology and Evolutionary Biology at UC Irvine, who began using the campus's online evaluation system in 2001. "After that, I offered extra points and got very good compliance."

She found that by putting the evaluations online, and offering students an extra two points for completing them, she was able to get more students to participate than she could when she conducted evaluations on paper during class period. (On any given day, the 300-student course, Life Sciences 1A, has poor attendance, says Berkelhamer.)

Faculty and administrators who have used online evaluations will tell you that incentives are critical to making the process work. They will also tell you that it's worth moving to an online system, as students provide better feedback, including more thorough written comments, than they do on paper-based evaluations.

"My students seem much more interested in providing detailed comments and not just checking off boxes as they seemed to do on the paper evaluations," says Berkelhamer. "I used to get a handful of comments and now with the online system I get 30 pages. I receive more concrete suggestions for what kinds of things students would find helpful." Berkelhamer speculates that students prefer writing on a computer rather than by hand and that they feel less rushed to fill out the form when doing it on their own time.

Administrators at UCLA medical school have also seen improvement in the quantity and quality of written comments in the five years that they have used an online evaluation system (called CourseEval). "Written comments have increased dramatically," says Gary Diener, Assistant Director of Evaluation. "I would say that 80% of students write comments and they are longer and more detailed than the comments that they wrote on paper."

Diener says that their response rates are almost 100%, compared to a 50-60% response rate with paper evaluations. This statistic is not too surprising because UCLA medical school has made online evaluations mandatory, giving students an incomplete grade if evaluations are not submitted. However, Diener says that in the five years they have had this policy, they have never had to give a student an incomplete.

Diener credits the success of the online system with the school's careful planning process, which included soliciting student input. "The medical education committee, which includes student representation, sat down almost two years before we adopted this system and drafted a policy," says Diener. "Students were part of the decision making process from the beginning."

UCSF medical school started conducting course evaluations online (using Evalue) in 2001 in order to create a centralized system for students in all four years of the program. "Doing it by paper is much more tedious, especially because we have students doing clerkships who are spread out all over the place," says Carol Hodgson, Director of Educational Research & Evaluation.

UCSF does not mandate that all students complete evaluations. In fact, students are randomly assigned to provide evaluations for certain courses, so that they don't have to fill one out for every course. However, the class as a whole must complete 70% of their evaluations or else their grades and exam scores will not be posted.

"You have to be careful with how you ask students to spend their time. You don't want to overburden them with too many evaluations," says Hodgson.

UCSF also encourages participation by educating students about the evaluation process and how the data will be used. Dr. Hodgson gives an introduction to the students at the beginning of the first and second year about how to give constructive feedback and the school provides the results back to the students after the reports are created. "Until you tell students that people actually read these, they don't realize that they can hurt faculty in tenure and promotion discussions," says Hodgson. "When they realize that I think they are very responsible in writing their comments."

Thus far throughout the UC system, professional schools, such as medical and dental programs, have been more likely to use online evaluation systems than undergraduate programs. The only UC campus to implement an online evaluation process for use by the entire campus is Irvine. The campus built their online system, which can be used for both midterm and end-of-term evaluations, into their existing Electronic Educational Environment, better known as EEE, which is already used by faculty for course management.

However, participation rates, compared to those with paper-based evaluations, are relatively low. For the 2004 spring quarter, only 16% of courses that used online end-of-term evaluations had a response rate of 50% or more, compared with an average response rate of 60-75% participation for paper evaluations, says Shohreh Bozorgmehri, of UCI's Network and Academic Computing Services.

"We have learned that if faculty do not provide incentives, or encourage student participation, they do not get high participation," Bozorgmehri says. "However, faculty who do a good job of informing and reminding students about evaluations and encourage participation get good response rates."

Another concern with online systems is whether they can ensure anonymity for students, especially if instructors are tracking them in order to give credit for submissions. With paper forms, anonymity is typically ensured by having a staff person type in the written comments so that an instructor cannot tell by the handwriting who wrote which comment. The downside is that this practice takes significant time and resources.

A common practice to ensure anonymity with the online systems is to use two databases that separate the login information from the actual feedback. The system tracks who submitted an evaluation in one database and what they wrote in another so that the two cannot be correlated.

"It's important to us that students not feel inhibited in what they say by a fear of lack of confidentiality," says UCI's Berkelhamer. "It turns out that the online system is even more anonymous than the handwritten evaluations."

Although anonymity issues are somewhat easy to deal with, there are other challenges to online evaluation systems that may prevent or hinder their widespread adoption. The startup cost of purchasing or developing a system that can handle thousands of students and hundreds of courses may be prohibitive in these budget times, even though the system may result in cost savings down the road. Some faculty may have ethical concerns with providing incentives, such as assigning extra credit to students for completing evaluations, or providing disincentives, such as withholding grades if students don't complete evaluations. Whether the campuses will weigh in favor of the benefits or the challenges of online evaluation systems remains to be seen.