Funding

  Participants:
  UCLA, UCR

  Principal Investigators:
 Orville L. Chapman, Associate Dean for Educational Innovation, UCLA

This proposal requests funding for the second year of our three-year project to implement Calibrated Peer Review™ [http://cpr.molsci.ucla.edu] in the Life Science core courses and the school-wide Engineering leadership and ethics courses at UCLA, and in both introductory and advanced courses in the Chemistry department at UCR.

Calibrated Peer Review™ (CPR) is a comprehensive, web-delivered, instructional tool, which enables instructors to use frequent writing assignments even in very large classes without increasing their grading load. CPR was developed under an NSF-funded initiative awarded to Orville Chapman at UCLA for curricular reform in chemistry. [DUE-95-55605 Molecular Science: Network-based Instruction.] CPR, however, expands the mandate of the NSF chemistry initiative: CPR can serve any subject. Based on scientific peer review as a model, this instructional tool provides for anonymous on-line peer review of students' written assignments. After submitting their essays, CPR trains and calibrates students to be reviewers of their peers' work. Only when students become competent reviewers and pass the calibration exercises do they evaluate their peers' documents. The final step in this program has students evaluate their own essays using the same criteria they have used for their peers. CPR achieves many important educational objectives. Not only do students understand more deeply when they write about what they are learning, the evaluation process of CPR requires that they develop higher-order, critical-thinking skills. But TA-graded writing assignments are simply not a viable instructional option in the large classes (often over 200) that typify the sciences and engineering. CPR removes that grading barrier.

Learning to write and writing to learn are not equivalent. English composition courses lead to well-crafted prose, which may not be a sufficient condition for deep understanding of complex ideas and arguments in a given discipline. Good writing skills do not guarantee good thinking skills in other subjects. Calibrated Peer Review is first and foremost a "writing-to-learn" device that focuses student learning on thinking about important concepts and content in any discipline. Resnick [Resnick, L. B. Education and Learning to Think, Washington, DC. National Academy Press, 1987] argues that transfer of higher-order thinking skills ["Higher-order thinking is non-algorithmic, tends to be complex, often yields multiple answers, involves nuanced judgement, involves application of multiple criteria, often involves uncertainty, involves self-regulation, involves imposing meaning, requires effort." (Resnick, 1987)]is not easy, and she exhorts educators that if this level of understanding is important in the discipline, it must be taught within the discipline. CPR requires students to use higher-order thinking in each discipline in which it is used.

  Specific Aims:

The long-term goal of our Dissemination of Calibrated Peer Review project remains to introduce and institutionalize the use of CPR writing in the large lecture classes of Life Science and Engineering Ethics and Leadership at UCLA and in the large introductory and advanced courses in Chemistry at UC Riverside. Our objectives to meet this goal form a continuum over three years. Specifically we proposed:

Year 1

• obtain baseline student performance data in each discipline while developing initial assignments
• implement CPR in selected courses
• prepare a summative evaluation of implementation issues
  

Year 2

• expand the use of CPR to additional faculty and courses within the targeted disciplines while developing a large pool of assignments for faculty to draw from
• prepare a preliminary evaluation of student performance
• obtain baseline data of student performance in at least one other discipline
  

Year 3

• complete development of assignment libraries in life science and engineering
• implement CPR teaching in TA training in these departments
• as appropriate, implement CPR in additional disciplines and/or expand to additional UC campuses in targeted disciplines.

We have completed our objectives for Year One and are well into our Year Two objectives in both Biology at UCLA and Chemistry at UCR. We are behind schedule in Engineering, having developed a preliminary assignment only and not fully implementing CPR in Engineering 195. We have re-allocated resources in the second year to complete our Year One goals and to put us back on track for Year Two in Engineering. We will continue to move ahead with our objectives for Year Two and move into Year Three objective as much as possible for Biology and Chemistry.

  Activities and Results

UCLA Life Science 1 (Steve Strand, James Rudd, Kelly Thomas, Nick Manoukis)

Dr. James Rudd, a post-doctoral scholar in Chemistry Education with research experience in the design and evaluation of instructional methods, joined Dr. Russell's group in September. He has guided the Life Science 1 component of this project. This course presents our biggest challenge. The freshman course is enormous, enrolling over 3000 students per year; it is taught by many different faculty; and as presently configured, it often uses only multiple choice tests to assess student understanding. Convincing faculty and students to change depends on strong evidence both that students can and will learn more with CPR (see Evaluation section), that students have positive attitudes towards writing in the LS1 course, and that CPR can be implemented into the course seamlessly by the faculty and TA's. We have made substantial progress on all counts this year.

Implementation depends on adequate instructional materials and technical infrastructure.

Instructional Materials:
Last fall, the life science team designed an outline for a set of CPR assignments based on the five main themes in the course. Variations on the themes were planned to allow for long-term instructor flexibility as well as variability between terms. Their coherent set of 58 potential assignments has provided concrete direction for resource development for the remainder of the project. Full development of the CPR assignments identified in bold in the list below has given us the instructional materials to implement the program both in the honors section of LS1 in Year One and starting immediately in a large 400- student section of the Life Science 1 course this coming Fall.

Evolution by Natural Selection: Variation, Inheritance, Differential Reproductive Success, Exponential Growth, Logistic Growth, Individual Fitness, Blending vs. Particulate Inheritance, Charles Lyell and Thomas Malthus, The Voyage of the Beagle - South America, The Voyage of the Beagle - The Galapagos, Historical Figures, Social Response.

Convergent Evolution: Dolphin, Penguin, and Ichthyosaur; Fish, Whale, and Ichthyosaur; Eels and Snakes; Bats and Birds; Monkfish, Halibut, and Ray; Desert Plants - Cacti and Euphorbs; Natural Selection; Divergent Evolution; Analogy and Homology

Symbiosis: The Continuum of three basic types of symbioses; Mutualism - Corals; Mutualism - Lichen; Mutualism - Mycorrhizae; Mutualism - Nitrogen Fixing Bacteria; Parasitism - Dodder;
Parasitism - Mistletoe; Commensalism - Cattle Egrets and Grazer; Commensalism - The Fat Innkeeper; Ants and Aphids; Ants and Acacias; Natural Selection and Parasite-host Interactions

Biodiversity: The Value to Medicine; Introduced Species - Cane Toads in Australia; Introduced Species - Rabbits in Australia; Introduced Species - Dingos in Australia; Tropical Rainforests and Coral Reefs; Kelp Forests; Endangered Species Act; Causes of Extinction; Habitat Fragmentation; Managing Wild Populations; Effect of Aquaculture; Ecosystem Engineers; Genetically Engineered Species; Trophic Cascades

Community Ecology: Keystone Species - Bison; Keystone Species - Pisaster; Interspecific competition; The Lynx-Hare System; Resource Partitioning - MacArthur's warblers; Resource Partitioning - African ungulates; Character Displacement; Competitive Exclusion - Barnacles; Competitive Exclusion - the European starling; The Role of Fire

Infrastructure:
In the fall quarter, a traditional written homework assignment proved the viability of writing in this course, and the first assignments were developed. Implementation and assessment issues were the focus of the winter quarter. Rudd worked closely with Kelly Thomas, the TA, to develop handouts that would facilitate student use. A 30-minute presentation was sufficient to get students started; individual e-mail technical questions were then resolved.

The CPR program ran without technical problems. The difficulties that the Biology students encountered were not related to equipment but to understanding the features of the CPR program.

Student C: "I don't think I found the peer feedback. When I got my score back, it was really confusing."
Student F: "I didn't know there was peer feedback. If I knew, I would have read it.

This will easily be addressed by revising handouts for next year.

UCLA Engineering 195 (Don Browne, Tim Su, Orville Chapman)

Instructional Materials:
In Year One of the project, the Engineering team worked on developing CPR assignments for implementation into Engineering 195, Art of Engineering Endeavors, a course taught in Fall 2001 and Winter 2002 by Don Browne. A prototype assignment was developed and field-tested in the Winter quarter, but full implementation as a CPR assignment remains for next year. Using our successful experience with the two-step CPR implementation model in Biology in Year One, we will provide the assistance in the remainder of Year One to transform the prototype into an on-line CPR assignment and move directly toward achieving CPR implementation in Year Two.

UCR Chemistry 112 (Mark Midland, Keith Hollis, Leo Schouest, Dan Bernier, Tim Su)

At UCR, CPR has been used in three courses by three different instructors in this year:

Fall 2001:
Chemistry 112A Introductory Organic Chemistry, 328 students, Mark Midland
Spring 2002:
Chemistry 112B Intermediate Organic Chemistry, 272 students, Keith Hollis
Spring 2002:
Chemistry 211 Advanced Organic Chemistry, 10 students, Daniel Bernier

In June 2001, Dr. Leo Schouest participated in three-day CPR faculty training workshop on CPR held at CSU, Fullerton. E-mail correspondence over the summer and a site visit by Tim Su, our CPR technical support, led to the pilot implementation of CPR as an extra credit activity in Chemistry 112A in the Fall quarter. Prior to the Spring quarter we conducted a TA and faculty training workshop at UCR in preparation for the use of the program on a regular basis in Chemistry 112B in the Spring. In addition to Mark Midland and Keith Hollis, another instructor attending the workshop decided at that time to implement the program in his advanced organic course this quarter.

In contrast to Biology, implementation in Chemistry at UC Riverside during Year One involved only infrastructure and no materials development. Existing Chemistry assignments, which were prepared under the original NSF-funded project, matched the course content and could be used without modification.

The support and assistance of the UCR Faculty/Student Technical Services in integrating the program into the existing learning-management systems has facilitated student access. We will recommend this model to other institutions adopting the program.

The major technical problems that we encountered at UCR in the Fall quarter happened to students who did not have high-speed internet access when they tried to run the animations associated with the content of the assignment. The following kinds of comments were made by a small percentage of the students on an e-mail survey giving the instructor their feedback to the program:

"The chime plugin was a little problem because I too use netscape 6.1 so I ended up using Internet Explorer 6.0. It handled the plugin well and did not give me any problems. The web site performed decently with a 56K modem connection, except the chime illustrations took about 30sec to 45sec to load but acceptable. Now that the calibrations are available the calibration screens seem very user friendly but I would tell everyone that it is best to use a screen resolution of at least 800x600 because there is so much to info to display. The questions take half the screen and the essay takes up the other half."
"My name is XXX, and I'm writing this letter in relation to the CPR assignment. I personally did not have any general problems with it. One of the main difficulties I had was with downloading the program needed to observe the 3-D structures. Overall, everything was easy to understand."
"There was one thing that caused difficulty and that was the installation of the CHIME program."

To address this issue in the future, we will recommend that special equipment requirements for individual assignments be made very clear to students on the syllabus and that they are aware of convenient resources on campus if their own computer systems are not adequate.

Training TA's in CPR is essential. The workshop before the Spring quarter had a large impact. We realize now this cannot be a Year Three goal as we had originally planned and will ensure that it is in place in every course where CPR is being started. The completion rate of assignments depends on students learning how to log in and being encouraged to do the assignment by their TA's. At UCR this Spring, 90% of the students wrote and entered their CPR text; this is 20-30% higher than we normally expect.

  Assessment

Note: In order to obtain information on student attitudes and performance that we can publish in research journals, the University of California requires that all researchers obtain "human subjects approval." Last Fall, Dr. Rudd obtained this approval from Office for the Protection of Research Subjects (OPRS) for evaluation of the LS1 component of our project; we will work with the Center for Teaching Excellence at Riverside to obtain permission for a similar Chemistry study.

Student Attitudes and Assessment of Their Learning Gains:

In both Biology and Chemistry student attitudes were overwhelmingly positive towards CPR instruction.

Life Science 1
In the honors section of LS1, 96% (25/26) of the students stated that CPR positively affected their learning of course material. They cited three main reasons for CPR's effectiveness:
(1) CPR helped their understanding of topics:

Student K: "It (CPR) made me understand the topic more. I think in LS we're more concerned about memorizing because it's about how much you can remember. You don't necessarily go into thinking anything deeper than just what you need to know or what the definitions are. I think writing essays made me do that (thinking deeper) more because then I actually had to think about whatever topic we were covering."
Student M: "I think it (CPR) was really beneficial for LS, especially LS1, because you have so many topics to learn and there's so much information thrown at you at once. It really helps you identify the major ideas. It gives you some sense of the big topics that you are introduced to, to get a really good understanding of those."
Student V: "We were seeing connections between things. It (CPR) seemed to connect different ideas. It made us think."

(2) CPR increased feedback by using current technology:

Student U: "At a school like UCLA ... [with] classes that have hundreds of people in them, it's really hard to get feedback on your work. The feedback aspect is probably one of the most important parts of learning, and you get that in CPR whereas you don't really get that as much [typically]. With a TA it's like "Okay this is wrong, but what's the right answer?" You can go to office hours and talk to them, but it's more convenient if it (the feedback) is just right there on your computer. And most people don't go to office hours."
Student L: "You wouldn't be able to do this ten years ago; you didn't have the computer to be able to put it (your work) in and have other people look at it."

(3) CPR provided an opportunity to write about science:

Student S: "I've never really had experience in writing about science like this. Usually when you write in an English class, you write on that kind of topic. I enjoyed writing about science."

Chemistry 112
While we have not completed the quantitative analyses of the hundreds of responses that students in Chemistry provided at UCR, their comments on the voluntary feedback survey in the Fall quarter reiterate the Biology students' comments.

(1) CPR helped their understanding of topics:

"I thought CPR to be good learning tool. The project made the participants review the course material. As a result a better understanding of the material is attained.... It's like having two lectures instead of one. Making an environment where students have to think is a good way to make the material understandable."
"As to what I thought of the CPR program I actually think it was worthwhile. It did help me remember much of what I learned about alkanes and to put it into more manageable groups."
"I thought that the CPR was really interesting. I liked the fact that they made us look at 3D models before we could answer the questions because it gave us a good idea of the concept involved in the question. I think that the process we had to take before writing the essay was great because it took us step by step to the answer, allowing us to thoroughly comprehend the strain (angle and torsional) in cycloalkanes in comparison to their alkane counterparts and to other larger or smaller cycloalkanes.... I also like that we will get to grade other students' essays and later see if our own essays were of good quality. I believe that it will let us know if we really understand the part that strain plays in cycloalkanes."
"I think CPR is a great program that helps us understand the material that we have covered in the lecture and labs in a more comprehensive way."
"I think CPR is an excellent way for students to acquire a deeper understanding of a subject. It requires a lot of critical thinking that the students must do in order to truly understand the subject in question."

(2) CPR increased feedback by using current technology:

"This program is similar to something that my girlfriend is doing in her Christian theology class at BIOLA. She writes a 400-word response to a "poll" question based on the 400-500 page reading assignment that week, every week. Then she reviews the work of three others (but not her own), followed by an in class discussion once a week. Of course, an in-class discussion is not appropriate for a chemistry course and the material covered requires more work than a simple read and response. This is why I feel that the CPR program would be more beneficial if it were assigned more than once a quarter. Something more like one assignment every 2-3 weeks. I really feel that this would improve the overall information retention of the class."
"I think that doing CPR gives good feedback to students."
"I found CPR extremely useful as a tool for increased understanding of conceptual material, especially considering the limited time we have with the professor in lecture."

(3) CPR provided an opportunity to write about science:

"By writing this assignment in a mini-essay form I was able to come at my own conclusions and express the material in my words. I believe that this was very helpful because in order to understand the material it is extremely useful to be able to describe and put the material one learned into his/her own words."
"I enjoyed the CPR assignment. I liked having another format to think about organic chemistry. If you do this again, I think optical rotation would be a great subject. I thought the writing assignment was a good length--not too much to read six examples of writing."
"The Calibrated Peer Review (CPR), in one simple description, was one of the best experiences ... this quarter. I have never had an integration of writing in essay form in the arena of science."
"I thought this assignment was very helpful and useful. As science majors we don't have very many chances to work on our writing skills."

Not all students were positive. Many only begrudgingly acknowledged there were benefits to the assignment:

"I thought the idea of CPR was a very good idea because it helped expanding students' knowledge and developing insight into the materials we had studied. However, I didn't really enjoy this project. I didn't like it because there was too much research involved and questions were very hard to answer. This project itself was time consuming as well. It took me a long time to finish the practice before I moved onto the actual project. Another time-consuming part was the peer reviewing. It took me about an average of 10 minutes each to review one paper. It required a lot of concentration as well."

Student Performance [James Rudd, Summary Report: CPR Implementation in LS1 Honors Winter 2002 (April, 2002)]

Biology:
Having received OPRS approval in the fall, Dr. Rudd collected and analyzed the winter LS1 student performance on the two mid-term exams and the final exam. The multiple-choice exam questions were categorized as (1) relating directly to the topics in the CPR assignments, (2) requiring transfer of understanding of CPR topics, or (3) addressing non-CPR topics.
Comparing student performance on questions that directly or indirectly related to CPR topics with performance on questions covering non-CPR topics, students generally performed higher on CPR topics than on non-CPR topics (see Figure 1). The mean performance was 84.8% on CPR topics and 79.7% on non-CPR topics. A paired t-test indicated the higher performance on the CPR topics was statistically significant (df = 25, t = 3.035, p = 0.006).

Figure 1. Student performance on multiple-choice exam questions vs. student course performance (paired t-test results: df = 25, t = 3.035, p = 0.006)

This data is consistent with the findings that Pelaez reported last year. [Nancy Pelaez, Biology 310, CSUF 2000]

It is interesting and important to note that in both studies the students at all levels perform better on the topics covered with CPR asignments than on the topics not covered with CPR asignments. In both cases, the largest gains are for the weaker students. This may be attributable to the obervation noted by Dr. Hollis in his e-mail message below. CPR focuses the students on the relevant aspects of the topics.

Chemistry:
At this time, the only indication we have of student learning gains from Chemistry is anecdotal. Shortly after the beginning of the first CPR assignment in his class this quarter, Dr. Hollis sent the following e-mail:

"What is clear to me from the questions that I have taken is that the students are engaged with the topic! It is presented in a form that they like (i.e. they get to play with their computers) and it is a very directed study forum. I wish that I had hundreds of problems presented through an internet interface like CPR, covering the entirety of the material in my course. Computers and the internet have been around long enough and there is enough familiarity now, that the technology no longer seems to be a hindrance to study. Instead, it engages the students through a medium with which they are familiar and through which they are receptive to instruction and learning.'" K.R. Hollis, April 2002

We will obtain baseline student performance data from his course this quarter and begin preliminary evaluation this summer.

  Faculty Roles

As indicated throughout the narrative, the faculty we identified last year in our original proposal have been actively involved in Year One and are committed to continuing their implementation of CPR in the coming year.

The LS1 sections in Fall and Winter quarters were taught by Steve Strand, the Chair of Life Science at UCLA. Strand has provided guidance to the team in the development of the assignments and the handouts. As a start to his goal of full-scale implementation of CPR in all sections of Life Science, he will teach the large 400-student section of this course in the Fall.
Steve Jacobsen, Associate Dean of Engineering and Applied Science, continues to endorse CPR and support his faculty's interest in its development. In addition to working with Browne, we plan to assist Joe Miller, who also teaches Engineering 195, in implementing CPR. Once assignments are in place, the barrier for new faculty is much lower.
Without question, UCR is a success story. Not only did Mark Midland and Keith Hollis implement CPR in their courses, another organic chemistry instructor, Dan Bernier, participated with his advanced course. TA training-a Year Three objective-was also put in place here. The institutional support from the Chair, Michael Rettig, as well as from Leo Schouest, who manages the Faculty/Student Technical Services, ensures stability and growth at UCR and serves as a model for other schools. Both Midland and Hollis have expressed interest in developing more assignments for their classes in the coming year.

  Year Two and Three Detailed Plans

Our plans for Year Two require only slight modifications to our original proposal. Year Three remains the same at this time.
In Year Two of the project, as we move to other and larger courses, we will need to build more libraries of new assignments. Our emphasis will shift in the second year from technical support for student implementation to editorial support for carefully written versatile assignments. Based on the ease with which we UCR was able to implement CPR, we will collaborate with the Center for Teaching Excellence to assess the interest of other departments in CPR and begin to work with them as appropriate. Biology is a logical new target department here, since we now have a large and growing core of Biology assignments available. If the interest exists a parallel structure to that used in Chemistry in Year 1 can be implemented with Biology in Year 2.

In the third year, we will focus on discipline structures. As Associate Dean of Educational Innovation in the College of Letters and Science, Orville Chapman, PI, will take particular responsibility for the national discipline acceptance and viability of CPR by the end of the TLtC project. On the recommendation of several NSF program officers, we have submitted a proposal to the National Science Digital Library solicitation to guarantee the continuation of the program after TLtC funding ends. The senior biology editor of W. H. Freeman has visited UCLA and is working with her textbook authors on ways to include CPR Biology assignments in their curricular materials. Textbook-publisher backing places a national sales force and dissemination tool behind the program. In Engineering we will seek external backing from the Engineering Accreditation board. Both Don Browne, who co-teaches with Joe Miller, and Dean Jacbosen believe that CPR will receive ABET approval for the requirement for writing in the engineering curriculum.

  Evaluation

Arlene Russell will continue to oversee the evaluation of both implementation activities and student outcomes.

• She will work with UCR to put in place an appropriate evaluation of the year-one student data that will be available in the summer. A revised evaluation of the Engineering activity will be developed in conjunction with Browne and the other Engineering 195 faculty. Dr. Rudd will continue to work with her in assessing the impact of the CPR writing-to-learn process on student understanding in the discipline.
• She will develop a revised evaluation of the Engineering activities in conjunction with Browne and the other Engineering 195 faculty.
• Dr. Rudd will continue to work with the LS1 team in the evaluation of CPR in this course in terms of the scale-up and student outcomes
• Dr. Rudd will continue his research with Dr. Russell in assessing the impact of the CPR writing-to-learn process on student understanding in a discipline.

  Project participants

• Orville L. Chapman, Associate Dean for Educational Innovation, UCLA
• Arlene A. Russell, Department of Chemistry, UCLA
• Steve Jacobsen, Associate Dean, School of Engineering and Applied Science, UCLA
• Steve Strand, Chair, Life Sciences Department, UCLA
• Michael Rettig, Chair, Department of Chemistry, UCR