Patton, M.  Utilization-focused evaluation. Beverly Hills CA: Sage Publishers, 1978. 

Perna, L. W.  Early Intervention Programs: A New Approach to Increasing College Access.    Paper presented at the twelfth annual NASSGP/NCHELP Research Network Conference. Ann Arbor, MI: University of Michigan, Center for the Study of Higher and Postsecondary Education, June 1995.

Richardson, J.  Colleges recruit minority students.    Sacramento, CA: The Sacramento Bee, September 15, 1996.

Richardson, J. & Ferris, J.  Quest for diversity takes colleges down new paths.  The Sacramento Bee, September 15, 1996.

Rochefort, D. &  Cobb, R.  "Problem Definition: An Emerging Perspective" in D. Rochefort and R. Cobb (eds.), The Politics of Problem Definition: Shaping the Policy Agenda., 1-32.  Lawrence, KA: University of Kansas Press, 1994.

Rodriguez, E. M.  State-level education reform: Collaborative roles for postsecondary education.    Denver, CO: State Higher Education Executive Officers, April 1994.

Scheirer, M.A.  "Program theory and implementation theory: Implications for evaluators." in L. Bickman (ed.), Using program theory in evaluation.  San Francisco: Jossey- Bass, 1987.

Smith, M. F. Evaluability assessment: A practical approach.  Boston: Kluwer Academic Publishers, 1989.

Stoel, C., Togneri, W. & Brown, P.  What works: School/college partnerships to improve poor and minority student achievement.    Washington, D.C.: American Association for Higher Education, 1992.

St. John, M. et al.  Evaluating a statewide professional development system.    Report 9.  Inverness, CA: Inverness Research Associates, 1995.

Stevens, H. S.  A Summary Report of the ACCESS Program in the Oakland Unified School District for the 1994-95 School Year.    Berkeley, CA: The Lawrence Hall of Science, August 1995.

Tierney, W. G.  The parameters of affirmative action: Equity and excellence in the academy.    University of Southern California, Center for Higher Education Policy Analysis, May 1996.

University of California.  Directory of outreach programs,  January 1996.

Underwood, C.  Making the future different.    University of California, Office of the President.

University of California.  Latino student eligibility and participation in the University of California.    Report number three of the Latino Eligibility Task Force, July 1994.

University of California, Office of the President.  Early Academic Outreach Program, Fall 1994,  July 1995. 

University of California, MESA Statewide Office.  MESA success rates,  July 1996.

University of California.  Preserving student diversity.    Report of the UC Berkeley Outreach Task Force.  Berkeley, CA, February 1996.

University of California, Office of the President.  Problem statements and strategies on eligibility rates.    Generated from the UC systemwide outreach retreat, July 1996. 

University of California.  The schools and UC: A commitment to the future of California.  A guide to the University of Californias pre-collegiate programs,  1995.

University of California, Office of the President.  The use of socio-economic status in place of ethnicity in undergraduate admissions: A report on the results of an exploratory computer simulation.    Occasional paper 5, May 1995.

University of California, Office of the President.  Undergraduate persistence and graduation at the University of California,    Parts I, II, & III, November 1994.

University of California, Outreach Task Force.  Summary notes,  February 1,  March 5, April 15, 1996.

University of California, Outreach Task Force.  Draft outline of pre K-16 subcommittee report. 

Wilbur, F. P. & Lambert, L. M. Linking Americas schools and colleges: Guide to partnerships and national directory.  Second edition.  Boston: Anker Publishing Co, 1995.     

______________________________

¹ Of particular help was a matrix generated at a UC Systemwide Outreach Retreat entitled Problem Statements and Strategies on Eligibility Rates,    July 1996.
²  The comparative four-year rates in 1994-95 were: Blacks-27.8 percent, Latinos-23.7 percent, whites-10.8 percent and Asians-8.7 percent.
³  More than 1 in 2 of Asian students  complete a-f requirements versus about 1 of 3 white, 1 in 5  Latinos and a little over 1 in 4  Blacks.  Similar patterns exist for advanced science and math enrollments.  White students are more than five times as likely to have taken advanced placement courses as Blacks and almost twice as likely as Latino students.
⁴  Almost 1 in 3 Asian students take the SAT or ACT, versus slightly over 1 in 5 whites,  an abysmal 1 in 14  Blacks, and an even worse 1 in 17 Latinos.
⁵  Saul Geiser, presentation to Outreach Task Force, February 1, 1996.

⁶ It is important to note that these categorizations describe programmatic activities at the local level, not whole programs.  Most local outreach programs will include some combination of program elements, however these elements are organized.  A  useful categorization scheme will allow researchers to classify programs based on their particular programmatic emphases. 
⁷ To simplify the discussion, we focus on the effects of programs on individuals.  However, the logic of evaluation design also pertains to programs, organizations, collaborative arrangements, communities and any other level where a social intervention is implemented.

^For a good discussion of the problem of causality in evaluation research see Kirst (1986).  Kirst argues that an evaluator or policy maker's view of causality follows from her definition of the problem and is ultimately a value judgment.  Kirst's suggestion is to combine program theory with "bottom up" studies that examine the implementation of programs at the local level.  Only by combining top-down and bottom-up research strategies can a realistic model of causality be developed.
⁹ This cost-analysis section is an adaptation of Henry Levin's Cost Effectiveness:  A Primer (Sage Publications: Beverly Hills, 1983).

¹⁰ This approach can be explained best through an illustration.  In a hypothetical study, three vocabulary enhancement programs are offered to separate classes of 11th graders with the intent of improving the students SAT scores.  Program A gives the students two vocabulary textbooks to study at a cost of $30 per student.  Program B uses the same textbooks but also employs a teacher to review the words once a week for five weeks at a cost of $100 per student.  Program C uses computers to teach the students vocabulary words.  The cost of the computer time, the computer programs, and teacher supervision totals $200 per student. Table 3 depicts the costs of each program along with the average gain on SAT scores of students enrolled in the program.

Table 3
Hypothetical Cost-Effectiveness Results for Vocabulary Enhancement Program

Cost  Improvement CE Ratio 
Program Per Pupil Points
   
    A $  30        5 $6 per point
    B $100      33 $3 per point
    C $200      40 $5 per point

The third column displays the cost-effectiveness ratio or the cost per student for a one-point improvement.  The hypothetical results show that Program A costs the most for every point of improvement, while Program B is the most cost-effective.  Significantly, the most effective program when costs per student are ignored (Program C) is ranked second when costs per point are taken into account.  Hence, a greater state or local investment in Program B is the preferred policy.
¹¹Comparative research can be done with random assignment if different program types or program components are identified and students are randomly placed in each.  In such a case, one program becomes the "treatment" and the other the "control" group.  The benefits of such an approach must be weighed against the difficulties of doing random assignment research noted earlier.

¹² A number of common types of evaluation research may be less helpful in answering the questions California has about its outreach programs.  A theme throughout this study is than random assignment is difficult for outreach programs and the benefits these studies provide may be outweighed by the difficulties in conducting this type of research.  Random assignment has its place in outreach program evaluation but the costs and benefits of such research should be carefully considered before being undertaken.  Constructed groups, such as used in Florida's CROP program, or a different research design may be able to better answer the questions asked in a given study.  A common quantitative methodology for determining what works in a policy domain is what statisticians call meta-analysis.  Traditional meta-analysis takes a large number of studies of a particular intervention and attempts to aggregate these individual findings into a collective result.  One difficulty in using meta-analysis with outreach programs is he paucity of evaluation research on these programs available.  Unlike some social policy arenas, higher education outreach programs, to this day, are under-researched and under-evaluated.  Such a situation makes meta-analysis very difficult.  A second problem comes from the assumptions of meta-analytic technique.  This research approach assumes that all studies have commonly defined program components, ask the same research question with the same research procedures and possess a set of common performance indicators or dependent variables (Knapp, 1994).  Such an assumption is far from tenable given the evaluations collected and analyzed in this report.

¹³  1990 is the latest available eligibility study undertaken by CPEC.  Currently, an update is underway.  Obviously, the newest data should be examined to ascertain whether these conditions still exist and whether such a strategy would still be viable.
¹⁴  All students are required to take the SAT or ACT.  However, for students with sufficiently high grade point averages, the score they receive is irrelevant.  Hence taking the test would make them fully eligible.
¹⁵  In Appendix B, we provide a set of questions to assist in designing evaluations tied to the short-term, intermediate term and long-term strategies.

¹⁶ Crafting effective counterfactuals is not a simple as we have laid out here and poorly constructed questions can bias results. 
¹⁷ We have greatly simplified the difficulties involved in random assignment in this section.  Waiting lists, for instance, may be biased if the treatment group is selected from early applicants.  These students or their parents may be more motivated than late applicants, opening up a  source of potential bias.  This is only one example of the many difficulties in conducting evaluation research with random assignment.  For a introductory discussion of these issues, see Babbie (1992).

¹⁸ A waiting list approach, with interested student matched with each other and put into comparison groups may address this motivation bias.  As pointed out above, waiting lists potentially have their own bias attached to them.

BACK  |  HOME  |  NEXT