Essential Principles in Evaluating Outreach and College Preparation Programs

In this section we take a step back from the specific program evaluations to review the lessons and experiences from over 30 years of evaluation research and the implications for the evaluation of higher education outreach programs.  Our purpose is to identify the important considerations and essential principles in evaluating outreach programs and other programs to improve college preparation of disadvantaged students.  These principles may be applied both to student-centered and to school-centered program evaluations.  Evaluation of school-centered and systemic programs is much more complex, however, especially in attempts to ascribe changes in student performance to the interventions provided by the program.

1.  A student information system is needed for program evaluation.
An essential principle in all program evaluation is that good evaluation requires valid and reliable data.  In evaluating outreach programs accurate information is needed on the numbers of students participating in programs and the benefits that accrue to these students from the services provided.  These programs have both immediate and longer term goals (e.g., eligibility for the University).  Students may participate in more than one program, either concurrently or sequentially.  While evaluation of the individual programs could be improved through more uniform and rigorous data-collection procedures, we believe that a longitudinal student information system is needed to evaluate the separate and collective effectiveness of student-centered programs.

2.  Program goals and intended outcomes must be defined.

Evaluators ask questions about specific sets of desired social ends.  Before answering, or even asking, these questions, we must determine the ends we wish to achieve.  This may seem an obvious point, but in most policy domains the answer is far from clear.  Such is the case with higher education outreach programs.  Though most observers would agree that the ultimate goal of outreach programs is for students to attend and complete college, an evaluation strategy that focuses exclusively on the outcome of college completion would be extremely limited.  The reason for this is that college completion sits near the end of a long chain of actions students must undertake beginning long before they are even able to apply to college.  Students need to stay in school, take strong academic courses, do well in these courses and in placement tests, apply to college, and get a degree.  Outreach interventions target students throughout this chain, often focusing on different links of the chain.  A useful distinction, therefore, can be made between three types of outcomes in higher education outreach programs:

a) Short-term outcomes include improved performance in elementary school, increases in school persistence, increased college entrance test taking, and increased a-f course taking in school.

b) Intermediate-term outcomes are the consequences of this first set of student actions: higher grade point averages, better a-f course performance, improved scores in entrance examinations, improved eligibility rates for higher education, improved rates of college attendance and reduced remediation needs.  An additional intermediate-term outcome might be successful collaboration among elementary, junior high, secondary schools and higher education institutions.

c) Long-term outcomes lie at the final end of the chain: higher education performance and graduation, career attainment, life-long earnings, and graduate school performance.  Also, improved secondary schools for all students might be a long term outcome for some types of outreach programs.

One could add additional outcomes to this list but would be hard pressed to remove any of these "desired ends" from the list of outcomes.  Desirable outcomes can be converted into performance indicators, descriptions of what can be observed that will signal achievement of activities, objectives and goals" (Smith, 1989 pp. 6), which can provide a comprehensive list of desirable ends for outreach programs.  They differ from the broad statements of mission, goals, and outcomes these programs often have in three ways.  First, they are measurable.  Second, the indicators follow directly from program activities, offering a way to assess whether programs have successfully and/or faithfully implemented their proposed tasks.  Third, performance indicators can be arrayed in a hierarchy from short-term to long-term objectives, allowing evaluators to see how each part of the chain contributes to the ultimate program or policy goals.

A number of lessons can be applied to the evaluation of outreach programs.  First, programs can and should develop clear and measurable performance indicators.  Additionally, programs should be clear about the part of the chain their program activities focus upon.  While most programs will claim the ultimate goal of student success in college, some programs conduct activities closer to this long-term goal than others.  With this in mind, programs might focus their evaluations on those indicators that they can reasonably be expected to address and measure.  The linkages among performance indicators and the overall effectiveness of programs can best be assessed where those indicators are most easily found, at the university or system level.

3.  Program interventions must be clearly described and implemented.
All programs and policies have more or less explicitly defined programmatic components or elements that are believed to bring about the types of "results" we discuss in the previous section.  Again, determining what these variables are for outreach programs is more complex than it seems at first glance.  Outreach programs involve a wide array of components, ranging from academic support and counseling, to curricular reform, to teacher development.

The condition of existing evaluation research on outreach programs offers little guidance in determining which of the multiple components are more or less useful.  Evaluation researchers suggest organizing and categorizing program components into ways that allow for systematic inquiry (Patton, 1986).  One categorization of program activities might distinguish those efforts that are student-centered (counseling, academic support, test preparation, motivation activities) from those that have a school-based focus (curricular and assessment reform, professional development of teachers, K-12 and higher education collaboration).  Another potential classification might be between program components that emphasize cognitive change in students and those that focus on effective factors that contribute to student college going.  Other key dimensions of outreach programs that might be studied include program time and intensity, targeted grade levels, and others.

In short, the array of program elements in outreach programs needs to be geared toward building up cumulative insights from evaluation research.  A working categorization of program dimension could be developed so that all actors in outreach programs at the system, campus, and school levels are aware of it and coordinate around it.  This does not mean that programs will all emphasize the same program dimensions.  Rather, the goal is to have a standard template that allows for the insights and results from local programs to be described and potentially compared.

BACK  |  HOME  |  NEXT