4.  Evaluations must be carefully designed to attribute results to interventions.
Perhaps no issue is more hotly contested in social research than the issue of causation.  Evaluators spend an extensive amount of energy trying to attribute results to social programs.  The main difficulty comes from the very artificiality of evaluation research.  In designing an evaluation, evaluators abstract certain features of programs and people from the real world.  In attributing causality, we must deal with these external influences on people or programs.  To claim a program or a set of program components were successful, we must ask what other factors may have contributed to the observed result.

In outreach programs, when we observe a participant going on to college, can we say this was caused by the program she participated in, or  by other student attributes such as motivation or previous academic preparation?  Even if students change during the course of the program, how do we know that the change was brought about by the outreach activities?  Perhaps the student has just gone through a period of personal development.⁷

Even if one is confident of the estimates of causal effect obtained after controlling for extraneous variables, this may not tell us why and how these effects come about.  What is it about the interaction or sequencing of program components that causes a desired result to occur?  Analysts who have taken this second path to address the causality issue try to answer questions like these by developing more explicit theories of designed policies and programs.  In evaluation research, a program theory is a means-end hierarchy, a cause and effect linkage of program activities with outcomes" (Smith, 1989 pp.5).  It is an idealized model that shows how program features interact to influence performance indicators, and produce desired outcomes.

A program theory contains three basic elements.  The first can be called the problem definition (Rochefort and Cobb, 1994).  All policies and programs presume a certain definition of the problem that guides the intervention.  In outreach programs, for instance, one can conceptualize the problem as one of student characteristics (motivation, ability, academic preparation), school characteristics (poor curriculum, poor counseling and support for students), higher education admissions policy (eligibility standards, coordination of K-12 and higher education) or some combination of these and others.  Different problem definitions will inhere different programs with different emphases.  A first step in clarifying the program theory is to make explicit this problem definition both within and across programs in a given policy domain.

Problem definition leads to a set of program components.  In the second element of a theory, these components are linked together to create a program logic (Smith, pp. 53).  The program logic maps how the components fit together to produce program objectives or goals.  The simplest way to represent program logic is with a set of if-then statements that link the components together.  In outreach programs, if students participate in the program then they will stay in school and get better grades.  If this occurs, then they will take harder courses, improve their test scores and so on.  How far one goes in completing the chain depends on program goals which themselves follow from the definition of the problem.

The third element of a program theory links program logic to program activities through the use of performance indicators.  For each component of the program, indicators are developed to show whether the program has successfully met all of its objectives.

Below each if statement is a performance indicator that measures whether that step has been successfully achieved.  To determine whether students stay in school, drop out rates will be compared.  If these are reduced, in the next step in the chain, better grades can be examined.  When long term indicators such as college attendance are not successfully achieved, program theory allows the evaluator to reconstruct a sequence of events and determine where in the chain the program may have fallen short.  This type of evaluation can be done for individual programs or whole systems of programs using the same tools.  In some cases, the weak link may be a program component that was not implemented.  In other instances, all components may be implemented but a flaw can be detected in the program logic or problem definition.  In both cases, program theory is used by evaluators to answer the difficult questions of why and how a set of program outcomes occurs.⁸ 

In summary, the choice is not between control strategies and program theory both are necessary components of developing causal inferences in evaluation research.  Even with the most rigorous controls, an evaluation that cannot tell us why or how outreach programs are successful is of little use to policy makers or practitioners.  Similarly, the most elegant program theory will be of little use if we cannot be sure the links in the chain occurred and were caused by the program intervention.  Though different schools of evaluation research emphasize one approach or the other, our conclusion is that both controls and program theory are necessary components of the systematic evaluation of outreach programs.

More information on procedures used in comparison group evaluation designs is presented in Appendix A.

5.  Evaluations should connect outcomes and cost.
By studying the costs as well as evaluating the effectiveness of a number of interventions aimed at a common goal, policymakers gain an understanding of the relative worth of each intervention.  This coupling of standard evaluation procedures with cost analysis enables a decision maker to choose the program or set of programs that provides the best educational results for any given state appropriation.⁹  For example, data in California suggest that the state has purchased, on average, only an extra four minutes for the school day from the hundreds of millions of dollars it has spent on longer school day incentives.  While state funds led some districts to restore the sixth period of high school, more often there was little impact on the course-taking patterns of high school students.  On the other hand, California mentor programs led to substantial changes in school procedures at relatively low costs.  These general figures of cost and program impact may suggest that California legislators might better redirect funds from longer school day incentives to the mentor teacher or other more cost-effective programs.

Although this illustration demonstrates the power of combining cost analysis and evaluation, such a cursory approach is not precise enough to enable a policymaker to choose among a number of similar cost-effective alternatives.  What is needed is a more systematic approach to combining the evaluation of the programs impact with an assessment of its costs.  One such approach is cost-effectiveness analysis (CE).  CE evaluates a series of alternative programs, all of which seek a common outcome, by comparing the ratios between each programs costs to each programs effectiveness.¹⁰

One limitation of cost-effectiveness analysis is that it can be used only to compare programs with similar or identical goals.  When a comparison of programs with disparate goals is needed, a second type of analysis, cost-benefit (CB) should be employed.  CB analysis evaluates a series of alternative programs by comparing the ratios between the programs costs to the monetary value of the programs benefits.  This approach presupposes that a programs results can be quantified in monetary terms.  CB analysis, for example, could compare two short term vocational education programs by looking at the jobs (and salaries) of the participants after completion of each program.  A problem with this approach is the difficulty of determining monetary gains (such as how much money students will earn).  Since determining the monetary benefits of these outreach programs would be extraordinarily difficult, this section does not look at CB analysis in depth.

Any analysis of a programs costs begins with two assumptions: 1) the cost of any program is the value of all the resources that the program uses, and 2) costs refer to the least expensive set of alternatives that will satisfy a given need.

Given these two assumptions, the question arises: Why can't the costs of a program be determined simply by looking at the budget expenditures for each education program or state reform?  While such a straightforward approach would facilitate the analysis, there are four reasons why a budget fails to predict a programs true costs accurately.  First, budgets often do not include all resources used in a program.  For example, a budget does not account for contributed resources or other unpaid inputs such as a free building or volunteer classroom aides.  Second, budgets generally do not include costs of resources such as buildings or heavy equipment that are used in a state reform but have been paid for in past years.  Third, budgets may lump funding of a number of different state interventions together, making it difficult to distill the separate costs of each program.  Finally, budgets represent plans for how resources should be distributed and often do not reflect the actual allocation of funds once a program gets underway.

What is needed is an approach to studying costs that identifies the startup, medium, and long-term cost, as well as the hidden costs of the programs.  Unfortunately, little analytic work exists on the true costs of different outreach efforts.

To conduct a cost evaluation the best place to begin is by following the money trail.  Evaluators need to know where and how outreach interventions have allocated resources.  In High School Puente, for example, how much was spent on program coordination, staff development, or community coordination?  The next step would be to explore cost-effectiveness, assuming that most outreach interventions have the same objectives, e.g. admission and persistence in a four-year university.  Henry Levin states the case for cost-effectiveness this way:

[Cost-effectiveness] integrates the results of [program] costs in such a way that one can select the best educational results for any given level of educational results for least cost.  It is important to emphasize that both the cost and effectiveness aspects are important and must be integrated.  Just as evaluators often consider only the effects of a particular alternative or intervention, administrators sometimes consider only cost.  In both cases, the evaluation will be incomplete.

Although cost-effectiveness can provide important policy information, it is limited to comparisons among programs with similar objectives.  A possible example would be the use of loans, scholarships, or higher-base salaries as a magnet to attract better-quality beginning teachers.

BACK  |  HOME  |  NEXT