A Research Agenda for Evaluating Program Effectiveness
There is no one study, or type of research approach, that should be emphasized. A multi-level, multi-method approach is most suitable for asking the diversity of questions stakeholders currently have about outreach programs.  Four types of approaches deserve careful consideration.

1) Individual studies of students  Individual studies follow one or a small group of students through a program or set of programs over a period of time.  Methods used in these studies are qualitative, involving "thick description" of students lives and their interaction with programs and program staff.  These studies can answer questions such as:

-how do individuals respond to outreach programs?
-what long-term effects do programs have on students?

Similar studies with a larger group can compare how different students respond differently to the same program or series of interventions.  Most evaluation research, by varying programs or components, assumes that students lives do not change and that all students are roughly the same.  Careful micro-level studies avoid this trap, holding programs constant while varying the personalities and life situations of students to see how different students respond to outreach efforts.

2) Longitudinal, time-series studies of students  These studies follow a large cohort of students over an extended period time. They examine how a constellation of attitudes and behaviors of students change over time and are influenced by program efforts.  Michael Knapp (1994) notes that following student behavior over time may be the best control strategy for seeing the real effects of program efforts.  The general form these studies take is to start with baseline set of measures prior to program participation, describe the program intervention, and then conduct systematic follow-up to show the pattern of student change after program participation (Knapp, 25).  The main advantage of such a design for outreach programs is in following students through the chain of events that lead to college success.  Tracking students through college attendance completion and career choices can develop the long-term indicators that most local programs cannot track.

3) Comparative studies of programs and program components  As noted earlier, outreach programs vary on a number of programmatic dimensions.  Local programs have expressed the desire to compare these features but such comparisons are best conducted at the systemic level.  Comparative studies of programs can ask such questions as:

-what is the relative impact of student and school-centered outreach activities?
-should programs focus on early grades, middle schools, or upper-grade high schools?

Comparative studies rarely compare whole programs; rather they vary key programmatic dimensions and mixes of services.  In addition to program features, cost variables can also form the basis of a comparative study.  This research can be done confidentially if local programs fear they will be pitted against one another in the evaluation.

Comparative research is usually done under non-experimental conditions. However, when designed carefully, such studies can uncover the range of conditions that lead to different program outcomes.¹¹

4) Backward mapping  These types of studies start with measures of successful programs and ask what about these programs are successful and what can other efforts learn from them.  Successful programs are usually identified through reputational measures though more systematic indicators can be used to identify such efforts.  The goal of backward mapping studies is to identify "exemplary practice" and explain the conditions and factors that produce it (Knapp, 1994).  Backward-mapping studies can also be done on program failures, to examining why a program did not work, take hold, or get implemented.

5) Other possible studies  Many local evaluations have professional development for teachers as an important outcome.  These programs theorize that well trained and enthusiastic teachers can make significant differences in student lives and outreach program success.  Longitudinal studies (see 2 above) can track change in teachers through professional development and possibly link these changes to changes in student attitude or performance.  Similarly, some program managers argue that school reform is an important outcome of outreach efforts.  Time-series studies that track school change and its effect on students may be appropriate for these types of programs.

Many outreach programs involve collaboration among K-12 schools and institutions of higher education.  An often unstated assumption of these programs is that better collaboration will lead to better student outcomes.  Studies of the collaborative process may provide insight to how outreach efforts can work better.  A model for these types of studies are the evaluations of human and social service collaboration that have been conducted in recent years (Kagan et al, 1990, Gomby & Larson, 1992).¹²

BACK  |  HOME  |  NEXT