EVALUATION MUST EXTEND

ΤΟ RESEARCH AND

PROJECTS

DEMONSTRATION

Particular attention should be given to evaluation of research and demonstration projects as an important way of determining which programs work best. The act makes provision for research and demonstration which carries with it the authority and responsibility to evaluate the results of such projects. The very nature of these projects makes evaluation difficult. There are many initial operational problems to work out, and the program objectives are often either unclear or unspecified, the result being that there is not a generally acceptable standard against which to compare initial program results.

If valid statistical evaluations of research and demonstration projects become possible, much better information about program effects can be provided for use in policymaking decisions. These considerations suggest that research and demonstration projects. should, in the future, be set up with statistical designs, including control groups, so that all possible factors can be held constant except the program variables under consideration. However, we are not suggesting that all OEO programs be set up with statistical designs.

OEO has a fairly significant program of evaluation in support of its CAP demonstration projects. OEO's plans call for an increased role of evaluation in this regard. The CAP Research and Demonstration Division has the responsibility for evaluating demonstration programs. For example, the adoption of better criteria for eligibility should be related to evaluations of the results of particular programs. This would require more evaluative effort in determining what is happening to individuals and families who have been served by a program, and more specific attention to the differences in the services provided to them during participation in the programs. However, in order to guide policy, this evaluation must be representative of national problems and results. The role of evaluation, specifically of research and demonstration, should be given careful consideration in all antipoverty efforts.

EVALUATION SHOULD EXTEND TO ALTERNATIVE PROGRAMS

Extensive as the requirements for evaluation in the act are, they do not provide that evaluation shall specifically include the study of alternatives to OEO programs. Section 631 of the act does require the Economic Opportunity Council to assist the President in developing basic policies and setting priorities with respect to programs and activities related to the act. However, the Council is not in existence.

Alternate programs should not only be tested diagnostically, they should also be compared in cost, for equal or similar results. Population and economic trends should be used to project costs and results over future years. Sensitivity of these results to shifts in population and economic trends should be studied so that the program planners and/or policymakers can be in a position to adjust the mix of programs to respond to such shifts. For example, where full employment conditions exist, job creation programs may look relatively less desirable than they would under times of adjustment in the job market following an economic recession or a military deemphasis.

In recent years several comprehensive evaluations have been conducted concerning the effects of present and proposed programs for

providing cash payments to the poor (often called income maintenance programs). The present programs had their origin in 1930's; however, their impact on poverty and consideration of alternative programs was not evaluated until the mid-1960's. This recent interest in evaluation is largely due to the widespread concern with poverty in the 1960's.

In retrospect, it is unfortunate that such efforts were not conducted at an earlier date. Recent evaluations have better defined the number of persons in poverty who do not receive cash payments under current programs and, for those that receive payments, the limited extent to which the payments lift them out of poverty. Evaluations of alternative programs, such as expansion of the social security program, and of the basic changes that would have to be made for public assistance programs to be more effective have provided some very informative results for policy decisions.

Past evaluations at OEO have not provided information that is pertinent for planning of broad changes in program orientation. The evaluations that are now underway will likely be more relevant for such planning.

THERE MUST BE AN ADEQUATE EVALUATION STAFF

The evaluation function can be carried out most effectively by a staff which is independent of program operations and which is also associated with the decision process. In chapter 10, we have recommended that the evaluation responsibilities be removed from OEO and relocated with the coordination and planning functions in a staff office in the Executive Office of the President.

Working out the details from the broad specifications given above will require further consideration. The past failure to develop highquality evaluation groups of sufficient size in most of the operating agencies indicates that the new organization would have to recruit much of its own talent. In addition to doing the job at the Federal level, such a staff would need to coordinate with and assist, when necessary and appropriate, State, city, and community organizations. with their planning and evaluation.

The operation of an evaluation staff separate from operating responsibilities carries the danger of insulation from knowledge of the programs. This can be avoided in various ways, for example, by forming special study groups, with participation from the evaluation staffs and/or program staffs of the involved agencies.

CONTINUING RESEARCH MUST BE CARRIED ON

The making of evaluation studies will require that continuing research be carried on. A greater understanding must be developed of the causes of poverty through basic research that identifies reasons for the existence of poverty and through data that depict more clearly and frequently the demographic dimensions of poverty.

OEO's responsibilities in the conduct of basic research are not defined by the act. Some of this can be done by the colleges and universities but some should be done within the Government. A potentially viable arrangement is being tried by OEO with the development of the Institute for Research on Poverty which is being

supported at the University of Wisconsin. Many of the publications and analyses of the Institute are quite pertinent to poverty program evaluation. This experience should be reviewed and similar types of arrangements with other universities might be indicated.

RELIABLE AND PERTINENT DATA MUST BE AVAILABLE

Better research and evaluation studies depend to a great extent on improved data concerning the characteristics of the poverty population. The Bureau of the Census is the principal collector of general poverty statistics and OEO has been working closely with Census for improving these statistics.

În addition to overall demographic statistics, evaluation requires a great variety of data that is reliable and pertinent to the achievements of the programs. It is imperative that attention be given to this need. Without adequate data the evaluation effort can never be fully satisfactory. Furthermore, some of the data needed for evaluation is also needed at local and other management levels for planning, operation, and monitoring of the programs involved. To the extent that the quality of data reporting from local levels can be improved, the supplementary data gathering required for evaluation purposes will be simplified.

Our contractor made a review of data which had been collected in the agencies' central files partly for purposes of the agencies' own evaluation. Numerous instances of incomplete information were found, particularly followup information on program participants after they had left the programs. This kind of data is among the more important data needed to evaluate achievement of program objectives. The headquarters files contain data required to be reported by local agencies.

A major reason for widespread noncompliance with the requirement to report followup data was a shortage of qualified personnel at the local or project level. Although we could make no accurate estimate of the additional personnel which would be required to fully comply with the requirement on a complete and periodic basis, such data is very costly to collect, particularly on participants in poverty programs. For many reasons, including fluid housing arrangements, fear on the part of former participants, and other reasons, it can be a difficult and costly task to track down all program participants.

Another reason for the failure to report data requested is that the agency evaluation staffs, unable to directly control and rectify the reporting deficiencies at the local level, have tended to ignore, in favor of one-time collections of missing data, the formal information systems that require the local agencies to report data. Realizing that the evaluation staffs rely on such special data collection efforts, personnel at the local level have little incentive to take the time and effort to provide data which is of little apparent interest.

Even though certain participant information was not always available in the agency's central file, we did expect to find much of it in the participant files kept at each project location, but such information was frequently not available there. The lack of such information in local files has important implications with regard to program operations as well as evaluation, as pointed out particularly in chapters 4

and 5. At 44 locations, we attempted to obtain information on 3,522 participants in 96 separate manpower projects. These local project files contained sufficient information on only 121 participants.

The need for improvement in the mechanism for collecting pertinent and reliable data for evaluation purposes warrants particular attention by the staff agency in the Executive Office of the President (which we recommend in chapter 10).