the extent of pre-existing differences between the treatment and comparison areas can never be fully determined.

The absence of adequate comparison groups resulting from the way the program has been implemented limits the confidence one can place in evaluations of PSRO program effectiveness. While the severity of the problem varies with many aspects of evaluation design, generally one can place less confidence in comparions between a single treatment area and a single comparison area than in evaluations that use information from many areas. The latter type of evaluation does not solve the problems of inadequate comparison groups, but the large numbers tend to reduce the magnitude of potential bias. This report therefore considers the evaluations conducted by HEW, in which a national sample of active PSROS was compared with a national sample of comparison areas, to be the best available test of the effectiveness of PSROs. 14/

Differences in Pre-Program Patterns of Utilization Change. An additional, often very serious, problem can arise even when the treatment and comparison areas seem to be comparable just before the program is implemented. The potential problem is that different patterns of change may have been underway in the two areas that would not be apparent from a single, pre-program measurement. For example, two areas could show identical average lengths of hospital stay the month before PSRO review was started in the treatment area, even if average length of stay had been rapidly declining (from a higher initial level) in one area and increasing (from a lower level) in the other. In such a case,

average length of stay in the two areas would be expected to become less and less similar as time went on, merely because of pre-existing trends. This increasing dissimilarity could easily be mistaken for an effect of the program.

Problems of this sort are particularly likely to occur in studies that involve only a few areas. They can, however, occur in large studies as well, if assignment to treatment and comparison groups is nonrandom. Even the national PSRO evaluations, which involved nearly 200 areas, may have been affected by distortion caused by pre-existing patterns. The only way to rule out these problems is to measure utilization (or other variables of interest) repeatedly for a considerable time before the start of the program.

14/ OPEL, PSRO, Vol. I & III; HCFA, PSRO: 1978 Evaluation.

The severity of this problem is illustrated by two evaluations of individual PSROS which did collect repeated, preprogram utilization measures that allow one to check for differences in pre-program trends. The first is OPEL's study of the Colorado Admissions Program (CAP). 15/ The CAP study is methodologically superior to most individual studies and contains extensive baseline data. The second is a study of the Massachusetts organization mentioned earlier, CHAMP. 16/

Colorado. CAP began before implementation of the PSRO program and became one of the first active PSROs as the program evolved. The evaluation of the CAP program used Kansas and Nebraska, which had no PSRO or PSRO prototypes, as a comparison group.

The most important analysis in the Colorado study looked at utilization by Medicare patients both before and after the implementation of CAP in 1973. Utilization rates (days of care per 1,000 Medicare enrollees) for the four years from 1969 through 1972 were averaged to establish a baseline, which was then compared with two post-implementation years, 1974 and 1975 (1973 was disregarded as a transitional year). The pattern of change in the Kansas/Nebraska comparison area was used to adjust the Colorado data for contemporaneous changes that had nothing to do with CAP. This analysis suggests declines associated with review of 7.6 percent from baseline to 1974 and 3.1 percent from baseline to 1975.

Examining year-by-year changes during the baseline period, however, rather than treating them as a single average, reveals a very different picture. The major divergence between the Colorado and Kansas/Nebraska utilization rates occurred in 1972--the year before implementation--and in 1973, the transitional year. The first two post-implementation years show a leveling off and then a lessening of this difference. As a result, the Colorado-Kansas/Nebraska difference was virtually the same size two years after the program started (1975) as it

15/ OPEL, PSRO, Vol. V: A Comprehensive Case Study: The Colorado Experience.

16/

Fulchiero and others, "Can the PSROS Be Cost Effective?".

was both one and four years before the program (1972 and 1969). This pattern casts doubt on the conclusion that the reduction found by the first analysis was truly an effect of CAP. A point made in the abstract earlier is thus illustrated concretely: multiple baseline measures, extending for a considerable time before the implementation of the program, can be invaluable in sorting out the effects of the program from other misleading trends. They are particularly important when the treatment and comparison groups have not been randomly assigned and are not equivalent. The lack of multiple baselines accordingly is one of the major weaknesses of the national PSRO evaluations discussed below.

Massachusetts. CHAMP was a PSRO prototype, encompassing the entire state. The CHAMP study compared trends in utilization among Medicaid patients (subject to CHAMP review) with trends among non-Medicaid patients (not subject to review). A greater decline in utilization among Medicaid patients than among nonMedicaid patients (5.3 percent) was found and attributed to the CHAMP program.

Like the CAP study, the CHAMP study included more than one baseline measure (unfortunately, however, only two), and again the apparent effect of the program is called into question when one looks at the year-by-year pattern of change. Utilization by Medicaid patients was declining even before the program's inception, while that of non-Medicaid patients was increasing slightly. Medicaid utilization after the program began was at a rate that would have been anticipated from an extrapolation of the pre-program trend, whereas non-Medicaid utilization was at a lower rate than would have been expected. 17/

Problems of Inadequate Data. The preceding discussion has dealt with problems of research design, focusing mainly on comparison groups. Regardless of the quality of design, however, the results of a study are only as reliable as its data are good. Unfortunately, many evaluations of PSROs have suffered from severe data problems.

17/ For detailed analyses of the CAP and CHAMP studies, including graphic presentation, see Appendix A.

A major problem of data quality arises in choosing a measure of program impact. 18/ Accurate evaluation of reduced hospital utilization requires examination of its three components--admissions, length of stay, and intensity of service (that is, volume of medical service per patient day). In order to avoid mistaking a change in the size of the population under study for a PSRO effect, utilization should be expressed as a rate. That is, measures of utilization should take into account the size of the consumer population--for example, days of care per 1,000 Medicare enrollees. Inability to measure all of the components of utilization or to express utilization as a rate can detract from the reliability of measurements of PSRO effects.

In practice, however, no evaluations of PSROS have met all of the criteria outlined above. Comprehensive data on service intensity are lacking in all available studies. Furthermore, the use of rate data unfortunately has been quite rare. This is partly due to the difficulty of obtaining the necessary information about the number of people in relevant categories. For example, rate data on utilization by Medicaid eligibles have often been precluded by a lack of adequate information on the total pool of eligibles.

are

Other problems frequently encountered in PSRO evaluations that data from either the pre-program or the post-implementation period are incomplete, or that the data from the two periods are in some way not comparable. For example, Medicare data for years up to 1974 are not comparable to data for later years because of major changes in the Social Security Administration's data collecting system. Lengthy claims-processing

18/

Ideally, a comprehensive evaluation of the effectiveness of PSROs would include measures of PSRO effects on both utilization and quality of care. Measurement of the quality of care is incomparably more problematic. The major reason is the absence of consensus on the measurement of quality, and the particularly intractable problem of aggregating better outcomes from a wide variety of medical procedures. This has not been a severe problem in practice, however, because PSRO evaluations generally have paralleled the program in placing primary emphasis on utilization rather than quality.

procedures prevent use of the most recent data. 19/ As a result, the most recent HEW evaluations were restricted to comparing 1974 with 1977.

Aside from these sometimes intractable data problems, inaccuracies have often occurred because sensible collection practices are not followed. For example, one evaluation used Medicare data that included hospital types not covered by PSROS and hospitals outside the PSRO geographical area. Two other evaluations compared total days of care during the pre-program period with just those days of care certified by the PSRO during the post-program period. 20/

Giving responsibility to individual PSROS to assess their own impact risks perpetuating errors such as those mentioned above, since PSROS tend not to have evaluation specialists on their staffs. Strong technical assistance from a central evaluation team is likely to be necessary to avoid such problems.

National Evaluations of the PSRO Program

The 1977 OPEL report and the 1978 HCFA report included a variety of studies, of which three are national evaluations of the program's effects on utilization.

The first of these studies was an evaluation of hospital discharge abstract data. 21/ (Abstract data are summaries of the medical records of a patient's stay.) Despite the ambitious scope of the study (over one million Medicare and Medicaid

19/ These lengthy claims-processing procedures can lead to serious distortions if data are used prematurely. For example, the General Accounting Office (GAO) examined the claimed cost savings of six PSROS and estimated the savings to be overstated by 672 percent. A major factor in the overstatement was the use of the latest year's data before late billings could be tabulated. See PSRO, Hearings, Pp. 2-8.

21/ OPEL, PSRO, vol. 4: Acute Care Utilization Impact: Inferences from a Sample of Hospital Abstract Data.

49-034 0794