PSRO's is a mistake. Similarly, if an evaluation uses an inappropriate comparison group-that is, a group that differs at the outset from one that receives the program-preexisting differences between the groups can be mistaken for effects of the program.

Despite the problem self-selection causes in evaluations, it has been and continues to be the primary method of implementing PSRO activities. The statute establishing the program mandated that the Secretary of HEW establish a PSRO in each area, at the earliest practical date. Accordingly, PSRO activities began in each area as soon as the qualified physicians' organization requested PSRO status and organized appropriate activities. The experimental groups thus selected themselves. Similarly, the comparison groups used in all studies of concurrent review were self-selected: They comprised those areas where no organizations were eager to perform review or where the chosen organizations were slow to implement it.

Self-selection likewise poses a problem for evaluating the current shift to focused review. It is crucial to evaluate not only whether focusing actually increases the cost effectiveness of PSRO's, but also what types and what degree of focusing are best. Present policies, however, virtually insure that these questions will never be answered reliably. The method and extent of focusing is entirely up to the PSRO's themselves. Presumably, their choices will be rational ones based on such factors as the nature of their patient populations, the severity of their budgetary constraints, and predominant attitudes among physicians. Differences in these respects, which are likely to be severe, and moreover, are largely unmeasured, will be confused with any real effects of focusing.

Some legislative action may be called for because, in some instances, HEW may be prevented by the existing statute from restricting selfselection. The statute mandates that nationwide implementation be carried out as quickly as is "practicable." This can be interpreted as prohibiting the temporary designation of areas without PSRO's to serve as comparison groups. Likewise, the statute gives individual PSRO's considerable autonomy in establishing their own criteria for review, and this may hinder evaluations of some new methods of review.

In addition to improving evaluations, a strategy of restricting selfselection-that is, implementing components of the program initially only in carefully selected sites-has an important additional value in times of budgetary restraint. Initial expenses would be decreased, since not all areas would get all new components of the program. Finally, comprehensive implementation could then await a determination of which components of-or changes in the program are most costeffective.

Finally, I should note that the steps required for good evaluation often involve a tradeoff. In many instances, it may be decided that the need for a program is so acute, or the probability of its success so great, that the delay in full implementation needed for reliable evaluation is not justified. When reliable evaluation is a primary goal, however, program implementation must be tailored accordingly.

That completes my statement. Thank you for giving us this opportunity. My colleagues and I will be happy to respond to any questions that you have.

[The prepared statement and attachment follow:]

STATEMENT OF ROBERT D. REISCHAUER, DEPUTY DIRECTOR, CONGRESSIONAL BUDGET OFFICE

The Professional Standards Review Organizations (PSRO) program was established by the Social Security Amendments of 1972 to "promote the effective, efficient, and economical delivery of health-care services of proper quality for which payment may be made under the Act." The amendments call for the creation of local or statewide organizations of physicians to monitor services to ensure that such services:

Conform to appropriate professional standards,

Are provided only when medically necessary, and

Are provided in the most economical but medically appropriate settings. Since its inception the PSRO program has had two goals. The first is to curb the growth in expenditures for health care by curtailing unnecessary utilization by federal beneficiaries designated by the Act. The second is to assure the quality of care delivered under the Act. To date, however, the program's primary emphasis has been on curtailing expenditures.

The principal means by which PSROS attempt to meet this goal is "concurrent review," which involves two activities: "admissions review," in which cases are reviewed shortly after a patient's admission to determine if admission was needed, and periodic “continued-stay reviews," in which cases are re-evaluated to determine if continued inpatient care is warranted. A negative determination can result in denial of reimbursement by Medicare or Medicaid for further inpatient care. (Sanctions against providers and practitioners are also established in the statute, but regulations governing their imposition have not yet been issued by HEW.)

Now, seven years after the program was initiated, is an appropriate time to ask whether the program is meeting its goals. Should it be continued or changed? Should funding be maintained? Cut back? Increased? And are additional evaluations of the program needed?

The importance of these questions extends beyond the PSRO program itself, since some recent health insurance and cost containment proposals include the use of PSRO-type organizations to solve health-care utilization problems. For example, the Long-Ribicoff Catastrophic Health Insurance and Medical Assistance Reform Act (I am referring to Section 2104 of S. 350) would rely on professional review to limit outpatient mental health services.

At the request of this Subcommittee, the Congressional Budget Office (CBO) has prepared an analysis of the PSRO program, and of the evaluations that have been done so far. The study is being released this morning. My testimony summarizes the CBO paper and addresses the following three issues:

Do PSROS reduce inpatient use of hosptial facilities by federal beneficiaries?

Do the reductions in health-care expenditures attributable to PSROs exceed the costs of the program?

What would be the best way to proceed in order to learn how to improve the performance of the program?

DO PSRO'S REDUCE SHORT-STAY HOSPITAL USE?

The 1978 Health Care Financing Administration (HCFA) study, which up to this date has been the best available evaluation of the program, concluded that PSRO concurrent review has reduced Medicare days of care by 1.5 percent. CBO's reanalysis of HCFA's data suggests that the reduction in Medicare days of care is closer to 2 percent. While PSROS were active in only about half of the nation at the time the data were collected, the program has since expanded to include most of the nation. Accordingly, the CBO estimate of a 2 percent reduction projects the effects of PSROS to the nation as a whole. CBO's estimate differs from HCFA's primarily because CBO took regional differences into account.

The conclusion that PSROs reduce Medicare days of care, however, is subject to very important qualifications:

It is based on only one good study and does not reflect any sort of research consensus. Most of the other available evaluations of the program suffer from the use of inadequate data and unsound methods. Moreover, their findings are inconsistent.

The HCFA evaluation does have some weaknesses. The most serious stem from the way in which the program itself was implemented and from the lack of sufficient pre-PSRO data. The HCFA analysis generally handled these problems well, but some of the problems are such that they cannot be fully corrected in the analysis. Moreover, as the General Accounting Office will explain, some avoidable problems of data quality remain. Therefore, the estimate of a 2-percent reduction is still uncertain.

The calculations are based only on Medicare patients. The results probably cannot be generalized to Medicaid and other patient groups. Furthermore, the reduction in days of care will probably be offset to some degree by increases in other types of care, such as care in nursing homes and ambulatory care.

ARE SOME PSROS MORE EFFECTIVE THAN OTHERS?

Some PSROs may be more effective than most in reducing utilization, while others are probably ineffective. Identifying the better PSROs would be helpful in managing the program for two reasons. First, HCFA might choose to increase funding to the more effective ones or decrease funding for others. Second. identifying the characteristics of the better ones might provide a basis for overall program improvement. Unfortunately, there are no reliable studies that identify the most effective PSROs. HCFA's 1978 ranking of PSROS used methods that tend to overestimate the effectiveness of older PSROS. Such a ranking could result in rewarding or sanctioning the wrong PSROS and giving the wrong characteristics credit for PSROS' effectiveness.

DO PSROS SAVE MONEY?

Does the estimated 2-percent reduction in utilization translate into a net savings in health expenditures? That is, are the savings greater than the costs of the program itself?

This question can be answered by calculating a ratio of savings to costs. CBO estimates that this ratio is about 0.7-to-1-in other words, the program saves about 30 percent less thaon it costs. HCFA, on the other hand, has estimated the ratio to be 1.1-to-1-that is, the program saves about 10 percent more than it costs. To calculate such a ratio, however, one must define what is meant by savings and make many assumptions.

Savings can be defined in two ways:

or

The total savings that result from PSRO-induced reductions in utilization;

The resulting savings to the government-in this case, through lowered Medicare reimbursement.

The importance of the difference between these definitions can be illustrated by an example. Suppose that Medicare days of care decreased as a result of PSRO activities, but hospitals made up every cent of lost revenue by charging non-Medicare patients higher prices. Medicare savings might then be substantial, but total savings would be zero.

HCFA chose to use the second definition-savings to the government. In fact, the HCFA analysis assumed that total savings were zero and that all of the reported "savings"-that is, Medicare savings-are nothing more than costs transferred to other patients or third-party payers (mainly insurance companies). In contrast, CBO felt that the first definition-reduction in total hospital expenditures should be used in measuring PSRO accomplishments. If PSROs are working, they should be freeing resources now consumed in the delivery of hospital care so that they can be put to other uses. It is unlikely that the Congress conceived the PSRO program as a $150 million per year mechanism to transfer financial burden from the Medicare trust fund to the private sector.

The assumptions required for the savings estimates are numerous. Among them, for example, are assumptions about increases in outpatient care that might accompany decreases in hospital utilization and about the degree to which hospital costs vary in proportion to utilization changes. The necessity of making these assumptions means that any savings-to-cost estimate including both CBO's and HCFA's-has a wide margin of error.

If there were any savings, they would have to be compared to the magnitude of the problem which the program is designed to remedy. It is worth noting, there

fore, that if HCFA's estimate were correct, the net savings would amount to less than one-tenth of 1 percent of relevant Medicare expenditures.

HOW CAN THE PROGRAM BE IMPROVED?

If the PSRO program is continued, both the Congress and HEW will want to improve its effectiveness. Some have argued that existing PSRO activities might be improved by moving toward "focused review." That is, rather than reviewing all cases, PSROs are beginning to review only those cases that are most in need of review. If focusing can eliminate superfluous review without simultaneously ending much useful review, the effectiveness of the program would be increased. The criteria used in review might also be improved. For example, a new review system based on criteria of severity of illness and intensity of services-rather than on specific norms for length of stay, which is the usual criterion-is being tested in some PSROs.

Others have argued that the overall effectiveness of the program may change as the program expands into other types of medical care. For example, review of ambulatory care, long-term care, and hospital ancillary services has been planned, though implementation has not yet progressed very far.

Unfortunately, the information needed to design a more effective program does not now exist. Since there is no good evidence identifying the more effective PSROS, one can't ascertain which characteristics should be emulated. Moreover, good evidence on the effects of program changes such as those mentioned is lacking, and with a continuation of present policies, it is unlikely to become available in the future.

This lack of information is not primarily the result of inadequate evaluations. It is largely a result of the way the program itself has been implemented, which has seriously handicapped even the better evaluations. The crux of the problem is that PSRO areas have chosen for themselves whether or not to undertake PSRO activities. In other words, the active PSRO areas are self-selected, which distorts any assessment of their effectiveness. This problem continues to be severe, and unless changes are made, we are likely to continue to lack the information needed for careful program assessment.

In most instances, the only reliable method of evaluating a program of this sort is to implement the program only partially at first. It is necessary to produce two groups-groups of people, of hospitals, of PSROS-that are essentially alike before the program is begun. One group-the experimental group-then gets the program, while the other does not. The latter group serves as a comparison group. If the two previously alike groups differ in a relevant way after the program is begun, the differences can be attributed to the program.

If a program is evaluated without a comparison group, all sorts of trends can be mistaken for program effects. For example, many evaluations of PSROS reported a decline in average length of hospital stay after the program began. Other data, however, suggest that there has been a continuing trend in recent years toward shorter lengths of stay, independent of any effects of PSROS. In such an instance, attributing the shortened lengths of stay to USROS is a mistake.

Similarly, if an evaluation uses an inappropriate comparison group-that is, a group that differs at the outset from the one that receives the program-preexisting differences can be mistaken for effects of the program. While there are standard methods to "correct for" such pre-existing differences, such techniques are often inadequate.

Evaluations are most reliable when individuals—that is, people, or hospitals, or PSRO areas are deliberately chosen to receive or not receive the program in such a way as to produce a comparison group that is essentially the same as the experimental group. Ideally, the selection should be totally or partially random. This may sometimes be difficult in practice, but it is often feasible. A second-best alternative is for the program managers and evaluators to choose individuals to get or not get the program, on the basis of known, important characteristics, in order to minimize pre-existing differences.

At the other extreme, evaluations are generally unreliable when managers allow individuals to choose for themselves whether to receive the program. This approach is undesirable because it virtually guarantees that the comparison group will differ in fundamental ways from the group receiving the program. Furthermore, the difference in these cases will often be in terms of characteristics-such as attitudes and motivation-which are extremely difficult or even impossible to

measure.

Despite the problems it causes in evaluation, self-selection has been, and continues to be, the primary method of implementing PSRO activities. The statute establishing the program mandated that the Secretary of HEW establish a PSRO in each area "at the earliest practicable date." Accordingly, PSRO activities began in each area as soon as a qualified physicians' organization requested PSRO status and organized appropriate activities. Therefore, the comparison group used in all studies of concurrent review was self-selected: it comprised those areas where no organizations were eager to perform review or where the chosen organizations were slow to implement it.

Self-selection likewise poses a problem for evaluating the current shift to focused review. It is crucial to evaluate not only whether focusing actually increases the cost-effectiveness of PSROS, but also what types and what degree of focusing are best. President policies, however, virtually ensure that these questions will never be answered reliably. The method and extent of focusing is entirely up to the PSROS themselves. Presumbably, their choices will be rational ones based on such factors as the nature of their patient populations, the severity of their budgetary constraints, and predominant attitudes among physicians. Differences in these respects, which are likely to be severe, and moreover, are largely unmeasured, will be confused with any real effects of focusing.

Some legislative action may be called for because, in some instances, HEW may be prevented by the existing statute from restricting self-selection. The statute mandates that nationwide implementation be carried out as quickly as is "practicable" This can be interpreted as prohibiting the temporary creation of comparison groups of areas without PSROS. Likewise, the statute gives individual PSROS considerable autonomy in establishing their own criteria for review, and this may hinder evaluations of some new methods of review.

In addition to improving evaluations, a strategy of restricting self-selection— that is, implementing components of the program initially only in carefully selected sites-has an important additional value in times of budgetary restraint. Initial expenses would be decreased, since not all areas would get all new components of the program. Final, comprehensive implementation could then await a determination of which components of-or changes in-the program are most cost-effective.

Finally, I should note that the steps required for good evaluation often involve a trade-off. In many instances, it may be decided that the need for a program is so acute, or the probability of its success so great, that the delay in full implementation needed for reliable evaluation is not justified. When reliable evaluation is a primary goal, however, program implementation must be tailored accordingly.