FIGURE 3. HOSPITAL UTILIZATION DATA FOR KANSAS/ MASSACHUSETTS OBSERVED AVERAGE LENGTH OF ... 62 MASSACHUSETTS OBSERVED AND PROJECTED AVERAGE ... 67 SUMMARY The Social Security Amendments of 1972 established the Professional Standards Review Organization (PSRO) program in order to "promote the effective, efficient, and economical delivery of health care services of proper quality for which payment may be made under the Act." The PSRO program attempts to meet this goal by means of a peer review system that is funded by the U.S. Department of Health, Education, and Welfare (HEW). While the goals of the program are broad enough to include both reduction of expenditures and assurance of quality, the primary emphasis of the program has been to reduce utilization of--and thereby expenditures for--short-stay hospital care by means of "concurrent review." Typically, PSRO concurrent review consists of examining hospital admissions to certify that, from a medical standpoint, they are appropriate and reassessing each case periodically to determine whether continued inpatient care is warranted. Review and reanalysis of the research on the effectiveness of PSROS indicate that concurrent review is reducing the number of days of hospital care of Medicare enrollees by about 2 percent. This estimate has to be viewed with caution, however. Most extant evaluation studies are too flawed to be reliable, and furthermore, they yield inconsistent evidence. Even the best research available--a generally sound study conducted by HEW's Health Care Financing Administration (HCFA), on which the 2 percent estimate is based--also suffers from some important weaknesses. Because of the lack of relevant data, it cannot be assumed that PSROS are equally effective in reducing utilization by other federal beneficiaries (primarily Medicaid patients) whose care is subject to PSRO review. Similarly, it is not clear what effects PSRO review would have on other groups (for example, veterans and private patients) if the program's authority were extended to them. Although PSROS seem to be effective in reducing Medicare utilization, it is doubtful that they produce a net savings. The recent HCFA analysis concluded that the monetary benefits of the Medicare portion of the PSRO program have been about 10 percent greater than its costs. That analysis implies an extremely small net savings relative to expenditures for services that are currently being reviewed by PSROs (less than 0.1 percent of relevant Medicare reimbursements). A CBO reanalysis of the data revealed no net savings at all; CBO has concluded that the best estimate is that the savings generated by the program are about 30 percent less than program costs. Both the CBO and HCFA estimates, however, rest on controversial assumptions and are open to considerable error. A number of factors, including budgetary constraints, current concern with the containment of health-care costs, and continuing changes in the PSRO program, suggest that further evaluation of the effectiveness and cost-effectiveness of PSROS is needed. Moreover, the inconclusiveness of much of the existing research on PSROs indicates the importance of improving the quality of evaluations of the program. To some degree, quality can be increased by improving the research methods employed. However, the reliability of even methodologically sound evaluations--for example, the recent HCFA evaluation, which is for the most part a careful and well-designed study--have been limited by the way the program itself has been implemented. Unless changes are made soon in both implementation and evaluation, future evaluations of the program will continue to be unreliable--often to such a degree as to be useless in formulating policy. This problem extends both to new PSRO activities (for example, review of long-term care) and to refinements of existing activities (such as focusing review on certain diagnoses, providers, practitioners, or patient groups that offer the greatest potential for a PSRO effect). The most important improvement in the evaluation of PSROs would be a more careful use of comparison groups. When the effects of a certain component of the PSRO program are to be evaluated, that component must be implemented only in some areas (the "treatment" group), while other selected areas (the "comparison" group) are left without it. If the treatment and comparison areas are initially similar in all other respects, comparing them after the program is underway reveals whether seeming "effects" of the program are actually caused by other factors. For example, recent years have shown a general trend toward a shorter average length of stay for hospitalized patients; use of comparison groups would avoid mistaking this trend, which began before the existence of PSROs, for an "effect" of the PSRO program. On the other hand, comparisons between areas with and without PSROs can be seriously misleading if the treatment and comparison areas were not equivalent (or nearly so) before the program. For example, if the program were implemented in areas already experiencing a decline in average length of stay, and the comparison areas were those in which average length of stay was stable, the comparison would show a spurious "effect" of PSROS on length of stay. The way in which the PSRO program has been implemented has hindered reliable evaluation by preventing the creation of an appropriate comparison group. Ideally, the treatment and comparison areas should be chosen randomly; as a second-best alternative, they could be selected to be alike in as many respects as possible. To date, however, the implementation of the PSRO program has relied on "self-selection": that is, areas have chosen on their own initiative whether or not to participate. Those that chose to participate became the treatment group, while those that chose not to participate became the comparison group. Self-selection virtually guarantees that the treatment and comparison groups will be dissimilar in many respects--often in ways that will cloud evaluation of the pro gram. Depending on what specific component of the program is involved, changing the manner of implementation to permit the use of good comparison areas might require legislative as well as HEW initiative. For example, several PSROs are currently pilot testing a new method of concurrent review that makes use of information on severity of illness and intensity of medical services as well as broad diagnostic categories. In contrast, the more traditional form of concurrent review is built around regional, diagnosis-specific norms for length of stay. The new method has received considerable attention as potentially cheaper and more effective than the traditional method. To test the new method reliably, one would randomly assign some PSROS to use it, while other areas would be left to use the old methods. Since the current statute gives individual PSROs the authority to choose their own criteria for review, however, HCFA would be unable to assign PSROs to the new system without legislative initiative. Other improvements in the evaluation of the program could be made entirely on agency initiative. Multi-site evaluations should be stressed, and less emphasis should be placed on evaluations of individual PSROS. The measures of utilization employed should be comprehensive and should relate clearly to health-care costs. When feasible, utilization of health-care resources should be measured repeatedly over a considerable time span before the program is implemented; this allows one to assess pre-existing trends and clarify initial differences between the treatment and comparison areas, in order to avoid mistaking irrelevant patterns for PSRO effects. A few of the best evaluations of PSROs have incorporated some of these improvements, but further improvement is still greatly needed. Reliable assessments of the effects of a given PSRO program component are often feasible only at early stages of that component's implementation. As implementation continues and the number of areas with that component increases, it becomes increasingly difficult--and eventually impossible--to create a reasonable comparison group. For that reason, if current or pending changes in the PSRO program are to provide reliable evaluations that are useful in formulating future policy, improvements of the sort discussed here must be made in the near future. |