The ex when melded the regional dummy variable was CBO suggests, but does not name, other omitted variables which could be bla-ing the res ilts by their omission. HCFA will be happy to consider including d'aflara nada lutasites which (BO might Siggest. However, HCFA has reviewed all relevant ha'. Lil healt.. d'a tases and utilizing these adjustment variables for wh... data HAIS The naming of additional factors' is a rather simple task; the uno sion of such factors in an evaluation model is quite another issue, however, due to data constraints. Question What factors of any existed which you could not control for in Inil your 197 PSRO evaluation? Would these factors distort the cost effectiveness of the PSRO program? Answer. There are potentially hundreds, if not thousands, of idiosyncratic Variables which affee utilization benut or. No study will ever be able to control for all the potential factors which can bias results, even to a small degree. However, among the more important factors which could not be considered due to lack of data are the following: (a) Utilization Review (UR) Prior to PSRO Implementation-essentially the analysis assumes that prior UK had no effect on utilization. The costs of prior UK were also not considered. (b) PSRO Organizational Effect on Review Outcomes-The only two PSRO characteristics available were length of time of review and percent of all PSRO area admissions under review. Many organizational variables could also have an impact such as staff size and expertise, budget allocations, relationship with other health agencies, physician participation and the like. Efforts are being made to include this type of data in future evaluations in order to differentiate more clearly among PSROS. (e) Health Status It is possible that there are differing rates of illness among the aged which account for different utilization rates across PSRO areas. However, health status is exceedingly difficult to measure in a few individuals much less a nationwide population of 20 million. (d) Changing Medical Practices - Medicine is not a static science. It is constantly changing. Among the changes taking place are ideas about admissions criteria and length of stay. Local standards may changes faster or slower than national standards. Thus, it is possible that some PSRO effectiveness. Cor lack of effectiveness) is due to changing medical practices which cannot be measured. (e) The Attitudes Towards UR (PSRO) in Active and Inactive Areas.It may be that PSRO active areas represent milieux which are highly supportive of UR-type activities. If this is the case, PSRO active areas may have fostered lower use rates independent of PSRO activities. Bias is associated with type of contextual arrangement are particularly difficult to manage through quasi-experimental design. Question 5. In preparing the 1978 HCFA evaluation of PSROS was the cost of appeals considered in determining the cost effectiveness? Answer. To the extent that the cost of appeals was included in the Management and Support Cost categories, the cost of appeals would have been included in the cost component allocated to concurrent review. There was no measurable way to take account of the effectiveness or benefits from appeals. Hence, ommission of appeals from the calculations may have possibly biased the benefit/cost ratio downward. Appeals were not explicitly considered in the evaluation because, on average, they were expected to represent a small percentage of total review costs and because the process differs from place to place leading to differences in the way the benefits and costs would have had to be estimated. Further, the difficulties in measuring the benefits and costs including such variable costs as travel lodging, and lawyers potentially were much greater than the gain in sophistication in the benefit/cost calculations. FUTURE EVALUATIONS Question 1. Will Congress ever be able to assess the cost effectiveness of the PSRO program as currently constituted? Why or why not? Answer. This question is broader than the PSRO program cost-effectiveness issue. It relates to the determination of cost-effectiveness for all legislated pro grams. All evaluations of social programs which are not randomly implemented are hampered by the lack of true "control" groups. This means, in the strictist sense, that evaluations not based on randomly assigned control groups are subject to unknown bias. The true answer may never be known. Thus in a strict technical sense it is impossible to obtain an unbiased answer concerning PSRO costeffectiveness given the situation at hand. This does not mean, however, that evaluation results are meaningless or trivial. Biases can be reduced by the judicious selection of "comparison" areas and by statistical adjustments. The OPPR evaluation used approximately half of the country's PSROS as comparison areas and adjusts statistically for possible confounding variables. We believe that most bias has been removed due to these efforts. It is our position that the data and methodology used have produced answers within a useful range of accuracy. For instance, the degree to which smoking is related to cancer has never been "proven" in the sense that randomized trials were conducted, but such eausability is now widely accepted and policy statements are constantly made based on such evidence as exists. Similarly, we feel that the clustering of our benefit cost ratios in the neighborhood of 1.1 under a variety of assumptions is suggestive of conclusions supportive of policy decisions. We would like to point out that future PSRO evaluations will be even more difficult to interpret. As the number of inactive areas is reduced, the use of inactive areas as comparison areas becomes more and more suspect. When no inactive areas remain the effectiveness of the program can only be measured in a relative sense. A meaningful evaluation can still be carried out in the absence of comparison areas, but its main results will be the estimation of relative PSRO impact. That is, the difference in performance between more and less efficient PSRO areas will be estimated. Absolute impact at this point in time can still be estimated, but only through statistical modeling which will be extremely difficult to interpret. Cost indicators will however become more precise as data bases and computational techniques are refined. Thus, the cost of the program will be known with increasing accuracy. In conclusion, while the PSRO evaluation is not based on a randomized design we feel that non-randomized designs, with appropriate controls, can produce answers to significant policy questions. Question 2. What is your reaction to CBO's recommendation that control groups be used in future evaluations of long term care and other areas to determine their cost effectiveness? In this regard, what are your plans for future evaluations of the PSRO program and components of it? If you reject their recommendation, how do you respond to their conclusion that not to use control groups will tend to lead to inconclusive results? Answer. We agree that the CBO recommendation of using randomly assigned control and experimental groups is a method of implementation of a program if primary concern is scientifically unimpeachable evaluation of the program's results. The experimental design methodology reduces the number of factors that can compete in the interpretations of results. Because current authority does not permit the use of experimental design, the evaluation plans have been based on quasi-experimental analytic techniques. During the next year, we plan to have a contract to specify an appropriate evaluation design for long term care review. Although evaluation based on experimental design of long term care review implementation may be desirable, anticipated difficulties in getting agreement with Medicaid State agencies for randomly assigned review implementation and data collection in nursing homes, may inhibit its feasibility. We do not reject the CBO conclusion that a randomly assigned experimental design is the desired methodology. However, quasi-experimental techniques need not necessarily produce inconclusive results if the measured "impact" is great. However, if the impacts are not great, then the interpretation of whether or not observed changes are due to the program under study is more difficult. COSTS OF PSRO REVIEW Question 1. Your study found that a wide range in cost per discharge of PSRO review exists and that the average cost is above the $8.70 projected last year. Why are the costs so high and the range so great? What are your expectations for the future? Answer. In most cases, the highest unit costs were caused by absorbing the fixed costs among a small number of discharges. However, an extensive list of potential explanations could be suggested for the wide variation in review costs. An attempt was made in the 1978 Evaluation to determine the factors responsible for the variation in review costs for delegated hospitals (pages 155-159). There was some evidence that hospital size, delegation status, ownership, scope of review, and urban-rural location affect review costs, but the data base was inadequate to specify these relationships. A major component in the 1979 Evaluation will be devoted to the continued study of review cost variation. Since the data base for the current evaluation has been expanded and enriched, we expect the 1979 Evaluation will yield a more complete picture of the factor responsible for the high cost of review. It should be noted in conclusion, however, that cost data are notoriously unreliable. It is very common for cost data to have a relative standard error of 1, i.e., the standard deviation is equal to the mean. In this regard it is not surprising that PSRO review costs are highly variable. Similarly, there are several reasons to expect unit costs to decline in the future. As the number of patients under review grows, unit costs fall as fixed costs are amortized across more patients. As start-up costs are written off, costs will decline. Finally, focused review may result in reducing the cost of review. In sum, there are strong arguments leading to the expectation of decreasing cost of review and increasing benefits in the future. Question 2. If the HCFA 1978 Evaluation of PSROS had been adjusted properly for shifting patient care costs from Federal patients to non-Federal patients, what would the study have shown? Answer. The HCFA evaluation of benefit is not premised on the belief that costs shifted to other patients become savings to Medicare. As in the real world of hospital finance, the PSRO evaluation technique apportions the savings and costs between Medicare and other users, based upon the relative shares of respective utilization. Since one cannot directly observe the change in expenditures associated with the effects of PSRO review of Medicare patients, the benefits are calculated by estimating the change in utilization in number of days saved and multiplying these by the estimated value of a day saved. It is in the estimation of the value of a day saved where the cost shifting among patient groups becomes an issue. One possible method is to take the conventional view that variable costs represent about 40 percent of total costs. In this case, the benefit cost ratio would be higher than the 1.1 in the 1978 Evaluation. However, rather than establishing the value of a day saved purely by assumption, the evaluation takes a more sophisticated approach by using a sensitivity analysis to simulate changes in Medicare reimbursements using cost data from 5,000 hospitals. This results in an estimated 36 percent savings for Medicare in the average per diem. The design of the methodology and choice of assumptions is a conservative measure of the value of a day saved. In order to determine accurately the cost shifting between Federal and nonFederal patients due to PSRO review, expenditure data on all patients would be required plus sufficient data describing the health care environment to permit adjustments to cost shifting due to confounding factors other than PSRO review In the absence of a comprehensive hospital cost reporting system, data on hospital expenditures for non-Federal patients are not available. Since data constraints necessarily limited the PSRO Evaluation calculations to Medicare beneficiaries, the only patient group for which there were abundant data, it is clear that the cost shifting issue cannot be answered empirically until data are available for non-Federal as well as Federal patients. One could develop a crude estimate of total cost reductions from calculations involving some of the numbers produced in the HCFA sensitivity analyses. Using the same methodology as CBO used to estimate a benefit cost ratio, total cost reductions due to a Medicare utilization change of 2 percent would yield a benefit cost ratio of .917. Question 3. CBO stated that any cost-benefit analysis of the PSRO program has a sizable margin of error. Your testimony implies the same thing. If the actual cost-benefit ratio falls somewhere in the range established by your estimate and theirs, isn't this range a poor showing for the program? Why wasn't the range disclosed by HCFA? Answer. The flexible range for the benefit/cost ratio is evident in several places in the 1978 PSRO Evaluation Report. The range in the benefit/cost ratio across PSROs is discussed in the disaggregated analysis (Pages 172-181). The sensitivity of the benefit/cost to different unit cost calculations is demonstrated in this section (4.3.5) as well. In the section on the benefits of decreased utilization (Pages 162-169), four subcomponents were identified: resource realloca · tion effect and an input purchase effects for both routine and ancillary services. Offsetting substitution effects were also suggested for increases in the use of ambulatory care and long term care. Calculations of the potential benefits from reduced utilization were performed for ten different combinations of subcomponents. Long term care substitution ratios were tried between zero and 0.25 and ambulatory care substitution ratios between zero and 1.0 were considered. In addition, six different weighting schemata were used for redistributing the cost savings among hospitals in PSROS by length of stay, cost per discharge and other characteristics. As different assumptions were applied in the benefit calculations, the changes in savings were shown in dollars and in the percent of average per diem. Hence, as the value of the day saved was shown to be sensitive to alternative assumptions, a range of benefit/cost ratios is inevitable. As mentioned in Dr. Gaus' testimony, the various assumptions lead to benefit/ cost estimates from 0.55 to 1.65. These are not made explicit in the benefit/cost section of the report. However, estimates tended to cluster at 1.1, making it the appropriate figure to stress in the report. Further, it was more important to show the changes in the benefit/cost ratios among PSROS in the benefit/cost results section rather than alternatives to the 1.1 aggregate benefit/cost. The CBO and HCFA estimated benefit/cost ratios encompassing a range from C.7 to 1.1. If the "true" benefit/cost ratio is within this range, one might assume a benefit/cost in the neighborhood of unity or perhaps 0.9. We do not believe that such a benefit/cost ratio speaks poorly for the program because as the program matures, the benefits are likely to be sustained or increased. As PSROS become more proficient, the benefits should grow. Also, numerous nonmeasurable benefits such as spillover to the non-Federal patients are not reflected in the benefit/cost ratios but they may nevertheless be significant and should not be ignored in judging PSRO effectiveness. Question 4. CBO has suggested that current statute would stand in the way of evaluating some PSRO activities by preventing HEW from setting up a good comparison group. In your opinion, what are the specific PSRO activities which you cannot evaluate by assigning PSRO to a comparison group, due to current law? Answer. Section 1155 (A) (1) of the Social Security Act states that: "... it shall be the duty and function of each Professional Standards Review Organization for any area to assume, at the earliest date practicable, responsibilities for the review of the professional activities in such areas of physicians and other health care practitioners and institutional and noninstitutional providers of health care services. . . . The Congressional intent of the above has been interpreted to mean that the PSRO review of all provider of health care services should be implemented as soon as possible in all parts of the country. The time frame is explicit in parts of the statute, as in paragraph 7(g) (2) : "The Secretary shall require any (PSRO) which is capable of exercising review responsibility with respect to ambulatory care services to perform review responsibility with respect to such services on and after a date not earlier than the date the organization is designated as a (PSRO) . . . and no later than two years after the date the organization has been so designated. . . ." Given the mandate to implement PSRO review as soon as possible in all areas, the Department has not felt that PSRO areas could be randomly assigned to "active" and "control" groups. The advantage of using experimental design in implementing a program is to minimize the number of "competing hypotheses" about what factors may be causing change or hindering a desired change. Although random controlled experimentation is one desirable method, there are a number of quasi-experimental analytic techniques which have been developed which permit one to assess the impact of a program. Therefore, although the results will not be absolutely definitive, we feel it is possible to evaluate most aspects of the PSRO program without random assignment. Question 5. Dr. Weeks, the President, AAPSRO, indicated during his testimony that intermediaries sometimes pay claims that are not authorized by PSRO. Was the HCFA evaluation able to take this type of factor into consideration? Answer. Medicare pays for days of care denied by PSROS in cases where a provider has been granted "favorable waiver presumption". Such providers are eligible for program payments for services that are denied as medically unreasonable or unnecessary, or as constituting custodial care because such providers have demonstrated that they have made all reasonable efforts to ensure that they only render covered services. Some days are also paid for because the PSRO certifies them at a lower level of care due to the unavailability of a long term care bed. This issue is discussed in some detail on pages 88-92 of the 1978 PSRO Evaluation Report. Basically, the position taken by HCFA in the analysis is that PSROS are effective only to the degree that reimbursed days are reduced. The HCFA analysis reflects a 1.5 percent reduction in paid claims. Days used but not certified by the PSRO or certified at a lower level of care, but nevertheless reimbursed by Medicare, are not savings. At best, these days represent potential savings from PSRO review. QUESTIONS FOR DR. SMITS PREVIOUS GAO REPORT AND SUBCOMMITTEE HEARINGS Question 1. We have just received HEW comments to GAO's report of last year. I am distressed that you seem to misconstrue GAO's findings on PSRO executive director salary levels. Does this mean you plan to take no further action to correct this problem? What do you expect your "re-examination" of the proper comparison groups to find? (Due June 30, 1979) Answer. We have not misconstrued the findings of the General Accounting Office (GAO) on PSRO Executive Director salaries. The Executive Director salary levels specified in PSRO Transmittal No. 55 (November 1977), which had been developed by Hay Associates, were deemed excessive by the GAO. To address this problem, the levels used to compute Executive Director salaries have not been changed in spite of increases which have occurred in these salary levels. We have continued to use the original levels as the basis for determining Executive Director salaries. Hay Associates had established criteria for setting maximum salary levels for PSRO Executive Directors using salaries for business executives as the comparison group. We are currently developing and refining the process for determining salary levels for Executive Directors. Based on a re-examination of comparison groups which could be used, PSRO Executive Director salary levels will be compared to salary levels for executives in service-related institutions, rather than businesses. Also, the upper level of the Executive Director's salary will be held at the Civil Service Executive Level Five or Three ($52,000 per year) for fiscal year 1979/80. We will be consistent with Congressional intent with regard to wage controls each fiscal year. And finally, we will provide the PSRO Boards of Directors with guidelines for evaluating the performance of each Executive Director to maintain high standards of performance. Question 2. What progress has been made in developing the 100 planned management information systems for PSROS? How will this information be used? Answer. As of August 1979, 120 PSRO data systems were in operation. An additional 50 PSRO data systems are scheduled to initiate operations within the next six to nine months. The information generated by PSRO data systems is used in a variety of ways. These data help to identify areas for focusing review and setting objectives by highlighting both problem areas and areas of good performance. PSRO data serve an important ongoing monitoring function-documenting that previously identified deficiencies have been corrected and that areas of good performance continue to maintain high standards. Further, a good data system will provide an economical means for PSROS to survey the care of patients at regular intervals, while focusing review resources on only a subset of that population. Question 3. What progress has HCFA and the AMA made in targeting significant areas of unnecessary surgery for PSRO review? Answer. A great deal of progress has been made in targeting surgical procedures for PSRO review. A major effort was undertaken to develop criteria to screen medical necessity of surgical procedures. A contract was awarded to the American Medical Association (AMA) to coordinate a project to develop sample sets of criteria for reviewing the medical necessity and appropriateness of surgical procedures. The AMA subcontracted with thirteen specialty organizations for criteria set development and assistance in the preparation of the criteria for publication. Each specialty organization addressed those procedures which account for approximately 75 percent of the surgical activity within its area of specialization. The final set of criteria will be distributed in October 1979. |