"When the comments oppose the report's findings, conclusions, or recommendations, and are not, in the auditors' opinion, valid, the auditors may choose to state their reasons for rejecting them. Conversely, the auditors should modify their report if they find the comments valid. "Si Agency comment on GAO reports serves several purposes. Obtaining agency comments significantly reduces the potential for factual errors in a GAO report or misunderstandings about the research and recommendations. It reveals language or phrases that would convey unintended meanings or raise sensitivities in the agency or broader audience that GAO may not anticipate. The opportunity for agency comment also increases the organization's awareness of and cooperative attitude toward the GAO work, and thus significantly increases the possibility for successful implementation of GAO's recommendations and for effective improvements in that organization. For GAO, the comment process offers additional advantages: to reveal more clearly in advance political values and policy differences related to a particular report, to help define the costs of and obstacles to implementing the recommendations, and appropriate boundaries of GAO recommendations, and to signal areas requiring additional fact checking. The major arguments presented by congressional and GAO staffs against soliciting agency comments are to avoid further delay, to avoid pressure on GAO to change its findings, and to allow the requester to control the release and hearing process. Nevertheless, GAO's failure to obtain comments from the organization under study damages GAO's credibility, reduces the accuracy and thoroughness of its work, and harms its relationships with executive agencies. In fact, the absence of comments in many cases merely postpones debate; GAO reports become more controversial, and GAO becomes more vulnerable to criticism than it might be with reasonable consultation with the agency before the report is released. Routinely obtaining agency comments on GAO reports need not lengthen the total time GAO takes to complete a job; by adjusting internal work processes as the panel recommends and making use of the information and fact-checking the agency provides, GAO will be able to reduce the time it devotes to sequential internal reviews, offsetting the time spent on obtaining agency comments. To contribute to the fact-gathering and cooperation in the course of an audit or evaluation, GAO should consider giving agencies an earlier draft of its findings and conclusions, while continuing to review the draft internally." It should organize the project team to respond quickly and constructively to agency comments. 51"Yellow book," pp. 95-96. Private audit firms also obtain comments from the organizations they audit or evaluate. 52GAO should make exceptions from these procedures for reports involving criminal investigations or audits that involve individual company's business contracts or potential legal action. GAO should also continue to limit the review and distribution of reports on classified national security issues. RECOMMENDATION: GAO should seek agency comments on its reports with a 30-day limit for Follow-up and Performance Indicators After GAO reports are issued, the law requires agencies to respond to the findings and recommendations within 60 days3 (though agencies routinely respond late or not at all). GAO has been surprised by the content of some responses and the strength of agency disagreement, reflecting weakness in GAO's prior communications. In some cases, the misunderstandings or disagreements between GAO and agencies persist over several years and several reports. Breakdowns of this kind in coordination and communications with agencies are likely to block implementation of GAO recommendations. This is another reason for constructive and cooperative relationships and communications with the agency subject to study in advance of release of a GAO report. This point has been demonstrated in GAO's general management reviews, which have involved a high degree of cooperation and consultation with executive agencies (see pp. 53-54). In following up on its completed reports, GAO uses a variety of indicators to assess its performance and the effectiveness of its work. One of the most visible measures is the percentage of recommendations that have been implemented. GAO tracks "open recommendations" -- i.e., those that are not yet reported to be implemented -- and reports on them twice a year, with statistics compiled to cover what has been done in the most recent four- and five-year periods. For example, GAO reported that from fiscal years 1989 to 1992, it made 6,140 recommendations. By the end of 1992, the legislative branch had implemented about 55 percent of the recommendations for congressional action, while the executive branch had implemented 75 percent of the recommendations for executive branch action. The document does not report on the specific elements, costs, or results or impacts of implementation -- i.e., whether government performance improved and the public benefited. The many GAO reports that do not have recommendations but simply produce facts and "findings" are not included in the report on open recommendations. In addition to tracking recommendations, GAO's staffs also file accomplishment reports. These provide information on both dollar savings and non-quantifiable impacts that can be associated with their recommendations. The 294 accomplishments that GAO reported in fiscal year 1992 included $36.2 billion in savings, of which $30.16 billion came from defense-related issue areas. Of that, $14.7 billion were attributable to rescission from the "Desert Storm" account appropriated to pay for the Persian Gulf War. GAO had testified that a special $15 billion appropriation requested by the Defense Department might not be needed, and that it should be kept in a special restricted account 53See 31 U.S.C. § 720; OMB and agencies also issue directives to implement the requirement. which could not be tapped until foreign contributions had been used. Congress did rescind that amount from the defense budget." GAO's current performance measurement systems are primarily process-oriented. Undoubtedly, GAO contributes every year to government's effectiveness, but its performance indicators do not capture that contribution because they do not focus on ultimate results. One GAO indicator reports "financial benefits" that could be attributed to GAO recommendations, which under GAO policy are to be estimated net of the added costs associated with implementing the recommendations. 55 By the nature of its performance indicators, GAO creates somewhat distorted incentives for its employees. The major mission and goals of the organization are related not to the number of its recommendations that are put into action, but to their contributions to improved efficiency and effectiveness of programs, activities, systems, agencies, and the larger government. RECOMMENDATION: GAO should revise its internal performance indicators and accomplishment RECOMMENDATION: GAO should increase its internal resources and skills for analyzing the costs and budget impacts of changes it recommends. External Peer Review While GAO has a highly organized and systematic process for internal review before a report or formal work product is released, it currently lacks a formal process for external peer review of the body of work it produces. Objective third-party peer review is commonly recommended in academic and research organizations and in private sector audit firms. In fact, the comptroller general has expressed support for the idea of external quality review. "In interviews, OMB officials said that they were monitoring use of the appropriation and would not have permitted the Defense Department to retain the unused funds, whether or not GAO had testified on the matter. "Estimates of the costs of implementing GAO recommendations are not included in individual GAO reports. "See, for example, GAO/T-OPP-93-1, pp. 14 and 19. The large number of requests for GAO's reports and other products, month after month, would produce an overwhelming burden on any external body attempting to perform a formal review before jobs start or as part of the review process, before release of each report. Panels of experts could, however, assess a body of recently completed reports and testimony in a particular issue area or substantive category. Peer review panels, with expertise related to the particular category of work, could usefully evaluate the appropriateness and reliability of the sources and methodologies, the objectivity and quality of the research and presentation, the practicality of recommendations, and other elements related to the usefulness and value of products. Peer review panels would comment on individual reports and also should consider general GAO operations and the nature of the topics and types of work it conducted. - The panel recommends that GAO initiate an ongoing, systematic process of external peer review, which would examine three or four categories of GAO work every year. In the panel's view, GAO should devote a portion of its annual operating budget at least the cost of conducting one major research project to support such a peer review process, covering a sampling of recent GAO work products. One or more neutral outside parties should organize the review process or the individual panels, to avoid any perception of bias. This would satisfy GAO's interest in gaining the perspectives of its professional peers on its work, and also would provide regular objective assessments of GAO products and processes for consideration in regular oversight hearings. RECOMMENDATION: GAO should establish and commit to funding a continuing process of regular external peer review of its work. Panels of objective, third-party experts should be convened to assess a sampling of recently completed GAO reports and other work products in the panel members' areas of expertise. Three or four major areas or categories of GAO work should be selected for peer review each year. The review should cover the quality of individual studies, and more general questions on GAO methodologies, sources, practicality of recommendations and presentation of results. V. GAO'S FUNCTIONS AND WORK PRODUCTS GAO's work ranges across several broad categories, including financial-related audits; audits of economy and efficiency; program evaluations; policy analysis and work related to developing legislation; general management reviews and other general management-related activities; and special investigations." In addition, GAO performs a considerable amount of work that involves primarily gathering and presenting information, without substantial analysis. GAO also continues to participate in developing accounting and auditing standards for federal agencies. The panel and project staff reviewed examples of GAO work and conducted interviews related to reports in most of these categories, which are discussed in this chapter. Generally, the categories correspond to types of activity that GAO has developed and recognized as primary areas of work, as reflected in GAO's Policies/Procedures Manual, the "yellow book," and other internal planning and management documents. GAO does not have a single, uniform set of categories for classifying its work, and the categories it uses are somewhat loose. The panel reviewed GAO work and the categories it applies, and chose categories that are recognized and accepted in GAO and in the wider government and audit communities. 58 These categories may not be ideal; there are some ambiguities in the groupings. However, there are similar limitations in any other categorization. Many of these types of work are produced by a number of different issue area staffs; the categories do not all correspond to an individual GAO organizational unit. In addition, many GAO projects and work products overlap several categories." The categories, however, provide a useful starting point for considering the broad range of GAO's work. Definitions of Work Categories 1. The panel considered the following categories of GAO work: Financial audits. GAO financial audits include financial statement audits and financial-related audits. These include audits of financial statements or segments of financial statements, 57As part of several of these categories, GAO reviews the compliance of agency actions and programs with federal laws and regulations ("compliance audits"). It also reviews protested federal contract awards and provides other legal advice; the panel did not review GAO operations in these legal areas. "These categories are compatible with those that the Senate Committee on Governmental Affairs used in the statement of work for this study. "GAO recognizes that categories of work are not neatly defined and also that practice may not precisely match formal standards and objectives. GAO notes in the "yellow book" (p. 8): "These standards are more than the codification of current practices. They include |