RECOMMENDATION: GAO should seek agency comments on its reports with a 30-day limit for Follow-up and Performance Indicators After GAO reports are issued, the law requires agencies to respond to the findings and recommendations within 60 days (though agencies routinely respond late or not at all). GAO has been surprised by the content of some responses and the strength of agency disagreement, reflecting weakness in GAO's prior communications. In some cases, the misunderstandings or disagreements between GAO and agencies persist over several years and several reports. Breakdowns of this kind in coordination and communications with agencies are likely to block implementation of GAO recommendations. This is another reason for constructive and cooperative relationships and communications with the agency subject to study in advance of release of a GAO report. This point has been demonstrated in GAO's general management reviews, which have involved a high degree of cooperation and consultation with executive agencies (see pp. 53-54). In following up on its completed reports, GAO uses a variety of indicators to assess its performance and the effectiveness of its work. One of the most visible measures is the percentage of recommendations that have been implemented. GAO tracks "open recommendations" -- i.e., those that are not yet reported to be implemented -- and reports on them twice a year, with statistics compiled to cover what has been done in the most recent four- and five-year periods. For example, GAO reported that from fiscal years 1989 to 1992, it made 6,140 recommendations. By the end of 1992, the legislative branch had implemented about 55 percent of the recommendations for congressional action, while the executive branch had implemented 75 percent of the recommendations for executive branch action. The document does not report on the specific elements, costs, or results or impacts of implementation -- i.e., whether government performance improved and the public benefited. The many GAO reports that do not have recommendations but simply produce facts and "findings" are not included in the report on open recommendations. In addition to tracking recommendations, GAO's staffs also file accomplishment reports. These provide information on both dollar savings and non-quantifiable impacts that can be associated with their recommendations. The 294 accomplishments that GAO reported in fiscal year 1992 included $36.2 billion in savings, of which $30.16 billion came from defense-related issue areas. Of that, $14.7 billion were attributable to rescission from the "Desert Storm" account appropriated to pay for the Persian Gulf War. GAO had testified that a special $15 billion appropriation requested by the Defense Department might not be needed, and that it should be kept in a special restricted account 53 See 31 U.S.C. § 720; OMB and agencies also issue directives to implement the requirement. which could not be tapped until foreign contributions had been used. Congress did rescind that amount from the defense budget.“ GAO's current performance measurement systems are primarily process-oriented. Undoubtedly, GAO contributes every year to government's effectiveness, but its performance indicators do not capture that contribution because they do not focus on ultimate results. One GAO indicator reports "financial benefits" that could be attributed to GAO recommendations, which under GAO policy are to be estimated net of the added costs associated with implementing the recommendations. 55 By the nature of its performance indicators, GAO creates somewhat distorted incentives for its employees. The major mission and goals of the organization are related not to the number of its recommendations that are put into action, but to their contributions to improved efficiency and effectiveness of programs, activities, systems, agencies, and the larger government. RECOMMENDATION: GAO should revise its internal performance indicators and accomplishment RECOMMENDATION: GAO should increase its internal resources and skills for analyzing the costs and budget impacts of changes it recommends. External Peer Review While GAO has a highly organized and systematic process for internal review before a report or formal work product is released, it currently lacks a formal process for external peer review of the body of work it produces. Objective third-party peer review is commonly recommended in academic and research organizations and in private sector audit firms. In fact, the comptroller general has expressed support for the idea of external quality review.56 "In interviews, OMB officials said that they were monitoring use of the appropriation and would not have permitted the Defense Department to retain the unused funds, whether or not GAO had testified on the matter. "Estimates of the costs of implementing GAO recommendations are not included in individual GAO reports. "See, for example, GAO/T-OPP-93-1, pp. 14 and 19. The large number of requests for GAO's reports and other products, month after month, would produce an overwhelming burden on any external body attempting to perform a formal review before jobs start or as part of the review process, before release of each report. Panels of experts could, however, assess a body of recently completed reports and testimony in a particular issue area or substantive category. Peer review panels, with expertise related to the particular category of work, could usefully evaluate the appropriateness and reliability of the sources and methodologies, the objectivity and quality of the research and presentation, the practicality of recommendations, and other elements related to the usefulness and value of products. Peer review panels would comment on individual reports and also should consider general GAO operations and the nature of the topics and types of work it conducted. The panel recommends that GAO initiate an ongoing, systematic process of external peer review, which would examine three or four categories of GAO work every year. In the panel's view, GAO should devote a portion of its annual operating budget at least the cost of conducting one major research project -- to support such a peer review process, covering a sampling of recent GAO work products. One or more neutral outside parties should organize the review process or the individual panels, to avoid any perception of bias. This would satisfy GAO's interest in gaining the perspectives of its professional peers on its work, and also would provide regular objective assessments of GAO products and processes for consideration in regular oversight hearings. RECOMMENDATION: GAO should establish and commit to funding a continuing process of regular external peer review of its work. Panels of objective, third-party experts should be convened to assess a sampling of recently completed GAO reports and other work products in the panel members' areas of expertise. Three or four major areas or categories of GAO work should be selected for peer review each year. The review should cover the quality of individual studies, and more general questions on GAO methodologies, sources, practicality of recommendations and presentation of results. V. GAO'S FUNCTIONS AND WORK PRODUCTS GAO's work ranges across several broad categories, including financial-related audits; audits of economy and efficiency; program evaluations; policy analysis and work related to developing legislation; general management reviews and other general management-related activities; and special investigations." In addition, GAO performs a considerable amount of work that involves primarily gathering and presenting information, without substantial analysis. GAO also continues to participate in developing accounting and auditing standards for federal agencies. The panel and project staff reviewed examples of GAO work and conducted interviews related to reports in most of these categories, which are discussed in this chapter. Generally, the categories correspond to types of activity that GAO has developed and recognized as primary areas of work, as reflected in GAO's Policies/Procedures Manual, the "yellow book," and other internal planning and management documents. GAO does not have a single, uniform set of categories for classifying its work, and the categories it uses are somewhat loose. The panel reviewed GAO work and the categories it applies, and chose categories that are recognized and accepted in GAO and in the wider government and audit communities.58 These categories may not be ideal; there are some ambiguities in the groupings. However, there are similar limitations in any other categorization. Many of these types of work are produced by a number of different issue area staffs; the categories do not all correspond to an individual GAO organizational unit. In addition, many GAO projects and work products overlap several categories." The categories, however, provide a useful starting point for considering the broad range of GAO's work. Definitions of Work Categories 1. The panel considered the following categories of GAO work: Financial audits. GAO financial audits include financial statement audits and financial-related audits. These include audits of financial statements or segments of financial statements, 57As part of several of these categories, GAO reviews the compliance of agency actions and programs with federal laws and regulations ("compliance audits"). It also reviews protested federal contract awards and provides other legal advice; the panel did not review GAO operations in these legal areas. 58These categories are compatible with those that the Senate Committee on Governmental Affairs used in the statement of work for this study. "GAO recognizes that categories of work are not neatly defined and also that practice may not precisely match formal standards and objectives. GAO notes in the "yellow book" (p. 8): "These standards are more than the codification of current practices. They include 2. 3. 4. financial information and financial systems, internal controls over financial reporting (including computer-based systems) and over compliance with laws and regulations governing contracts and bids, actual compliance with laws and regulations, and related allegations of fraud.“ Economy and efficiency audits. This category of GAO's work is designed to determine whether an entity is acquiring and using resources economically and efficiently, the causes of inefficiencies or uneconomical practices, and whether the entity has complied with laws and regulations concerning matters of economy and efficiency."1 Program evaluations. The term "program evaluations" is the term commonly given to reviews that assess whether, and how much, the desired results or benefits established by the legislative or other authorizing body are being achieved by organizations, programs, activities, or functions and their effectiveness in achieving those results. Such reviews can relate to a particular program, a single component of a program or agency function, or a group of programs and activities across the agency or agencies. 62 Policy analysis and policy development. Perhaps the most difficult category to define discretely, this activity includes presentations of information and analytical findings, evaluation of options for addressing specific policy objectives and interests, assessments of past experience and projections of probable costs or outcomes, or both, of particular policy options. This category also includes GAO's comments on policy changes proposed in legislation, as well as assistance in drafting legislation that would formulate new policy. 5. General management reviews and related activities. In the last 10 years, GAO has conducted a series of studies that it generically labeled general management reviews, which examined the operations, strengths, and weaknesses of overall management structures, approaches, and systems in federal organizations. They have focused primarily on an agency's general management systems, organizational structure, and processes, rather than on the functioning of specific programs. 6. - Special investigations. Special investigations at GAO follow up on indications of "fraud, 60" Yellow book," pp. 12-14. "The "yellow book" uses the term "performance audits" to refer collectively to economy and efficiency audits and to what refers to as "program audits," a type of audit that considers the effectiveness of programs (p. 14). This study discusses reviews of program effectiveness under the category of "program evaluations," a category of work that by definition looks more deeply into questions of effectiveness than the more process-oriented category of audits. 62GAO also uses the term "program results reviews" for this type of study of the effectiveness of the outcome or results of programs. Policies/Procedures Manual, November 1992, p. 2.0-2. |