which could not be tapped until foreign contributions had been used. Congress did rescind that amount from the defense budget. 54 GAO's current performance measurement systems are primarily process-oriented. Undoubtedly, GAO contributes every year to government's effectiveness, but its performance indicators do not capture that contribution because they do not focus on ultimate results. One GAO indicator reports "financial benefits" that could be attributed to GAO recommendations, which under GAO policy are to be estimated net of the added costs associated with implementing the recommendations.55 By the nature of its performance indicators, GAO creates somewhat distorted incentives for its employees. The major mission and goals of the organization are related not to the number of its recommendations that are put into action, but to their contributions to improved efficiency and effectiveness of programs, activities, systems, agencies, and the larger government. RECOMMENDATION: GAO should revise its internal performance indicators and accomplishment RECOMMENDATION: GAO should increase its internal resources and skills for analyzing the costs and budget impacts of changes it recommends. External Peer Review While GAO has a highly organized and systematic process for internal review before a report or formal work product is released, it currently lacks a formal process for external peer review of the body of work it produces. Objective third-party peer review is commonly recommended in academic and research organizations and in private sector audit firms. In fact, the comptroller general has expressed support for the idea of external quality review. “In interviews, OMB officials said that they were monitoring use of the appropriation and would not have permitted the Defense Department to retain the unused funds, whether or not GAO had testified on the matter. "Estimates of the costs of implementing GAO recommendations are not included in individual GAO reports. See, for example, GAO/T-OPP-93-1, pp. 14 and 19. The large number of requests for GAO's reports and other products, month after month, would produce an overwhelming burden on any external body attempting to perform a formal review before jobs start or as part of the review process, before release of each report. Panels of experts could, however, assess a body of recently completed reports and testimony in a particular issue area or substantive category. Peer review panels, with expertise related to the particular category of work, could usefully evaluate the appropriateness and reliability of the sources and methodologies, the objectivity and quality of the research and presentation, the practicality of recommendations, and other elements related to the usefulness and value of products. Peer review panels would comment on individual reports and also should consider general GAO operations and the nature of the topics and types of work it conducted. - The panel recommends that GAO initiate an ongoing, systematic process of external peer review, which would examine three or four categories of GAO work every year. In the panel's view, GAO should devote a portion of its annual operating budget at least the cost of conducting one major research project to support such a peer review process, covering a sampling of recent GAO work products. One or more neutral outside parties should organize the review process or the individual panels, to avoid any perception of bias. This would satisfy GAO's interest in gaining the perspectives of its professional peers on its work, and also would provide regular objective assessments of GAO products and processes for consideration in regular oversight hearings. RECOMMENDATION: GAO should establish and commit to funding a continuing process of regular external peer review of its work. Panels of objective, third-party experts should be convened to assess a sampling of recently completed GAO reports and other work products in the panel members' areas of expertise. Three or four major areas or categories of GAO work should be selected for peer review each year. The review should cover the quality of individual studies, and more general questions on GAO methodologies, sources, practicality of recommendations and presentation of results. V. GAO'S FUNCTIONS AND WORK PRODUCTS GAO's work ranges across several broad categories, including financial-related audits; audits of economy and efficiency; program evaluations; policy analysis and work related to developing legislation; general management reviews and other general management-related activities; and special investigations." In addition, GAO performs a considerable amount of work that involves primarily gathering and presenting information, without substantial analysis. GAO also continues to participate in developing accounting and auditing standards for federal agencies. The panel and project staff reviewed examples of GAO work and conducted interviews related to reports in most of these categories, which are discussed in this chapter. Generally, the categories correspond to types of activity that GAO has developed and recognized as primary areas of work, as reflected in GAO's Policies/Procedures Manual, the "yellow book," and other internal planning and management documents. GAO does not have a single, uniform set of categories for classifying its work, and the categories it uses are somewhat loose. The panel reviewed GAO work and the categories it applies, and chose categories that are recognized and accepted in GAO and in the wider government and audit communities.58 These categories may not be ideal; there are some ambiguities in the groupings. However, there are similar limitations in any other categorization. Many of these types of work are produced by a number of different issue area staffs; the categories do not all correspond to an individual GAO organizational unit. In addition, many GAO projects and work products overlap several categories." The categories, however, provide a useful starting point for considering the broad range of GAO's work. Definitions of Work Categories 1. The panel considered the following categories of GAO work: Financial audits. GAO financial audits include financial statement audits and financial-related audits. These include audits of financial statements or segments of financial statements, 57 As part of several of these categories, GAO reviews the compliance of agency actions and programs with federal laws and regulations ("compliance audits"). It also reviews protested federal contract awards and provides other legal advice; the panel did not review GAO operations in these legal areas. 58These categories are compatible with those that the Senate Committee on Governmental Affairs used in the statement of work for this study. "GAO recognizes that categories of work are not neatly defined and also that practice may not precisely match formal standards and objectives. GAO notes in the "yellow book" (p. 8): "These standards are more than the codification of current practices. They include 2. 3. 4. financial information and financial systems, internal controls over financial reporting (including computer-based systems) and over compliance with laws and regulations governing contracts and bids, actual compliance with laws and regulations, and related allegations of fraud.60 Economy and efficiency audits. This category of GAO's work is designed to determine whether an entity is acquiring and using resources economically and efficiently, the causes of inefficiencies or uneconomical practices, and whether the entity has complied with laws and regulations concerning matters of economy and efficiency." 61 Program evaluations. The term "program evaluations" is the term commonly given to reviews that assess whether, and how much, the desired results or benefits established by the legislative or other authorizing body are being achieved by organizations, programs, activities, or functions and their effectiveness in achieving those results. Such reviews can relate to a particular program, a single component of a program or agency function, or a group of programs and activities across the agency or agencies.62 Policy analysis and policy development. Perhaps the most difficult category to define discretely, this activity includes presentations of information and analytical findings, evaluation of options for addressing specific policy objectives and interests, assessments of past experience and projections of probable costs or outcomes, or both, of particular policy options. This category also includes GAO's comments on policy changes proposed in legislation, as well as assistance in drafting legislation that would formulate new policy. 5. General management reviews and related activities. In the last 10 years, GAO has conducted a series of studies that it generically labeled general management reviews, which examined the operations, strengths, and weaknesses of overall management structures, approaches, and systems in federal organizations. They have focused primarily on an agency's general management systems, organizational structure, and processes, rather than on the functioning of specific programs. 6. Special investigations. Special investigations at GAO follow up on indications of "fraud, 60" Yellow book," pp. 12-14. "The "yellow book" uses the term "performance audits" to refer collectively to economy and efficiency audits and to what it refers to as "program audits," a type of audit that considers the effectiveness of programs (p. 14). This study discusses reviews of program effectiveness under the category of "program evaluations," a category of work that by definition looks more deeply into questions of effectiveness than the more process-oriented category of audits. 62GAO also uses the term "program results reviews" for this type of study of the effectiveness of the outcome or results of programs. Policies/Procedures Manual, November 1992, p. 2.0-2. The panel set out to evaluate GAO reports in those categories, beginning by reviewing the range of work in GAO's Indexes and Abstracts of Reports and Testimony for fiscal years 1990-1992. The panel also reviewed recent reports that GAO identified as its best, and examples of reports or streams of work that GAO or congressional members identified as particularly controversial or otherwise problematic. As examples of work within a more concentrated time period and a more specific program area, the project staff also read all GAO reports and testimony released in June 1993 and all 146 of GAO's transportation-related reports and testimony from 1989 to 1993. From these hundreds of documents, the panel and staff concluded that although the above list of categories of work accurately describes the range of GAO work, individual projects and work products often cannot be discretely categorized as belonging exclusively to one or another type. The following sections reflect assessments of reports and testimony that fit into the particular category of work at least in part. Financial Audits Financial audits, including audits of agency financial statements and other financial-related audits, are most closely associated with GAO's historic role. GAO's most established and best developed standards and policies for audit work cover financial-related audits.“ 63 In the decades before the CFO Act, GAO devoted decreasing staff time to financial auditing. Since 1990 and passage of the CFO Act, a growing component of GAO's financial work has involved advising and assisting agencies in conducting their own internal accounting, financial reporting, financial auditing, and financial management -- a pressing need today in the federal government. GAO has focused more attention on this area in the last decade as Congress, federal agencies and the public have placed renewed emphasis on financial accountability. In its Policies/Procedures Manual, GAO describes "financial management reviews" as one of its major areas of work." GAO's 1992 Financial Management transition report listed 25 major reports on financial management issues from 1985-1992.65 One product in this category of work is GAO's "High-Risk Series, "66 a set of red-covered volumes which GAO first issued in December 1992 and sent to the congressional leadership, committee and subcommittee chairs and ranking minority members, the President, OMB, and affected agencies. These reports summarize the challenges and deficiencies in internal controls and financial management systems in 17 programs, agencies, or both, that GAO identified as especially vulnerable 63" Yellow book," pp. 12-14 and 32-61. "GAO identifies the key objectives of financial management reviews as "improving agency and program accounting and financial management. Reviews having these objectives... help ensure that funds are spent prudently, as intended by the Congress, and are properly accounted for; that property is adequately controlled; and that managers have the timely and reliable information they need to manage their operations effectively." Policies/Procedures Manual, p. 2.0-1. ❝GAO/OCG-93-4TR, pp. 37-40. The list does not include GAO's High-Risk Series. "GAO/HR-93-1 through GAO/HR-93-17. |