Economy and Efficiency Audits Economy and efficiency audits of programs and activities are a major category of audit work set out in both GAO's Policies/Procedure Manual and the "yellow book."69 This category of work has been an important component of GAO work, particularly since GAO's shift in the 1950s from voucher auditing to more comprehensive auditing. GAO defines these audits as: Reviews having economy and efficiency objectives determine how waste and inefficient The category of economy and efficiency audits also includes GAO's audits of operational or program compliance with legislative or administrative requirements. GAO's "yellow book" notes that economy and efficiency audits may consider whether an organization or program is following sound procurement practices, using the minimum amount of resources in producing the appropriate quantity of goods and services, avoiding duplication of effort, avoiding idleness and overstaffing, using efficient operating procedures, complying with requirements of laws and regulations, and maintaining an adequate system for measuring and reporting performance on economy and efficiency.” GAO economy and efficiency audits have covered a broad range of government programs through various types of studies. Most of these audits are performed by issue area staffs assigned to work on the particular government programs and issues. While many reports reflect relatively narrowly focused examinations of processes and efficiencies, some of GAO's reports in this category move into more subjective issues beyond the scope of "yellow book" auditing standards. Often, Congress requests studies of this kind; GAO also initiates studies to keep up with the range of programs and perform periodic spot checks of agencies and programs, particularly to detect problems with economy and efficiency in high-risk programs and any need for improvements in other programs. Due to the volume and diversity of work that is partly or wholly in this category, the panel cannot offer definitive judgments on the quality and usefulness of this entire category. The quality, depth, and approach of the reports reviewed by the panel vary enormously. Based on the reading and interviews conducted for this study, the panel offers the following observations: GAO's relations with the organizations it reviews in its economy and efficiency audits range from positive and cooperative to adversarial. While a healthy tension exists in most armslength auditing between the auditor and the organization being audited, constructive relations "Policies/Procedures Manual, p. 2.0-2, and the "yellow book," pp. 14-15. Chapters 6 and 7 of the "yellow book" present in considerable detail standards for conducting and reporting this type of audit. Economy and efficiency audits are covered in the "yellow book" as one component of "performance audits," the other component being what it calls "program audits" which review program effectiveness ("yellow book," pp. 14 and 16). 70 Policies/Procedures Manual, p. 2.0-2. 71"Yellow book," p. 15. produce recommendations that are more realistic, better grounded in an understanding of the program and organization, and more likely to be implemented. Some of the congressional and agency staff interviewed for this research expressed skepticism as to whether GAO evaluators maintain an "independent attitude and appearance" in their audits. Some interviewees in both the legislative and executive branches stated that they regard a negative report (sometimes referred to as a "gotcha" report) as the predetermined outcome of such audits. GAO's statements of its objectives in this category seem to support this suspicion, and the titles of the reports contribute to that impression. Many titles include such words as "mismanagement," or highlight waste, abuse and fraud, which may create negative hearings and headlines and contribute to agency perceptions that GAO is only featuring criticism and may be "out to get" the agency or program. Congressional and agency staff generally do not appear to respect GAO's expertise in judging economy and efficiency as highly as its skill in financial management and accounting. In interviews, many staff members said that GAO auditors and evaluators often begin with little knowledge of agency or program operations in the area audited, and sometimes end with findings that seem relatively superficial, unremarkable, or already widely recognized by the people working in or involved with the subject area. GAO's economy and efficiency audits tend to focus on findings and recommendations related to process, controls, and oversight. Two themes appear frequently in this type of GAO work: "better management needed" or "adequate oversight lacking." The panel reviewed a number of reports with these phrases in titles or summary recommendations. Of course, there is always room for improvement in efficiency and economy of management, and oversight is never perfect. GAO, however, appears to assume that more oversight will result in better performance. Yet, generally, it provides no evidence of the changes in performance to expect from implementation of its recommendations. For example: U.S. Army Corps of Engineers: Better Management Needed for Mobilization Support (GAO/NSIAD-93-116) asserts that headquarters oversight is inadequate, but its 10 pages never address the costs of implementing the recommendations or the expected performance improvements. The recommendations are procedural and the report acknowledges that the agency is making changes in the recommended direction (for example, shifting plans from global to regional conflict scenarios). Eastern Europe: AID's Indefinite Contracts Assist Privatization Efforts But Lack Oversight (GAO/NSIAD-94-61) concludes that, in general, host governments were satisfied with AIDfunded efforts and compared AID contracts favorably to other donor programs; nevertheless, the GAO report calls for increased AID oversight. The report does not specify what program benefits might ensue from creating more oversight. 72"Independent attitude and appearance" is a standard provided in the "yellow book," p. 22. In summary, much GAO economy and efficiency auditing tends to revolve around examinations and recommendations related to process, but does not analyze cost, priorities relative to other agency challenges, and likely impacts on effectiveness. GAO's conclusions and recommendations may make headlines, but they lack a good record of leading to actual change or to real performance improvement. In fact, they tend mainly to make managers increasingly risk averse. The problems that GAO notes in its audit reports often continue for years after it issues final recommendations, raising questions as to whether GAO is pursuing the most effective means of reviewing and reporting on the economy, efficiency, and overall value of program expenditures. GAO should recognize the diminishing returns of increasingly detailed procedural regulation and external oversight, and focus more on recommending practical action steps by agencies and program managers that can lead to concrete internal improvements in operations and capacity to accomplish core missions more economically and effectively. It is easy to say that improvements could be made; the more challenging issues are how, at what cost, and with what results. GAO should shift its perspective from immediate shortcomings of agencies and programs to a longer term view of systems that will result in better management and more accurate and reliable information on costs and associated outputs. RECOMMENDATION: GAO should establish standards for diagnosing root causes of problems and develop methods to evaluate costs and benefits of potential changes. GAO also should substantially increase its internal capacity and resources for performing cost analysis and assessing the costs and benefits of agency steps to make internal efficiency improvements. RECOMMENDATION: GAO economy and efficiency audits should include an assessment of the potential costs of implementing its recommendations as well as the likely benefits. These considerations should be included in the terms of reference developed for a job before the work starts. RECOMMENDATION: GAO should avoid making recommendations or using report titles that call for increases in oversight or for improvements in management without specific empirical findings that the recommendations likely will result in more cost-effective program outcomes. Program Evaluations Over the past two decades, GAO has significantly increased the volume of program evaluations it conducts. This trend has been driven by congressional requests, statutory authority established in 1970 and 1974 to do this type of work, and GAO's own recognition of the importance of reviewing and enhancing the performance of federal programs. At the same time that Congress asked GAO to develop program evaluation capacities and conduct evaluations in the 1960s and 1970s, agencies responsible for implementing programs also were developing program evaluation functions and expertise. However, the attention and staff resources that executive agencies devoted to these issues 89-838 98-6 fell rapidly in the 1980s” while congressional requests for GAO to do program evaluations increased. GAO uses the term "program evaluation" (used in GAO's authorizing legislation and reflected in the name of a division at GAO) and also the term "program results reviews" for work that evaluates how effectively programs and activities are meeting intended goals and purposes. GAO identifies the primary objective of "program results reviews" as "improving the extent to which federal programs and operations achieve congressionally-intended results." The audit framework of the "yellow book" covers "program audits" that consider effectiveness of programs. The range of GAO's program evaluation work has expanded considerably since GAO developed its "yellow book" standards in the 1970s and 1980s. The book suggests that what it calls "program audits" may: a. b. C. Assess whether the objectives of a proposed, new, or ongoing program are proper, suitable, or relevant; Determine the extent to which a program achieves a desired level of program results; Assess the effectiveness of the program and/or individual program components; d. Identify factors inhibiting satisfactory performance; e. f. Determine whether management has considered alternatives for carrying out the program that might yield desired results more effectively or at a lower cost; Determine whether the program complements, duplicates, overlaps, or conflicts with other related programs; g. Identify ways of making programs work better; h. Assess compliance with laws and regulations applicable to the program; i. j. Assess the adequacy of the management control system for measuring, reporting, and monitoring a program's effectiveness; and Determine whether management has reported measures of program effectiveness that are valid and reliable." "See Federal Evaluation--Fewer Units, Reduced Resources, Different Studies from 1980, GAO/PEMD-87-9; and Christopher G. Wye and Richard Sonnichsen, eds., Evaluation in the Federal Government: Changes, Trends, and Opportunities, 1992. Policies/Procedures Manual, p. 2.0-2. " Policies/Procedures Manual, p. 2.0-2. 76* Yellow book," p. 16. - These are appropriate objectives for GAO evaluations although the first item related to questioning the stated objectives of an existing program established by law potentially raises concerns in Congress among both supporters and critics of the programs GAO studies. In setting out the objectives for its program evaluations and in outlining purposes such as item (f) above, GAO also suggests a further application for this work: evaluating activities and results across programs and agencies. In interviews for this study, congressional and executive branch representatives cited the value of GAO's work in comparing and integrating assessments of programs with the same or similar objectives. One example is a recent summary from GAO's Education and Employment Issue Area of the number of federal programs and agencies involved in job training. GAO is now extending that analysis to compare the objectives, performance measures, and relative costs of the various programs in achieving their training purposes. Program evaluations are inherently controversial. Any positive or negative evaluation of a program will trigger the emotions of interested individuals and groups. Some of GAO's best and also some of its weakest work in this category has become the subject of controversy in Congress and the agencies. GAO can avoid some controversy by building strong interdisciplinary evaluation teams, giving special attention to establishing clear, objective, broadly accepted definitions of the questions the evaluations address, and relying on solid, credible data sources as the starting point for the analysis, and selecting appropriate methodology. - In 1980, GAO created the Institute for Program Evaluation, which became the Program Evaluation and Methodology Division (PEMD) in 1983. PEMD along with the Design and Methodology Technical Assistance Groups (DMTAGs) and the economics units in the divisions, and the Training Institute -- have contributed to improvements in GAO evaluation work. PEMD performs independent evaluations and advises other divisions. Interviews conducted for this study indicate that PEMD's recommendations to other issue area staffs about methodologies to use in conducting evaluations its special area of expertise do not always carry substantial weight. PEMD has a smaller staff and budget than any other GAO division (less than 2.5 percent of GAO's staffing resources in fiscal year 1993). The larger professional and academic community involved in program evaluation and congressional interviewees aware of PEMD's work generally respect PEMD's studies, which combine social science and auditing skills. PEMD occasionally has had difficulty satisfying its interest in methodological and analytical rigor and also meeting congressional time demands and needs for immediately and easily usable data." Consequently, PEMD has developed fast-track approaches to provide congressional requesters with information and evaluations more quickly. Program-oriented issue area staffs perform many of the GAO studies that fit into the category of program evaluations. They carry out these evaluations within several serious constraints. With few exceptions, the issue areas cannot afford or effectively manage time series studies, experimental research designs (with control populations, double blind observations, etc.), or large-scale surveys. They must design program evaluation strategies matched to their capabilities and to congressional needs, often under the serious time pressure of Congress's schedule of program and policy debates. "PEMD's major recent product, The U.S. Nuclear Triad: GAO's Evaluation of the Strategic Modernization Program, resulted in eight classified volumes (GAO/C-PEMD-92-1 to 92-8). |