criticism from others. The panel concluded that the highly centralized, cumbersome review process does not weed out problems effectively or protect solid but controversial findings from criticism. The objectives behind the sequential tiers of review do not seem clear to participants or observers, and the outcomes are inconsistent. The reviews reflect an emphasis on process, not the content and effectiveness of GAO work. In addition, GAO uses a hierarchical approach that is not organizationally suited to interdisciplinary research. Overall GAO quality control procedures give insufficient attention to careful selection and design of jobs and consideration of methodology at the beginning of work and rely too heavily on multiple reviews at the end of jobs. In essence, the process has more drawbacks than advantages. In the panel's view, the weaknesses in GAO's internal job design and work processes can be alleviated by having clear "terms of reference" (TORs) before a job starts. TORS should be agreed to by GAO management and staff involved in the study and also by congressional requesters. The TORS should clearly define the objectives and scope of work; outline the general questions to be addressed and the methods to be used in the research; indicate the skills that the study team will apply; suggest sources and data requirements; and estimate time and costs to complete the work. Having clear, mutually agreed TORS would increase the likelihood that GAO would be responding to the questions a requester wants answered. In addition, the TORS should be part of the job-start review process within GAO and serve as the foundation for design of the work. Clear TORS would also create a solid basis for any presentations and briefings on a project either inside GAO or to Congress and to agencies that are the subjects of study. Finally, TORS would supply operational criteria for the GAO review process. This approach should: Produce clarity on a study's objectives and topics; Help GAO and requesters identify and pursue priorities; Effectively target and limit GAO's work; Improve communications with congressional and executive branch representatives involved; Strengthen substantially the quality of GAO work; and Reduce the possibility of, or need for, conflict, disappointment, or rework in end-ofstudy reviews. GAO has already launched a number of initiatives in the direction of improving work design and quality control, emphasizing quality in a job from the beginning and bringing top management into designing the work and participating in deliberations about its progress throughout the course of the project. As part of its quality management initiative, GAO is conducting pilot studies on "reducing rework" and improving the communications and involvement among staff and managers in field and headquarters.50 One theme of GAO's quality management effort is to build in quality from the start of a job. If the approach is faulty at the beginning of the job, it is difficult to produce a strong product at the end. The quality of the work, and the shared understanding of job objectives, are much improved if GAO includes in discussions at the start the senior managers and others likely to be responsible for reviewing the report at the end. RECOMMENDATION: To improve the quality of its work, GAO should, first and foremost, develop thorough "terms of reference" for each job, to which GAO staff, managers, and requesters, if any, agree, before the job is approved and work begins. Terms of reference should include a clear written statement of the objectives and research questions; scope of work; general methods and sources; staff skills, costs, and time needed to complete the work. RECOMMENDATION: GAO should form interdisciplinary teams at the start of a job, to participate jointly RECOMMENDATION: GAO should increase the level of management involvement with staff in planning Agency Comment The final stages of GAO jobs may include holding an exit conference with an executive agency involved in the study and obtaining written comments from the agency on a draft; arranging for release of the report as the requester requires; providing for distribution, hearings and testimony, if any; and securing agency response to the published report, in a "60-day letter." soThe Food and Agriculture Issue Area is testing one model for organizing headquarters and field operations, referred to as "Team Agriculture," involving efforts to coordinate headquarters and field staffs, improve electronic communications, and reduce delays in reviews and other parts of a project. Over the last decade, some committees and members of Congress have altered the processes involved in concluding projects and releasing reports in ways that appear to serve political or publicitydriven objectives. The indicators of this trend are: Reduction in what used to be standard GAO procedure -- getting written comments from the agency subject to the audit or evaluation (Some committee chairs even ask GAO not to brief the agency orally before a report is completed and released); Congressional restrictions on release of GAO work, ranging from holding only confidential briefings on GAO findings to 30-day "embargo" while a committee or member arranges for a press conference, hearing, or special release of the report; Scheduling of hearings related to GAO work without substantial notice to, or consultation with, the agency or other congressional members and committees likely to disagree with the report; GAO's occasional (though rare) release of self-initiated reports without clear prior notice to legislative or executive officials likely to have an interest in the work, taking these officials by surprise; and GAO exit conferences with an agency subject to review that generate serious misunderstandings between GAO and the agency in question (because, for instance, the draft findings reviewed by the agency and the final report differ, or the agency and GAO's managers had widely differing perceptions of what the conference concluded). Such practices conflict with GAO's own standards, particularly for obtaining agency comments. Its Government Auditing Standards (the "yellow book") states: "Auditors should report the views of responsible officials of the audited program "One of the most effective ways to ensure that a report is fair, complete and objective is to obtain advance review and comments by responsible auditee officials [in the organization being audited] and others, as may be appropriate. Including the views of responsible officials produces a report that shows not only what was found and what the auditors think about it but also what the responsible persons think about it and what they plan to do about it." "The auditor should normally request that the responsible officials' views on significant findings, conclusions, and recommendations be submitted in writing. When, in these cases, written comments are not obtained, oral comments should be requested." "Advance comments should be objectively evaluated and recognized, as appropriate, in the report. Advance comments, such as a promise or plan for corrective action, should be noted but should not be accepted as justification for dropping a significant finding or a related recommendation." "When the comments oppose the report's findings, conclusions, or recommendations, and are not, in the auditors' opinion, valid, the auditors may choose to state their reasons for rejecting them. Conversely, the auditors should modify their report if they find the comments valid. "51 Agency comment on GAO reports serves several purposes. Obtaining agency comments significantly reduces the potential for factual errors in a GAO report or misunderstandings about the research and recommendations. It reveals language or phrases that would convey unintended meanings or raise sensitivities in the agency or broader audience that GAO may not anticipate. The opportunity for agency comment also increases the organization's awareness of and cooperative attitude toward the GAO work, and thus significantly increases the possibility for successful implementation of GAO's recommendations and for effective improvements in that organization. For GAO, the comment process offers additional advantages: to reveal more clearly in advance political values and policy differences related to a particular report, to help define the costs of and obstacles to implementing the recommendations, and appropriate boundaries of GAO recommendations, and to signal areas requiring additional fact checking. The major arguments presented by congressional and GAO staffs against soliciting agency comments are to avoid further delay, to avoid pressure on GAO to change its findings, and to allow the requester to control the release and hearing process. Nevertheless, GAO's failure to obtain comments from the organization under study damages GAO's credibility, reduces the accuracy and thoroughness of its work, and harms its relationships with executive agencies. In fact, the absence of comments in many cases merely postpones debate; GAO reports become more controversial, and GAO becomes more vulnerable to criticism than it might be with reasonable consultation with the agency before the report is released. Routinely obtaining agency comments on GAO reports need not lengthen the total time GAO takes to complete a job; by adjusting internal work processes as the panel recommends and making use of the information and fact-checking the agency provides, GAO will be able to reduce the time it devotes to sequential internal reviews, offsetting the time spent on obtaining agency comments. To contribute to the fact-gathering and cooperation in the course of an audit or evaluation, GAO should consider giving agencies an earlier draft of its findings and conclusions, while continuing to review the draft internally. It should organize the project team to respond quickly and constructively to agency comments. 52 51" Yellow book," pp. 95-96. Private audit firms also obtain comments from the organizations they audit or evaluate. 52GAO should make exceptions from these procedures for reports involving criminal investigations or audits that involve individual company's business contracts or potential legal action. GAO should also continue to limit the review and distribution of reports on classified national security issues. RECOMMENDATION: GAO should seek agency comments on its reports with a 30-day limit for Follow-up and Performance Indicators After GAO reports are issued, the law requires agencies to respond to the findings and recommendations within 60 days (though agencies routinely respond late or not at all). GAO has been surprised by the content of some responses and the strength of agency disagreement, reflecting weakness in GAO's prior communications. In some cases, the misunderstandings or disagreements between GAO and agencies persist over several years and several reports. Breakdowns of this kind in coordination and communications with agencies are likely to block implementation of GAO recommendations. This is another reason for constructive and cooperative relationships and communications with the agency subject to study in advance of release of a GAO report. This point has been demonstrated in GAO's general management reviews, which have involved a high degree of cooperation and consultation with executive agencies (see pp. 53-54). In following up on its completed reports, GAO uses a variety of indicators to assess its performance and the effectiveness of its work. One of the most visible measures is the percentage of recommendations that have been implemented. GAO tracks "open recommendations" -- i.e., those that are not yet reported to be implemented -- and reports on them twice a year, with statistics compiled to cover what has been done in the most recent four- and five-year periods. For example, GAO reported that from fiscal years 1989 to 1992, it made 6,140 recommendations. By the end of 1992, the legislative branch had implemented about 55 percent of the recommendations for congressional action, while the executive branch had implemented 75 percent of the recommendations for executive branch action. The document does not report on the specific elements, costs, or results or impacts of implementation -- i.e., whether government performance improved and the public benefited. The many GAO reports that do not have recommendations but simply produce facts and "findings" are not included in the report on open recommendations. In addition to tracking recommendations, GAO's staffs also file accomplishment reports. These provide information on both dollar savings and non-quantifiable impacts that can be associated with their recommendations. The 294 accomplishments that GAO reported in fiscal year 1992 included $36.2 billion in savings, of which $30.16 billion came from defense-related issue areas. Of that, $14.7 billion were attributable to rescission from the "Desert Storm" account appropriated to pay for the Persian Gulf War. GAO had testified that a special $15 billion appropriation requested by the Defense Department might not be needed, and that it should be kept in a special restricted account "See 31 U.S.C. § 720; OMB and agencies also issue directives to implement the requirement. |