How Do You Measure Program Results?

For more than five years, the Bush administration has focused a good portion of its rhetoric on performance, accountability and results. To that end, in 2001, the Office of Management and Budget (OMB) began to develop a mechanism called the Program Assessment Rating Tool (PART) to help budget examiners and federal managers measure the effectiveness of government programs. The PART has been changing and evolving since its inception and recently received a much more prominent place in the release of the president's fiscal year 2006 (FY 06) budget. Page four of the president's overview on the budget states that "the Program Assessment Rating Tool (PART) measures the success of programs in meeting goals and identifies which are achieving their intended results and which are not." The budget claims the PART helps the administration "to reward only those [programs] that succeed." The president's FY 06 budget states the PART was developed to help consistently evaluate a program's purpose, design, planning, management, results and accountability to determine its overall effectiveness. The PART has been used to review 60 percent of federal programs over the last three years and all programs will have been reviewed once by 2007. The PART consists of six questionnaires designed for different government activities -- competitive grant programs, block/formula grant programs, regulatory-based programs, capital assets and service acquisition programs, credit programs, research and development programs, and direct federal programs. Essentially, it consists of yes and no questions that are used to evaluate the program, although in the results section of the tool there are some additional gradations. The surveys of programs are then submitted to OMB and a resulting rating (ranging from effective to ineffective) is determined by budget managers. In his recent efforts to further promote a "good-government" approach, the president often referred to a list of 154 programs slated for deep cuts or elimination in his FY 06 budget because those programs were "not getting results." OMB Watch has analyzed this list and other sections of the FY 06 budget and compared program funding requests to the ratings received under the PART. This analysis has yielded some interesting and puzzling results. Out of the list of 154 programs to be cut or eliminated, supposedly for lack of results, more than two-thirds have never even been reviewed by the PART. It is unclear what kinds of determinations, if any, the president used to identify these failing programs when the White House budget staff has yet to assess them. Of all the programs on that list that have been reviewed, nearly 20 percent of programs receiving an "effective" or "moderately effective" PART score, the two highest ratings, were eliminated. Further, 46 percent of programs receiving the middle rating of "adequate" were eliminated. A quick review of programs rated under PART since its inception finds no logical or consistent connections with budget requests. Of the 85 programs receiving a top PART score this year, the president proposed cutting the budgets of more than 38 percent, including a land management program run by the Tennessee Valley Authority and the National Center for Education Statistics. Even stranger, some programs receiving the lowest score were not cut. For instance, the Substance Abuse Prevention and Treatment Block Grant, a program that provides grants to states to address addiction problems, was given the lowest possible rating of "ineffective" but received no reduction in funding. Moreover, the Earned Income Tax Credit Compliance Program - which targets poor people who have claimed the EITC and double-checks their eligibility for the credit - was rated ineffective, yet it received a funding increase. There appears to be no logical or consistent pattern to be found in reviewing program funding requests and PART score results. However, this is not the only illogical aspect of the PART. Another puzzling situation is how the PART relates to and is integrated with the Government Performance and Results Act (GPRA) of 1993. GPRA, which was fully implemented in 1997, set out to establish a system for measuring each agencies performance - both on a whole and for specific programs - that could be tied to the congressional appropriations process. It requires agencies and departments to develop three plans; a five year strategic plan, an annual performance review plan, and then a performance report, which is submitted to Congress. The PART and GPRA appear to be redundant functions in the federal government; an ironic twist as each was meant to promote accountability and efficiency in government. A January 2004 Government Accountability Office report detailed the use of the PART and its relationship to GPRA. The GAO found the PART was a parallel and competing approach with GPRA's Performance Management Framework and expressed concerns the emphasis on the PART would shift agency focus and come to drive the strategic planning processes in the federal government. The report concludes, "By using the PART process to review and sometimes replace GPRA goals and measures, OMB is substituting its judgment for a wide range of stakeholder interests." The relationship between the PART and GPRA is not clear and is often confusing to program officials and agency managers and undermines the efforts of both to promote efficiency and accountability. OMB Watch's current analyses of the PART have produced more questions than answers about its value and purpose. It is unclear how the PART scores impact budgeting decisions within OMB as there are no consistent patterns to follow. It is hard to determine whether the PART is measuring programs accurately, consistently and in a value-neutral way. Even if it achieves these, there has been little attention paid to the question of whether the PART is measuring the right kinds of outcomes. While a lot of emphasis is placed on efficiency and cost-savings measures, the PART does not give any extra credit for equity. OMB Watch will continue to monitor, analyze, and investigate the PART and its effects on budgetary processes, agency planning, and congressional action. Promoting accountability and efficiency are certainly good goals to achieve, but it is important to achieve them in a flexible, unbiased and transparent way involving multiple actors and stakeholders.
back to Blog