The National Academy of Sciences (NAS) released a report entitled "Evaluating Research Efficiency in the U.S. Environmental Protection Agency" yesterday that reviewed the way PART evaluates federal research and development programs. This review was requested by the Environmental Protection Agency (EPA) in 2006 in order to assist the agency in "developing better assessment tools to comply with PART, with emphasis on efficiency," according to the preface to the report. I don't know, but I suspect, EPA requested this study because they are frustrated with the poor ratings and inflexibility of the PART for EPA research and development programs and tired of feeling like the ugly duckling of the federal government, at least in OMB's eyes.
Turns out, the NAS study draws many of the same conclusions we have promoted about the PART, particularly its inability to correctly evaluate and capture the work of R&D programs. For instance, NAS finds that measuring research programs based on outcomes (i.e. does research on health policy make people healthier) is neither achievable nor valid. It further finds that efficiency in research should only be one part of evaluating the quality, relevance, and effectiveness of research programs.
These conclusions lead NAS to make three excellent recommendations for how the federal government should evaluate research.