Background on Data Quality Guidelines
by Guest Blogger, 5/28/2002
Most federal agencies recently issued draft guidelines on data quality. These guidelines were done within the parameters of implementing guidelines laid out by the Office of Management and Budget (OMB), which in turn were required by an appropriations rider -- both of which are summarized below. The Data Quality Act OMB's guidelines were required by an appropriations rider, sometimes referred to as the "Data Quality Act," contained in Sec. 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554; H.R. 5658). There were no hearings on the rider -- which was added at the last second by Rep. Jo Ann Emerson (R-MO), who serves on the Appropriations Committee -- and no debate on the floor, leaving little to judge congressional intent besides the legislative language. Specifically, the law directed OMB to issue, by Sept. 30, 2001, "policy and procedural guidance to Federal agencies" subject to the Paperwork Reduction Act (44 U.S.C chapter 35) requiring that they:
- Issue their own data quality guidelines, within one year of OMB's implementing guidelines, "ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated";
- Establish "administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated by the agency that does not comply with the guidelines"; and
- Report periodically to OMB once the guidelines are put in practice detailing "the number and nature" of data quality complaints received by the agency, as well as "how such complaints were handled."
- Adopt specific standards for data quality, consistent with OMB definitions of "objectivity, utility, and integrity" -- which given the expansive definition of "objectivity" is enormously crucial, as discussed further below.
- Develop a process for reviewing information for quality before it is disseminated (applicable to information disseminated on or after Oct. 1, 2002).
- Establish administrative mechanisms to allow for challenges from "affected persons" (which OMB leaves to the agencies to define) to seek and obtain "timely correction of information maintained and disseminated by the agency that does not comply with OMB or agency guidelines." According to OMB, this applies to any information disseminated by the agency on or after Oct. 1, 2002, regardless of when it was first disseminated. However, in their draft guidelines, several agencies have placed time limits on data quality challenges; the State Dept., for instance, provides that a challenge must be brought within 60 days after the information is first disseminated. Agencies must respond to a challenge "in a manner appropriate to the nature and extent of the complaint," including through personal contacts, form letters, press releases or mass mailings that "correct a widely disseminated error or address a frequently raised complaint."
- Implement an appeals process, which the statute does not speak to, allowing those who disagree with an agency's verdict on a data-quality challenge to "file for reconsideration within the agency."
- Submit draft guidelines to OMB for review no later than July 1, 2002. These guidelines, which OMB will review for consistency with its own implementing guidelines, are to be presented in a report that explains how the agency will achieve OMB's data-quality objectives. Once the guidelines are put in practice, agencies are to report to OMB on an annual basis beginning on Jan. 1, 2004, on the "number and nature of complaints" and "how such complaints were resolved."
- For "original and supporting data," agencies are to consult with "relevant scientific and technical communities" to determine which data is subject to the reproducibility requirement. However, reproducibility in this context means that there is a high level of transparency about research design and methods, negating the need to replicate work before dissemination.
- For "analytic results," there must be "sufficient transparency about data and methods that an independent reanalysis could be undertaken." OMB adds that this means that "independent analysis of the original or supporting data using identical methods would generate similar analytic results, subject to an acceptable degree of imprecision or error." However, the transparency necessary to achieve this is not meant to "override other compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections." In such cases where the public does not have access to data and methods, "agencies shall apply especially rigorous robustness checks to analytic results and document what checks were undertaken."
- For analysis of "risks to human health, safety and the environment maintained or disseminated" by agencies (emphasis added), agencies must "adopt or adapt" the principles related to risk analysis in the Safe Drinking Water Act. More on this below.