The Government Performance and Results Act (GPRA) requires each federal department to prepare three reports: 1) a strategic plan setting forth its broad mission and goals; 2) a performance plan listing its specific goals and indicators to measure the achievement of those goals, with a linkage to its budget; and, 3) a performance report on whether its goals were met and at what cost. DRAFT ANALYSIS OF THE CHILD CARE AND FAMILY SELF-SUFFICIENCY PERFORMANCE MEASURES: THE ADMINISTRATION FOR CHILDREN AND FAMILIES' PERFORMANCE PLANS UNDER THE GOVERNMENT PERFORMANCE AND RESULTS ACT I. Introduction II. Overview of ACF's FY 1999 and 2000 Performance Plans III. Comparison of Child Care and Family Self-Sufficiency Measures in ACF's FY 1999 and FY 2000 Performance Plans IV. Summary and Strategies I. INTRODUCTION The Government Performance and Results Act (GPRA) requires each federal department to prepare three reports: 1) a strategic plan setting forth its broad mission and goals; 2) a performance plan listing its specific goals and indicators to measure the achievement of those goals, with a linkage to its budget; and, 3) a performance report on whether its goals were met and at what cost. The Administration for Children and Families (ACF) is one of thirteen agencies under the Department of Health and Human Services (DHHS) that issued its own individual performance plan in conjunction with the issuance of a DHHS department-wide plan. Both the performance plans for FY 1999 and FY 2000 have now been completed and are available for review. This paper discusses the introductory narrative of each year's performance plans, and then compares two areas covered by the plans in both years-child care under the Child Care and Development Block Grant and family self-sufficiency under the Temporary Assistance to Needy Families (TANF) block grant. Within that comparison, some questions and concerns are raised that might be of interest to nonprofits and public interest organizations. The purpose is to stimulate and provide a basis for discussion of this and other performance plans by nonprofits in those areas of concern to their particular constituencies. The analysis should be especially helpful to human needs organizations concerned with block grant programs where responsibilities for administration under broad federal guidelines have been delegated to States, Tribes, local governments and private grantees. The requirement under GPRA that Federal agencies demonstrate "results" over programs administered by other entities that are not bound by GPRA requirements is a concern in the increasing devolution of formerly Federal responsibilities to States and other entities. More broadly, this analysis is meant to exemplify overall questions and issues applicable to the goals, measures and data sources within any of the Performance Plans issued in connection with requirements under the Government Performance and Results Act (GPRA). II. OVERVIEW OF ACF'S FY 1999 AND 2000 PERFORMANCE PLANS ACF's FY 2000 Performance Plan is both a revised FY 1999 Plan as well as a FY 2000 Plan. The descriptions of ACF's mission, its various programs, and the administration of its programs in conjunction with its partners and stakeholders are essentially the same in both plans. Mission Statement The mission of ACF as set forth in its Performance Plans is: ACF's mission is to provide national leadership and direction to plan, manage and coordinate the nationwide administration of comprehensive and supportive programs for vulnerable children and families. ACF oversees and finances a broad range of programs for children and families, including Native Americans, persons with developmental disabilities, refugees, and legalized aliens, to help them develop and grow toward a more independent, self-reliant life. These programs, carried out by State, county, city and Tribal governments, and public and private local agencies, are designed to promote stability, economic security, family-centered strategies, policies, and linkages among its programs, and with other Federal and State programs serving children and families. These programs assist families in financial crisis, emphasizing short-term financial assistance along with assistance in obtaining and maintaining employment. Its programs for children and youth focus on those with special problems, including children of low-income families, abused and neglected children, those in institutions or requiring adoption or foster family services, runaway youth, children with disabilities, migrant children, and Native American children. ACF promotes the development of comprehensive and integrated community and home-based service delivery where possible. ACF provides national leadership to develop and coordinate public and private programs and serves as a focal point for States in the provision of financial assistance and intervention programs, which promote and support permanence for children and family stability. ACF advises the Secretary on issues pertaining to children and families, including Native Americans, people with developmental disabilities, refugees and legalized aliens. The mission statement was developed as part of the agency's first GPRA report, its strategic plan. The strategic plan lays out the agency's mission and broad goals within its statutory requirements and must be developed in consultation with Congress and other "stakeholders," or people with a stake in the services the agency provides, including nonprofits. The General Accounting Office (GAO) explains that the purpose of the mission statement is to "…bring the agency into focus. It explains why the agency exists, tells what it does, and describes how it does it"(1). The strategic goals of the agency should flow from its mission statement, explaining the purposes of the agency's programs and the results they are intended to achieve within the overall framework of the agency's mission. The performance plan then translates those broad objectives into more specific goals, identifies the indicators that will be used to determine whether the goals are accomplished, and links the agency's performance with its budget. ACF's mission statement demonstrates that its responsibilities are largely those of leadership, direction, and coordination, rather than actual hands-on administration of programs. The mission statement also reflects the milieu surrounding welfare reform-goals of helping people "develop and grow toward a more independent, self-reliant life," providing "short-term financial assistance," along with assistance in obtaining employment, and promoting family stability. The mission statement, since it represents the actual raison d'être of the agency and is the underlying goal from which all of the agency's specific goals and objectives flow, is important in discussing specific performance measures. It is not politically neutral. It will likely change over time. Legislative changes may affect the mission of an agency. For instance, before the most recent "welfare reform," the mission of an agency like ACF might have been to alleviate the negative effects of indigence by providing income support and other services to people in poverty. That would mandate a very different set of performance goals. Since nonprofits, as "stakeholders," are required to be consulted in preparation of the strategic plan, this is an area in which nonprofits have an opening to participate in shaping an agency's mission to better reflect needs. Even though GPRA does not require consultation with stakeholders in preparation of the performance plan, the goals and objectives in the performance plan are supposed to flow from the strategic plan. Thus, the opportunity exists, at least, for stakeholders to indirectly affect the development of the performance plan. The strategic plans must be revised every three years (the next revision is due in September 2000) and may be revised at any time. Performance plans are to be done on a yearly basis. Major Programs of ACF ACF administers 22 legislative programs divided among 35 budget areas. Its performance plan combines these into thirteen major program areas. This paper discusses only two areas, Child Care and Temporary Assistance for Needy Families (TANF). The other program areas are Developmental Disabilities, Refugee Resettlement, Social Services Block Grant, Child Support, Head Start, Child Welfare, Youth Programs, Community Services Block Grant, Domestic Violence Programs, Low-Income Home Energy Assistance, and Native American Programs. Even at first glance, it is easy to see how difficult it is to separate out these areas, given the interrelationship of many of them, and then provide linkages between discrete program areas. Besides that, the budget categories don't line up at all neatly. Further complicating this task, ACF is just one agency within the Department of Health and Human Services (DHHS). DHHS has a plethora of agencies and bureaus. It manages more than 300 programs implemented by 59,000 employees under an approximately $350 billion budget. Under Secretary Shalala's administration, DHHS is managed as a largely decentralized Department(2), which may add to the difficulty of implementing GPRA as an agency-wide activity. Finally, DHHS is just one Cabinet Department within the Federal Government. GPRA requires an effort to link programs within and across agencies and departments where similar or complementing efforts exist. Agency Program Administration: Partners and Shareholders ACF's position of providing national leadership, direction, and coordination in partnership with States, Tribes, territories and local, nonprofit, and community based groups while allowing maximum flexibility is discussed at some length in the Performance Plans. ACF points out that State and local programs often differ on specific targets and outcomes based on their particular needs, even while their broad goals are similar to ACF. In terms of performance measurement, the first question that must be agreed upon is what the desired outcomes are, within the broad latitude allowed states in administration of programs. For instance, a recent study has shown that in the area of Temporary Assistance to Needy Families (TANF), local welfare offices vary considerably in conceptions of the goals of welfare reform. Some stress both welfare avoidance and work incentives, some solely stress work incentives, and some stress the correctness of eligibility decisions and effective anti-fraud programs(3). Depending on the perceived objectives, measures and the indicators necessary to determine results will vary. After those issues have been resolved, ACF must continue to "negotiate" with state and local partners to develop performance measures and targets and methods for data collection that are consistent and comparable, so that ACF can demonstrate results under GPRA. Finally, ACF has limited "teeth" to mandate goals or to require that States and other providers measure performance and collect data as ACF determines relevant and necessary. Importance of the Federal Evaluation Role After the discussion of the flexibility and latitude in program administration granted to states and other program partners, this section discusses ACF's role in national leadership and partnership. ACF identifies three areas in which this is important: 1) the development of reliable information; 2) technical assistance; and 3) the development and sharing of proven or promising methods for achieving and measuring success. ACF states that effective state decision-making is dependent upon timely and reliable information about the consequences of various program designs, public policy effects, and experiences of other States. This, then, is a role that ACF, working in concert with its partners, other Federal agencies, researchers and foundations, intends to play. This section seems to be a justification for the importance of Federal oversight and coordination even in a climate of devolution, state flexibility and power to administer programs, and mistrust of Federal solutions to local problems. Increasingly, it seems likely that the role of federal agencies in devolution will be that of information collection and organization to enable oversight over state and local programs, to support research about the actual results of federal policies, and to provide state and local service providers information and assistance on best practices. Data Issues Both plans note that baseline data are frequently not available, especially in new or changed programs. In both of our examples, this is the case. The creation of a single, integrated Child Care and Development Fund consolidated several separate child care programs, some of them embedded in broader programs, like Aid to Families with Dependent Children (AFDC). Temporary Assistance to Needy Families (TANF) entirely replaced AFDC and Job Opportunities and Basic Skill Training (JOBS), allowing States considerable flexibility to administer family assistance programs. In both Plans, ACF points out that States, Tribes and nonprofit grantees vary in their ability to collect, produce and report reliable data, making validation and verification highly complex. ACF also notes that investment in the design, development and implementation of data collection systems is costly, and those costs must be weighed against other priorities. GRPA provides for no funding to agencies to fulfill their new and extensive obligations under the Act, which may be another bar to its usefulness. In the FY 2000 Plan a long description is included, presumably in response to General Accounting Office criticism of ACF's FY 1999 Plan(4), citing various data system implementation surveys and studies that are in place. ACF notes that in a number of its major programs-TANF, Developmental Disabilities, Refugee Resettlement, Child Welfare, Child Support Enforcement, Child Care, and Low Income Home Energy Assistance programs-it must depend on State administrative systems for the collection of performance data. Other ACF programs-Head Start, Youth, CSBG, and Family Violence-rely on local community data systems. This greatly complicates the effort to obtain reliable data to show measurable outcomes that are consistent across States, Tribes, and localities. For example, the indicator for measurement of the outcomes of TANF is ACF's "High Performance Bonus" (HPB) program work measures. The HPB incentive system is a part of TANF and was designed to facilitate State work performance participation standards. It establishes financial penalties for not meeting the work participation targets and financial rewards for high performance and significant improvement. State administrative data in the form of aggregate (caseload summaries) and disaggregated (individual and family) data will be the underlying source for measurement under the HPB program, and thus the underlying success of TANF. Not surprisingly, the High Performance Bonus program rates "high performance" solely in terms of absolute job entry rate or increases over time in job entry rate, retention rate, and earnings gain. States may use quarterly unemployment insurance wage records, surveys, administrative records, or a combination of those data sources. In other words, states are also given wide latitude in the sources for the information they report. Given the newness of the program, ACF notes there is no baseline data from which to make comparisons. Performance Measurement Approach In both Plans, under a general discussion of its performance measurement approach, ACF indicates that "given the state of the art for measuring results [outcomes]" it must identify process and output measures in some instances. In other words, "outcome" measurement is, in some instances, not practicable or possible, even though the focus of GPRA is on "results," or outcomes. Outcome measures show the actual results of a program on whatever population it is supposed to serve. Output measures are measures of the actual activities of an agency, whether or not a positive effect flows from that activity. In areas where results can be quantified and where there are reliable data collection systems in place, results can be reported sooner. Where baseline measures are lacking, output measures will remain necessary until baselines can be established. In areas where outcomes are qualitative and more complex, ACF promises that effort will be devoted to achieving consensus on what outcomes are appropriate and how to design data systems that will accurately measure those outcomes. For instance, measuring the affordability of child care is more amenable to quantitative measurement than measuring the quality of child care, which is more subjective, complex, and difficult to accurately and comparatively measure. Additionally, although the ACF plan does not stress this, output measures are arguably far more important where the federal agency is not directly responsible for administration of the program. ACF does not have direct control over the results, but it can show the activities (outputs) in which it has engaged in an effort to assist state and other service providers in achieving results. Output measures documenting an agency's efforts can contribute to justifying or explaining the success or failure to achieve results when external factors may be largely responsible. Throughout this section, ACF reiterates that consensual "outcomes" are not a given, and that reaching consensus on the measurable goals of a program must be accomplished through a process of collaboration with state, local, and tribal entitles. ACF's GPRA Performance Plan addresses four major strategic goals with ten objectives. These remain unchanged between the 1999 and 2000 Plans. Strategic Goal 1: Increase economic independence and productivity for families. 1. Increase employment 2. Increase independent living 3. Increase parental responsibility 4. Increase affordable child care Strategic Goal 2: Improve healthy development, safety and well-being of children and youth. 5. Increase the quality of child care to promote childhood development 6. Improve the health status of children 7. Increase safety, permanency, and well-being of children and youth. Strategic Goal 3: Increase the health and prosperity of communities and Tribes. 8. Build healthy, safe and supportive communities and Tribes Strategic Goal 4: Build a results-oriented organization 9. Streamline ACF organizational layers 10. Improve automated date and management systems Status of Baselines and Targets This section is new in the FY 2000 Plan. ACF notes than less than one-fourth of the nearly seventy program measures do not have baselines or targets but remain "developmental" in nature. It then breaks down each area showing which have baselines and targets and which do not. For the purpose of our examples, ACF summarizes Child Care and TANF as follow:

• Child Care: five measures with no baselines or targets. (Child care measures are preliminary, pending ongoing consultation and consensus building with partners during FY '99-'00.)
• TANF: four FY 2000 measures, three of which do not have targets. (Timing for the formal development of these measures, including baselines and targets, is dependent on a continuing rule-making process with ACF, States and other partners.)

Among the thirteen program areas used in the Plan, these two are the most lacking in targets, with the exception of the Social Services Block Grant which has no specific measures at all. ACF intends, according to the accompanying narrative, to develop baselines where there are none. It states that once baselines are established, it will leave them unchanged because of its interest in "trends of improvement over time, measured against a carefully-chosen starting point, which will in some cases be the baseline established when a new program objective, statute or rule begins to have a measurable effect which can be evaluated with reliable data." This "benchmarking," or measuring progress over time is a recommended practice. On the other hand, without some initial baselines, it cannot be achieved immediately. Key Improvements in the FY 2000 Plan This new section explains how the FY 2000 Plan has been improved. It includes a comparison between targets as included in the FY 1999 Plan and the FY 2000 plan. According to the narrative, each program reviewed its measures and targets as set forth in the FY 1999 Plan and improved them, primarily in response to new or improved data and data collection systems. It is also likely that changes were made in response to GAO and congressional criticism of its FY 1999 Plan. ACF also includes discussions of its strategic approaches, considerations of external influence and coordination, resources, and data issues, including frequency of reporting. Given the lack of measures in the Social Service Block Grant program, ACF improved a description of how it SSBG supports other ACF program outcomes. The FY 2000 plan also includes an expansion of the budget cross-walk and a status update on FY 1999 measures. III. COMPARISION OF CHILD CARE AND FAMILY SELF-SUFFICIENCY MEASURES IN ACF'S FY 1999 AND FY 2000 PERFORMANCE PLANS A. CHILD CARE As noted in the previous section, both plans state the same definition of its child care program "to provide grants to States to assist low income working families who need child care that is safe, affordable and of high quality," with the addition of the word "block" before "grants" in the FY 2000 plan. Child care is included under two separate strategic goals. The first goal, objective, and approach are contained in the box below. ACF is concerned both with the affordability of child care to families who need it as well as the quality of child care, though the first measure concentrates solely on affordability (even though "quality" child care is included in the Approach. STRATEGIC GOAL 1: Increase economic independence and productivity for families. OBJECTIVE 4: Increase affordable child care. APPROACH FOR THE STRATEGIC OBJECTIVE: Increase access to affordable, quality child care for low-income, working families. In the text accompanying this section, ACF explains the legislative history. The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 established the Child Care and Development Fund requiring States to serve families through an integrated child care system funded through the Child Care and Development Block Grant. This combined child care provisions under three previous programs, Aid to Families with Dependent Children (AFDC), including Job Opportunities and Basic Skills Training (JOBS) and Transitional Child Care (TCC) and At-Risk Child Care (ARCC). This measure is tied almost entirely to the underlying goal of assisting welfare clients to achieve self-sufficiency and allowing low-income working families to maintain economic self-sufficiency by the provision of child care, so that parents can work. The "Performance Goals" section of the FY 2000 plan as it refers to childcare differs significantly from the FY 1999 Plan. Both explain that due to the recent consolidation of child care programs under PRWORA, goals and measures are preliminary and consistent baseline data is not available. In the FY 1999 plan, ACF states its intent to discuss the goals and measures in further conferences and Technical Assistance Group meetings with partners in the States, Territories, and Tribes over the coming months. In the FY 2000 plan, ACF states that it lacks the discretionary funds to do this extensively, so the FY 2000 measures remain the same preliminary measures as the FY 1999 measures. It advises that new measures await a consensus-building process that "incorporates significant opportunities for input from stakeholders including State child care administrators and representatives of relevant national organizations." This statement indicates that ACF is amenable to stakeholder input into its performance plan, even though there is no legislative mandate. It also exemplifies the difficulties of implementing GPRA in a meaningful way while lacking the resources to do so. The overall PROGRAM GOAL for this first objective is to: Increase the number of children of low income working families and families in training and education who have access to affordable childcare. As mentioned before, there is no mention of "quality" of care. Readers are referred to Objective 5 under Strategic Goal 2 (analyzed in the following section) for all discussions of quality of care. Following is the comparison between the FY 1999 and FY 2000 plans contained in the FY 2000 plan(5). 4 a FY 1999: Increase the number of children receiving subsidized child care from the 1997 baseline average of 1.25 million served per month. (developmental) FY 2000: (measure to be refined and baseline finalized in FY 1999 through consultation with partners) 4 b FY 1999: Annually increase the number of States that establish a family co-payment at 10% or less of family income. (developmental) FY 2000: (measure to be refined and baseline established in FY 1999 through consultations with partners) ACF asserts as part of its Performance Plan that the number of children receiving subsidized care is an output of the number of budget dollars invested. Its measures are then justified because the availability of child care directly supports self-sufficiency programs and is an important measure for improving family economic independence. Missing from this equation is any reference to increasing the number of children served by providing information to potential users about the availability of child care assistance programs and providers, insuring access to child care locations, easing application and renewal procedures for child care assistance, or strategies to increase availability of child care, for instance, during non-traditional work shifts. ACF released a report on October 19, 1999, subsequent to its FY 2000 Performance Plan, titled "Access to Child Care for Low-Income Working Families." This report contradicts the assertion that the number of children receiving subsidized care is a direct output of the budget dollars invested. One major factor is state discretion to set income caps lower than the federal maximum. During one month in 1998, only 1.5 million of the 9.9 million low-and moderate-income children eligible for service under the Child Care and Development Fund actually received help, just 15% of eligible children. This gap would be even larger if all of the states set eligibility limits at the maximum level under Federal law (85% of the state median income level). Then, an estimated 14.7 million children would have been eligible, of whom only 10% were actually served. The number of children served, then, is not entirely a function of the amount of money available, but also of state allocation of that money through different eligibility levels. The report also found that states were using all available Federal dollars for childcare. This report and the data it contains makes a strong argument for an increase in Federal funding of the Child Care and Development Block grant by estimating the number of children who would be eligible for child care benefits so that the actual number served can be compared. When matched against ACF's "Access" report," it is obvious that the FY 1999 objective, 4a, is very limited since it is based on a 1997 estimated baseline. According to the report, from April through September 1998, 1.53 million children per month received Child Care Development Funds for childcare. To be striving for a goal of 1.25 million children, already exceeded, is not an ambitious goal. While it is true that the report is new, and was not available when the Performance Plans were prepared, this points out the difficulties of establishing performance goals while lacking baseline information or having only outdated baseline information. Presumably, in its FY 2001 Plan, ACF will use 1.53 million children as its baseline, in order to set goals exceeding that limit based on estimates of the number of children eligible for services. Obviously, though, the new goals for FY 2001 will be based on (possibly outdated) 1998 figures. There is difficulty in establishing goals that are reasonably ambitious, within the limits of what can be expected to be achieved, and establishing such modest goals that success is guaranteed. One concern is that agencies may actually accomplish less under GPRA, because they will structure goals that can assuredly be meet in order to show that they are achieving their stated performance objectives. Another problem is that there is only so much money allocated by Congress for the Child Care and Development Block Grant Fund and a goal of serving all eligible children is impossible unless more money is allocated for the CCDF. The "Access" report is a useful tool in showing what, with additional budgetary resources, might be accomplished through ACF, which is another goal of GPRA-to provide good information for making budget decisions. Since states may opt to transfer up to 30% of TANF funds to the Children Care and Development Block Grant, and many have unspent TANF funds, the information contained in the "Access" report may encourage states to transfer funds for child care. This is one example of where compliance with GPRA has resulted in better information that can be used for effectively allocating resources. OMB Watch, in its report, "Measuring the Measurers," commented that one problem with ACF's 4a goal in the FY 1999 Plan was that rather than measure the number of children receiving subsidized care, a better measure would be increasing the number of children who are eligible for subsidized care. In other words, first you need to have an idea of the number of children who would be eligible for care, and then you try to increase the number of those children receiving that care. The "Access" report directly addresses this problem, in including estimates of how many children would be eligible under the various state rules for subsidized care. This was accomplished by contracting with the Urban Institute to produce estimates of the potential need for child care subsidies on a national and state-by-state basis(6). Again, presumably, ACF will adjust its measure to increase the percentage of children who are eligible for services that actually receive services within budgetary constraints. It will also be able to demonstrate the results that could be achieved with increased budgetary resources. The second measure, 4b, under this objective has to do with affordability of child care to families-as measured by the amount of co-payment that a family must pay. This is to assure that child care costs do not consume an unwarranted share of a family income. There is a significant variation of co-payment fee schedules across states. In the "Access" report, examples of co-payments show that a two-person family with $15,000 in income would be charged a co-payment of less than $50 per month in 7 states, $50 to $100 in 15 states, $151 to 200 in 10 states, more than $200 but less than full market price for care in 1 state, and full market price in 11 states. Given the variation, perhaps a better measure than annually increasing the number of States that establish a family co-payment at 10% or less than family income, would be a measure tailored to states' current practices. For instance, would it be more effective to concentrate on negotiating with states that require payment of full market price to establish a co-payment? Is it counterproductive to try to standardize co-payment amounts among all states, given the climate of state flexibility? Also, this is where "consultations with partners" becomes very important, since deciding on a goal will depend on successful negotiations with state partners to make changes that support that objective. Adding to that complication, one major factor affecting both affordability and quality is that in 24 states, providers are allowed to charge parents additional amounts above the co-payment if the provider's rate exceeds the state reimbursement level. These additional costs could be make child care unaffordable or limit families receiving child care assistance from choosing higher quality care. This factor is not addressed at all. Even if every state required no more than a10% of income co-payment, if the cost of childcare is high and exceeds the state's reimbursement rate, childcare could still be unaffordable. This is not addressed by any performance objective. ACF cannot compel states to do more than is required by the underlying federal regulations. The affordability measures suffer because they establish as a performance indicator the extent to which the agency is able to induce states to adopt a particular policy. ACF's authority to induce states to pursue this policy is limited. It is thus not a particularly good goal to judge the success, or lack thereof, of the agency or the Child Care Block Grant. The states may refuse to do more than the minimum legal requirement. ACF would then fail to meet its goals. Without the cooperation of states with ACF's performance objectives, it is difficult to see how results can be accomplished. The second child care goal, objective and approach follows: STRATEGIC GOAL 2: Improve healthy development, safety and well-being of children and youth. OBJECTIVE 5: Increase the quality of child care to promote childhood development. APPROACH FOR THE STRATEGIC OBJECTIVE: Provide high quality early childhood programs such as Head Start or accredited child care programs, so that early childhood experiences enhance children's development and school readiness. This entirely different section concentrates on the second part of the previous goal's intent-to improve the quality of child care. In the text accompanying this section, the purpose of childcare is not just a means to allow parents to succeed in employment and self-sufficiency, but recognition that even beyond the health and safety of children while parents are working, childcare impacts the cognitive, emotional and social development of children. The overall PROGRAM GOAL for this section is simply: The quality of child care services will improve over time. Again, it is clear from the narrative that the role of ACF is limited. It works with "State administrators, professional groups, service providers, and others to identify elements of quality and appropriate measures; to inform States, professional organizations, and parents about what constitutes quality in child care; to influence the training of child care workers and accreditation; to improve linkages with health care services and with Head Start, and to otherwise take steps to improve the quality of child care nationally." A minimum of 4 percent of the Child Care and Development Block Grant must be used by grantees to improve the quality of child care and offer other services to parents, like referrals and resources to select appropriate child care providers. ACF notes that a State's provision of training, monitoring, child care worker compensation enhancement, or other similar programs, reflect "outputs," not results (outcomes). There is no easy means of showing whether these efforts have achieved "results" by actually increasing the quality of childcare. This measure, in general, is much more difficult to quantify than the affordability measures. As noted above and recognized by ACF, affordability may impact the quality of childcare. If an outstanding child care provider charges more than the state's minimum provider reimbursement, and a parent cannot pay the difference, that child may receive a lower quality of care. Following is a comparison between FY 1999 and FY 2000 measures on this "quality" child care goal as contained in the FY 2000 plan. 5a FY 1999: Increase by 1 percent the number of child care facilities that are accredited by a nationally recognized early childhood development professional organization. FY 2000: (measure to be refined and baseline established in FY 1999 through consultations with partners) 5b FY1999: Increase the number of States that reimburse at or above the 75th percentile of market rate for high quality care. FY2000: (measure to be refined and baseline established in FY 1999 through consultation with partners) 5c FY1999: Increase the number of States that provide health services linkages with care (i.e., immunization, Medicaid outreach, and screening). FY 2000: (measure to be refined and baseline established in FY 1999 through consultations with partners) 5d FY1999: Increase the number of Head Start programs that partner with child care services to improve quality of care. FY2000: Deleted. The first measure, 5a, focuses on improving child care facilities by the process of accreditation. While this may tangentially address the quality of care, it is a very limited measure of quality. Accreditation in any field may not guarantee competence, and there are also instances of non-accredited facilities providing excellent care and services. Someone who provides day care to a limited number of children from his or her home may find it difficult to get accredited and yet the personal attention given children in that situation may be far better than the experience of a child in a large, accredited day care center. In other words, the quality of care as experienced by the children who receive that care and as perceived by their parents or custodians is a more accurate measure, even though it is much more difficult to attain. This is an example of how the GPRA requirement that measurable results be shown can result in the adoption of measures that are more geared towards ease of measurement rather than actual results-children who receive the benefits of high quality child care. The second measure, 5b, is an effort to influence affordability by encouraging states to reimburse high quality childcare at or above 75 percent of the market rate. This again is subject to state discretion. If States refuse to raise their reimbursement rates, ACF cannot meet its goal. States may reimburse less, and ACF has no authority to induce that policy change. Finally, there is no information about how many states currently reimburse at a 75 percent of market rate, or what rates are usual among states, if any, and it leaves vague the matter of what "increase" would indicate success in this measure. As the FY 1999 goal is written, if only one state increases its reimbursement rate, ACF could claim this as a success in meeting its performance goal. Certainly, the underlying program goal (the quality of child care shall improve over time) is imprecise. ACF has, perhaps, recognized these difficulties in its FY 2000 goal of refining and establishing a baseline in consultations with partners. The third measure, 5c, is again subject to State agreement. It is also unclear from this measure what "linkages" mean. Is it information about other services, direct referrals, assistance in choosing appropriate services, on-site services, etc? In order to accurately measure linkages as objectives, the indicators need to be more specific. Further, as in the previous measure, the number of linkages is not as important as the number of children actually receiving immunization, screening for disabilities, access to Medicaid and so forth. A better measure might be the actual services provided to children that are accomplished through the linkages with other programs. The final measure, 5d, of increasing the number of childcare providers that partner with Head Start programs, has been deleted from the FY 2000 Plan. This is apparently in response to criticism that Head Start, a part of DHHS, did not include a comparable measure(7. It appears that, to resolve this discrepancy, ACF simply discontinued the goal. This points to the problem of interconnections between programs within an agency, let alone those between agencies. How can all the goals be coordinated so that there is no duplication and so that one program goal is recognized by another program of which it is a part? Despite the difficulty, the solution of simply eliminating that measure is not a hopeful sign of finding a resolution to this thorny issue. ACF has noted in the text of its FY 2000 Plan that Head Start is vitally important in the effort to serve the full-time needs of low-income working families and that collaboration between Head Start providers and childcare providers remains an important goal. However, it is not included as an actual objective, with indicators for measuring the success of increasing partnerships between child care providers and Head Start programs. B. TEMPORARY ASSISTANCE TO NEEDY FAMILIES Under the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA), a comprehensive welfare reform plan, TANF replaced AFDC and JOBS, ending Federal entitlement to welfare assistance. States and territories, and Tribes if they so chose, operate their own programs, determining eligibility and benefit levels and services to be provided to needy families, within circumscribed Federal guidelines. ACF notes that the Federal government cannot regulate the conduct of states except in very limited statutory requirements, and states have a wide latitude in administering the program to "provide assistance to needy families so that children can be cared for in their own homes; to reduce dependency by promoting job preparation, work and marriage; to prevent out-of-wedlock pregnancies; and to encourage the formation and maintenance of two-parent families." The role that ACF describes for itself is to "develop reliable information on effective ways to do this, facilitate communication across States, identify best practices, promote peer technical assistance, and offer expert technical assistance." As mentioned earlier, states and local programs make very different determinations about which element(s) of this broad mandate should be emphasized, so, once again, just determining what outcomes are important becomes an issue. ACF notes that the High Performance Bonus incentive system is being used to develop data. This essentially is a system of bonus fund rewards for states that rank highest on absolute job entry rates and success in the work force, and performance improvement in these areas. No preliminary data was produced until December, 1998 (and the Performance Plan was issued in January, 1999), so targets are based on a "degree of speculation." For future years, DHHS is proposing rulemaking to address the applicable formula. Following is the TANF goal, objective and approach: STRATEGIC GOAL 1: Increase economic independence and productivity for families. OBJECTIVE 1: Increase employment. APPROACH FOR THE STRATEGIC OBJECTIVE: Increase employment and economic independence by reducing reliance on public welfare programs, providing job training and encouraging job creation. Focus on the abilities and skills of individuals, enabling them to be more self-sufficient and to pursue jobs in their communities. The goal of increasing family self-sufficiency has only the objective of increasing employment. Following are the measures and performance targets: 1a. FY 1999: All States meet the TANF work participation targets for FY 1999: (the targets and the FY 2000 "all families" increase below are statutory):

• All families: 35%
• Two parent families: 90%

FY 2000:…targets for FY 2000:

• All families: 40%
• Two parent families: 90%

1b. FY 1999: Increase from the FY 1998 baseline year, the number of adult TANF recipients who become newly employed. (See also "high impact" goal in Section V.) (developmental) FY 2000: Increase from the FY 1998 baseline year, the number of adult TANF recipients who become newly employed. (See also "high impact" goal in Section V.) (developmental) 1c. FY 1999: Increase from the FY 1998 baseline year the number of adult TANF recipients/former recipients employed in one quarter of the year who continue to be employed in the subsequent quarter. (developmental) FY 2000: Increase from the FY 1998 baseline year the number of adult TANF recipients/former recipients employed in one quarter of the year who continue to be employed in the subsequent quarter. (developmental) 1d. FY 1999: Increase from the FY 1998 baseline year the average quarterly earnings received by employed TANF recipients/former recipients over a previous quarter. (developmental) FY 2000: Increase from the FY 1998 baseline year the average quarterly earnings received by employed TANF recipients/former recipients over a previous quarter. (developmental) Except for a 5 percent (statutory) increase in work participation for all families, these measures do not change from the FY 1999 to the FY 2000 plan. As mentioned before, the sole indicator being used is the "High Performance Bonus" incentive program that awards States (or Tribes if operating their own program) for exceeding goals. The indicators of the success of TANF are limited to employment-getting a job, keeping a job (at least through two quarters), and earning higher wages. Specifically, keeping a job for six months may not be a good measure of actual long-term job stability. These measures also do not include consideration of the quality of the jobs, especially whether they pay a sustainable wage. More broadly, those concerned with issues of poverty have frequently criticized employment as the sole measure of the success of welfare reform, because it completely ignores the well being of welfare recipients. It does not measure poverty rates, the extent of homelessness, changes in nutrition, child health and infant mortality rates, or educational attainment. One response to this criticism is that PRWORA is largely based on getting people off welfare and is not weighted towards federal intervention in the alleviation of poverty except to provide assistance to get welfare recipients to work. ACF's strategic plan emphasizes this goal. Performance goals, like the underlying mission of an agency, are not value neutral. Nevertheless, within ACF's mission statement, there may be room to include other factors than job acquisition and retention. However, just as in measuring the quality of childcare, measures of the well being and actual "self-sufficiency" of families are more difficult to track, get reliable data on, and to quantify. With the issuance of the first High Performance Bonus (HPB) awards on December 4, 1999, a new proposed regulation was unveiled by the Secretary of Health and Human Services to add HPB categories to those of job placement, job success (measured by retention and earnings), biggest improvement in job placement, and biggest improvement in job success. These are family formation, enrollment in Medicaid and the Children's Health Insurance Program (CHIP), and enrollment in the Food Stamp program. This is recognition by ACF that newly-employed low-income parents need essential work supports including health care and nutritional assistance. Again, ACF is playing a "cheerleader" role, encouraging states to simplify enrollment and application procedures for Food Stamps, Medicaid, and CHIP, with the added incentive of financial rewards. This is a step in the right direction. While ACF does not have the authority to mandate eligibility levels for these benefits, the financial award carrot may be somewhat effective. The family formation measure awards states for increasing the number of children below 200 percent of poverty who reside in married-couple families. This measures seems to be predicated on the theory that children in single-parent homes are at greater risk for a myriad of ills than those in two-parent families and the best solution is marriage, rather than to address issues of poverty(8). Given the diverse objectives of TANF, which include the prevention of out-of-wedlock pregnancies and the formation and maintenance of two-parent families, this measure is supportive of TANF goals. At the same time, rewarding states for encouraging marriage, recognized as a very personal decision, is cause for concern. It will also be interesting to see both how states go about accomplishing this objective and how they will precisely measure the results. While some of the factors identified above-poverty rates, homelessness, child health and infant mortality rates, education attainment, or eligibility determination errors are covered in other ACF performance documents, no attempt has been made to cross-link those results to the success of TANF. This is an additional complicating factor in measuring results-one program may contribute to the success (or failure) of another program, yet only be measured in terms of itself. For instance, affordable, high-quality childcare is important to the success of TANF, since it allows parents to work. However, childcare is not part of the TANF measures. Since the General Accounting Office (GAO) has stressed that indicators should be few in number, agencies have a mandate to do less, not more, indicators. This increases the possibility that less reliable and valid performance measures will not be counterbalanced by other measures that are more reliable and valid. Even more practically, since states may use TANF funds for a variety of supports for needy working families and can determine the income level at which a family is considered needy, achieving results from TANF allocations should not be limited to measures of employability, especially given strong evidence that from one-half to two-thirds of those no longer receiving assistance are working in jobs where earnings are not sufficient to meet the federal poverty level(9). Rather, there should be an effort to identify continuing needs even among those recipients who are employed and use available funds to address those needs, especially given the proliferation of state TANF surpluses. Another criticism of the TANF measures is that they mitigate towards providing assistance to those welfare recipients who are most likely to succeed at employment, ignoring individuals who are harder to place. In order to show increases in employment and wage rates, it is feared that the most attention will be given to people who are employable over those people with the greatest barriers to employment-those who are disabled themselves or are caretakers for disabled children, older people, people who live in inner cities or rural areas where it is difficult to find any work or there is limited transportation to jobs(10), people who have limited education, are homeless, or have other barriers to getting a job, keeping a job, and earning higher wages. Often these are the people living in the most intractable poverty, who need the most help. But, to keep the numbers up, and show success in the program, these may be the last to get services. ACF has no separate indicators for these harder to serve populations, which leads to the charge that these performance measures are subject to helping the "cream of the crop"(11). ACF notes that "[a]ttention will also be given to removing barriers to work for welfare recipients who are victims of domestic violence, have developmental disabilities, or have serious personal or family problems which interfere with their ability to work." However, this statement cites only a few of the conditions that may make it more difficult for a person to work, and there are no corresponding performance objectives or measures. Even though, unlike the child care block grant, TANF has some built-in performance incentives for States and the ability to reward or punish States based on their achievement(12), the problem remains that ACF cannot control State administration of TANF. In the field of performance measurement, and mentioned throughout GPRA, "outcome" measures are the important indicators of success and agencies are told to limit "output" measures. This makes sense-it is not what the agency does, but what happens as a result of what it does-and supports the underlying purpose of GPRA to focus on government results. However, in block grant programs, where the agency has very limited control over the administration of programs by other entities that are not bound by GPRA requirements, measuring outputs becomes more important. Without output measures it will be difficult to gauge ACF's responsibility for any outcome changes that occur in both the childcare and TANF efforts. Further, outside influences have considerable impact, as we have seen particularly in recent efforts to get people off welfare and into employment. The fact that the economy is booming, unemployment low and there is an abundance of low-skill (but also low-pay) jobs has had considerable effect on the success of welfare reform. It is also argued that other policy changes like the expansion of the Earned Income Tax Credit has also increased the move from welfare to employment(13). Changes in demographics, and even attitudes towards family cohesion or the social acceptability of children born out of wedlock, are factors in family self-sufficiency. Without indications of agency outputs, it becomes even more difficult to attribute that success to the TANF program. While GPRA requires that agencies identify and list external factors beyond their control in their plans, and ACF follows this instruction, mere acknowledgement does not resolve the difficulty of determining cause and effect in many cases. IV. SUMMARY AND STRATEGIES While federal agencies are not required to consult with stakeholders, including nonprofits, in the preparation of their performance plans, ACF seems well aware of the benefit of such input given its reliance on state and local partners and lack of resources for outreach, especially in these initial stages of implementing the requirements of the Government Performance and Results Act. Nonprofits are in a unique position to assist agencies in establishing and refining useful goals and objectives and in developing accurate indicators for measuring progress in achieving goals. As stated before, even if nonprofits only take advantage of providing comments on agency strategic plans, the performance plans will be indirectly affected. The following are some of the questions raised in this paper that might be useful for nonprofits to consider when reviewing strategic and performance plans:

• Do the measures reflect the goals and objectives of the nonprofit and public interest communities?

Both the child care and TANF measures are tilted toward the value that employment is the primary goal. Affordable quality childcare allows parents to work. The goal of public welfare is to assist people in obtaining employment to support themselves and their families. While welfare reform mandates increasing the responsibility of parents to support their family through employment, many nonprofit and public interest organizations would argue still that the alleviation of the negative effects of poverty and the health and well-being of low-income families and children must remain visible in determining the success of welfare reform. Even within statutory requirements, indicators about well being remain important to determine whether welfare reform is making a positive difference in the lives of low-income families and children. In both the childcare and TANF provisions, the focus on the desired results is whether people get off welfare and become employed. TANF measures were limited to job entry rates, job retention, and earnings gains. Childcare measures revolve around the number of children receiving childcare and the quality of the care as measured largely by provider accreditation. This leaves out important measures of the well-being of the population being served in terms of, for instance, poverty rates, the extent of homelessness, educational attainment, infant mortality rates, family and child nutrition and health, or other measures of actual well-being. Some of these measures are also important in establishing whether the populations being served are actually receiving the benefits necessary to maintain employment, including good and affordable childcare, adequate nutrition, health care, housing, reliable transportation, and other needs of working low-income families. The addition of Food Stamp, Medicaid and CHIP access to the High Performance Bonus measures is an important addition, though perhaps sufficient to address the issue of the well-being of low-income families. Stakeholders represent different interests. Nonprofits are often in the position of representing large numbers of low-income or disadvantaged Americans who otherwise have little or no voice in the administration of government programs. Stakeholders represent different interests. Congress may have one idea of the purpose and objectives of an agency; business another; state and private grantees, yet another. Nonprofits can and should play a role in defining agency missions and goals to better reflect the needs of the people they serve, who often lack powerful means to influence government.

• Do the measures reflect values of equity and effectiveness, not just efficiency and economy?

Under the block grant system of providing services to families, a child in one state may receive more affordable childcare than a child elsewhere. A family in one state may be entitled to and given good access to services that are difficult to obtain in another. This leads to broader questions about the role of the federal government than are within the scope of this paper, but when considering issues of equity, nonprofits should be alert to equity problems that may be revealed by GPRA implementation. As pointed out previously in the discussion of TANF measures, states are allowed wide latitude in determining their own TANF and child care programs, and ACF is no longer in a position to set nation-wide goals and objectives (let alone apply regulatory mandates). At least in theory, if GPRA is successfully implemented in a meaningful way, nonprofits should have increased access to this kind of information. At least one commentator has made the point that equity is an important outcome of performance measurement, in that a "good performance measurement system will help officials demonstrate to the public and to policy makers that services are delivered fairly and this will build trust"(14). If true, this will not automatically just happen, rather nonprofits must become part of the process of GPRA implementation and evaluation. GPRA itself represents a movement in a federal regulatory and process orientation to an approach that emphasizes results. It is important that the values of equity and effectiveness often associated with process issues and regulatory requirements not become lost in this movement. The activities of agencies, while not the focus of GPRA, and the processes they use in providing services, must remain important. The "Access to Child Care for Low-Income Working Families" report by ACF addresses one issue which OMB Watch previously raised, that of the importance of determining of how many children are eligible for child care rather than just measuring the number of children receiving care. This is an important step. To determine the effectiveness of programs, the number of participants, or the increase in the number of participants, is not useful information without knowing the number of potential participants, whether we are talking about child care, Head Start, TANF, or any number of programs. Another problem with performance measurement is the temptation to focus on the most easily served populations or to limit goals to those that can be easily accomplished or documented in order to show results. For instance, the performance measures under TANF fail to adequately consider those welfare recipients who will have the most difficulty in obtaining and maintaining employment. This gives agency personnel the incentive to devote their efforts to helping those recipients least in need and avoiding the more disadvantaged populations that are most in need of help. It is important that nonprofits advocate for efforts that are inclusive and serve those people for whom measurable and positive results may be the most illusive.

• If the measures include targets, are the targets appropriately ambitious?

As mentioned before, the ACF targets for childcare for FY1999 were based on an old assessment of the number of children receiving services and were not useful. To be effective, it is important to guard against overly ambitious targets as well as targets that are designed to be easily met. Concurrently, targets need to have some specificity in order to be useful. A target of simply "increasing" the number of states that have a copayment of 10% or less of family income is not specific enough, since only one state needs to change its policies to claim success in this measure. Finally, it will remain difficult for ACF to set targets when it lacks any method of compelling states or other program partners to cooperate in changing eligibility standards where those are related to specific objectives. It is important that reasoned decisions be made over whether targets should be fixed or whether improvement should be measured over time. ACF has taken the position that once baselines are set, it will measure improvement over time measured against its baseline starting point. While this may be a good method, depending on external circumstances, it may be necessary to set targets. GPRA emphasizes the use of outcome measures-evidence of the actual results on the target population. However, given the influence of external factors and the lack of control that ACF has over its partners who are actually administering the programs, output measures will likely be necessary and valuable indicators. Nonprofits should give consideration to these questions. If an output measure is necessary and will provide some reliable information about an agency's efforts to achieve results, the agency should be encouraged to use those measures. The alternative may be the inability to show in any meaningful way that positive results were accomplished through the agency's efforts. If budget decisions are made on an agency's effectiveness in achieving results when those results were largely beyond its control, it is important that output measures are in place.

• Are the measures valid and reliable?

A primary issue is how to allow state and other program providers flexibility in administering programs while insuring that data is valid and reliable an