Developing Measures of the Implementation and Cost of High-Quality Early Care and Education

The goals of ICHQ are to (1) produce technically sound, systematic measures of the implementation and costs of education and care in center-based settings that serve children from birth to age 5; (2) produce implementation and cost measures to examine the variation in ECE center capacities and resources that can make a difference in quality and the experiences of children; and (3) develop a feasible and useful instrument to guide the collection, development, and reporting of the measures. The measures are intended for use in research to examine and explain variations in quality, in practice to inform quality improvement, and in policy to inform funding and technical assistance. This report details the methods for producing the draft measures of implementation and cost and prepare them for further testing.

Early testing with a purposive sample of 30 centers suggests that the ICHQ measures are capturing important information about implementation and costs and the variation that exists across centers. The measures are doing what they are intended to, and they hold promise for working together to inform pathways to quality.

The draft implementation measures have good psychometric properties and both the implementation and cost measures demonstrate validity based on patterns with observable characteristics. The report discusses key findings about the measures such as:

• The implementation scores for each of the five key center functions produced reliable scales. The items for each function measure the same thing in different ways and together they capture the essence of the function (they hold together).
• Implementation scores for each key function have distributions that can detect differences across centers. The scores for each key function have substantial variation; we are able to tell centers apart in the middle of the distribution as well as along the tails.
• Implementation scores for the key functions are related to each other, but each also provides distinct information. We found moderate, significant associations among the implementation scores of the five key functions.
• The cost measures show that the costs of staff compensation and facilities made up nearly 80 percent of total costs, on average, in line with past cost studies. Centers invest in staff compensation (salary and fringe benefits) as the largest proportion of costs (62 percent, on average). Facilities account for another 16 percent of costs, on average.
• Centers spent the largest portion of costs on the Instruction and Caregiving function. Centers in the sample directed 33 percent of costs to the Instruction and Caregiving function, on average, and 22 percent of costs to the Instructional Planning, Coordination, and Child Assessment function.
• Implementation and cost measures vary with center characteristics in ways we would expect. In our sample, centers with high QRIS ratings have higher implementation scores and costs per child care hour, on average, on each key function than centers with low QRIS ratings.
• The implementation and cost measures are related. The implementation scores and cost per child care hour for each key function are positively related, meaning the higher the implementation score, the higher the cost of the function.

A phased approach to data collection in the multi-case study provided opportunities to refine the measurement constructs and data collection tools and processes, and to develop draft measures. The multi-case study included (1) conducting semistructured interviews about the intentionality, structure, and consistency of implementation of key functions of a center that can support quality; (2) collecting cost data through Excel workbooks to assess center-level costs by key function and how resources are used across functions; and (3) administering surveys about staff time use—essential information for allocating costs to key functions, since labor is a large driver of costs in ECE centers.

The length of the data collection tools, the methods of collection, and the number of participating centers differed across the phases. The multi-case study began with an exploratory pilot study with three centers in the fall of 2015. We then recruited 15 centers to participate in a comprehensive formative phase of data collection in the fall of 2016. Based on what we found in Phase 1, we developed a set of measures to test in Phase 2. Phase 2 consisted of early testing of the new measures to ensure that they are technically sound (meaning they are reliable and the measures of implementation and cost are related) and can be used consistently across a range of ECE center-based settings.

The findings from the phased multi-case study provide preliminary evidence that the measures are capturing variations in implementation and costs among centers and are working together in ways that have the potential to inform pathways to quality. The measures are not yet fully validated, meaning they have not been tested in a large, representative sample of centers to look at associations with center quality or children’s outcomes. We are pursuing a step in this direction through a field test with a purposive sample of 80 centers in 4 states.

The ICHQ measures have the potential to support the efforts of researchers, practitioners, and policymakers to better understand pathways to achieving high quality. These insights can inform decisions about the level of resources needed and how to best use resources across functions at the center level to deliver high quality early care and education.

G. Kirby, P. Caronongan, A. Burwick, S. Monahan, D. Poznyak, T. Schulte, J. Lyskawa, and A. Kelly. (2022). Developing measures of the implementation and cost of high quality early care and education. OPRE Report 2022-04. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.