Introduction
Research Questions
- Are the domains and dimensions in the HBCC-NSAC Toolkit Provider Questionnaire reliable (that is, do they consistently assess provider practices as they are intended to)?
- Do the items in the Provider Questionnaire have the same meaning for providers with different characteristics?
- Are the domains and dimensions in the Provider Questionnaire measuring what we hypothesize they are measuring? That is, is there evidence of construct, convergent, and discriminant validity for the measures in the Provider Questionnaire?
Home-based child care (HBCC) is a vital part of the nation’s child care supply and the most common form of nonparental care in the U.S. (Datta et al. 2021). Compared with providers in child care centers, however, HBCC providers have fewer resources and supports available to them, and many face unique challenges as they provide care (Henly and Adams 2018; NCECQA 2020; Bromer et al. 2021a; Bromer et al. 2021b). As common as HBCC is, many widely used measures of HBCC quality have their roots in quality measures that were developed for center-based care. Those measures might not capture the features of care that researchers, families, and HBCC providers associate with quality in HBCC settings (Doran et al. 2022). This report describes (1) the development of the HBCC Nurturing School-Age Children (HBCC-NSAC) Toolkit and (2) the results of a validation study conducted in summer 2024 to examine the reliability and validity of measures in a questionnaire for HBCC providers.
Purpose
The HBCC-NSAC Toolkit Study aimed to develop a set of measures with HBCC providers and the specific features of HBCC quality in mind, and to provide evidence of the reliability and validity of the self-report measures in the HBCC-NSAC Toolkit Provider Questionnaire.
Key Findings and Highlights
Classical psychometric analysis and IRT Rasch analysis provided psychometric evidence supporting the reliability and validity of all the domains and dimensions.
Psychometric analysis in the validation study helped the study team refine the domains and dimensions in the Pre-Validation Provider Questionnaire. Sixteen items were dropped and 21 items were moved to a different domain or dimension. In addition, more optimal scoring for some domains was indicated. Confirmatory factor analysis of final structure and interfactor correlations of the Rasch measures validated the dimensionality and structure of the measures in the Post-Validation Provider Questionnaire.
DIF analysis provided preliminary evidence of comparable functioning (Rasch difficulties) for most items in the Post-Validation Provider Questionnaire domains and dimensions by provider type, urbanicity, and race and ethnicity. However, sample sizes for subgroups were small and overlapped in characteristics making it difficult to be definitive.
The study team observed preliminary evidence of convergent and discriminant validity of the Post-Validation Provider Questionnaire when looking at the correlations of its domains and dimensions with the validation measures. There was also evidence of convergent and discriminant validity when looking at correlations between Post-Validation Provider Questionnaire dimensions.
Most Post-Validation Provider Questionnaire domains and dimension scores were related to the number of years providers cared for children younger than age 13, suggesting that more experience is associated with completing higher quality practices in the domains and dimensions. Some of domain and dimension scores were also related to other provider characteristics, including level of education; having a certification as a Child Development Associate; completing courses in early childhood care, sociology, or psychology; and providing care on the weekend.
Ultimately, there were limitations in this validation study as the domains and dimensions were tested in a purposive sample of 148 HBCC providers—more research with a nationally representative sample of providers is needed to provide further evidence of reliability and validity. In addition, some analyses (such as the DIF comparison and FCC PQA correlations) were exploratory because of small sample sizes, which could be remedied with a larger sample size in future research.
Methods
The goals of the validation study include examining (1) the reliability of the measures, (2) the evidence of construct, concurrent, convergent, and discriminant validity of the measures, and (3) patterns of item responses across subgroups, including differences between family child care and family, friend, and neighbor providers; rural and urban providers; and providers from different racial and ethnic groups.
We examined descriptive statistics for the full sample and for subgroups of providers. We conducted psychometric analyses using both classical methods (that is, internal consistency, item-total correlations, a distribution of responses, and confirmatory factor analysis) and item response theory (IRT) Rasch methods including differential item functioning (DIF) analysis. We also assessed evidence of concurrent (convergent and discriminant) validity.
Appendix
Appendix
| File Type | File Name | File Size | The HBCC-NSAC Toolkit Study Final Report Appendix | 2,085.55 KB |
|---|
Citation
Li, Annie, Yange Xue, Natalie Reid, Jennifer Acosta, Giovanna Bautista Rodriguez, Sally Atkins-Burnett, and Ashley Kopack Klein. “The HBCC-NSAC Toolkit Final Report.” OPRE Report #2025-157. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2025.
Glossary
- Domains, dimensions, and items:
- Within the HBCC-NSAC Toolkit Provider Questionnaire, we organize single items, or questions, that reflect HBCC providers’ practices into domains and dimensions. Domains are higher order and reflect the overall intent of various items in the Provider Questionnaire (for example, how providers can support emotional development). Within domains, there are multiple dimensions that organize items into more specific categories. For example, within the Support for Emotional Development domain, specific items are organized into the Sense of Belonging or the Understand and Regulate Emotions dimensions because of the types of practices they reflect and how they are perceived to support children.
- Family child care (FCC):
- Regulated (license, certified, or registered) home-based child care.
- Family, friend, and neighbor (FFN) care:
- Home-based child care that is legally exempt from licensing or other regulations, whether paid or unpaid. FFN care includes care given by grandparents, other relatives, and non-relatives such as friends and neighbors.
- Home-based child care (HBCC):
- Non-custodial child care and early education offered in the provider’s own home or the child’s home. Although we use “HBCC” throughout the report, we recognize the role providers play caring for and educating children. Providers’ HBCC status is fluid, and individuals’ roles may change—those who care for a few children who are related to them, whether with or without pay; those who offer care as a professional occupation and a business; those who care for children over many years; and those who care for children sporadically in response to changing family needs. Many HBCC providers care for and educate children in mixed age groups. For this study, we included licensed HBCC providers (family child care) and unlicensed HBCC providers (family, friend, and neighbor care) who cared for at least one school-age child (age 5 and in kindergarten, or ages 6 through 12).
- Home-Based Child Care Nurturing School-Age Children (HBCC-NSAC) Toolkit:
- A set of measures developed with HBCC providers and the specific features of HBCC quality in mind. It includes a Provider Questionnaire and Family Questionnaire.
- HBCC-NSAC Toolkit Provider Questionnaire:
- A self-administered component of the HBCC-NSAC Toolkit that consists of newly developed measures for providers. The Provider Questionnaire is the focus of the validation study and this report. We refer to two versions of the Provider Questionnaire throughout the report: the Pre-Validation Provider Questionnaire and the Post-Validation Provider Questionnaire. The Pre-Validation Provider Questionnaire is the version that HBCC providers who participated in the validation study completed and includes five domains and 15 dimensions. The Post-Validation Provider Questionnaire is the version we refined during analyses; it shows evidence of reliability and validity and includes six domains and 17 dimensions.
- HBCC-NSAC Toolkit Study:
- A study conducted in summer 2024 to examine the reliability and validity of measures in the Provider Questionnaire and the focus of this report. We refer to this study throughout the report as “the validation study.”