Introduction
Research Questions
- What are the promising approaches and challenges in identifying, establishing, and maintaining new and existing partnerships?
- How are data being linked and used within and across agencies to: identify families in need of child abuse and neglect prevention services; identify the specific needs of families; make informed decisions about service provision; inform continuous quality improvement; and track outcomes?
- How are grant implementation activities structured and operationalized, within and across grantees?
- What factors promote or impede implementation of the child welfare community collaborations, within and across grantees?
- To what extent are grantees planning to sustain activities beyond the current grant, and what factors do they believe will help or hinder these efforts?
The Child Welfare Community Collaborations (CWCC) initiative is designed to mobilize communities to develop and evaluate multi-system collaboratives that address local barriers and provide a continuum of services to prevent child abuse and neglect. The initiative is funded by the Children’s Bureau (CB) within the U.S. Department of Health and Human Services’ Administration for Children and Families (ACF). In 2018 and 2019, CB awarded 5-year cooperative agreements to a total of 13 states, non-profit organizations, and Native American tribal organizations (referred to here as “grantees”).
To advance the evidence around collaborative approaches to preventing child abuse and neglect, the Office of Planning, Research, and Evaluation within ACF, in collaboration with CB, contracted with Abt Associates and its partner Child Trends to conduct the Building Capacity to Evaluate Child Welfare Community Collaborations project. The project includes:
- evaluation-related technical assistance to support grantees and local evaluators and their capacity to conduct their required project-specific evaluations and
- a cross-site process evaluation of the CWCC grants to better understand how communities came together to develop and implement their CWCC approaches.
This brief describes the design and methods used for the cross-site process evaluation of the CWCC initiative and is one of a series of products the evaluation team will produce as part of the cross-site process evaluation. The brief includes a discussion of the framework and research questions that guide the evaluation, the data sources and collection methods used, and the analysis methods the evaluation team used.
In October 2024, we updated this brief to reflect the final analytic samples and response rates for the collaboration survey and site visits after the fourth wave of data collection. We edited the text consistently throughout the report to reflect these updated response rates and added footnotes to flag these changes in the places where they appeared (see Exhibits 2-4). No other substantive changes were made to the report.
Purpose
The cross-site process evaluation aimed to support the goal of contributing to the primary prevention knowledge base by systematically investigating the structure and implementation of the CWCC projects. The cross-site process evaluation is intended to explore how grantees:
- formed and maintained partnerships across their communities,
- used data to support collaboration and program activities,
- implemented their CWCC strategies, and
- developed plans to sustain their work.
This brief describes the study’s mixed-methods design, and the instruments, procedures, and approaches the evaluation team used to answer the research questions.
Key Findings and Highlights
The methods described in this brief were used to inform evaluation findings and will be used as a reference point for subsequent briefs.
Methods
The CWCC cross-site process evaluation used a mixed methods design to answer the research questions. Data sources included an online collaboration survey, program documents and grantee-produced reports and deliverables (e.g., grant mandated semi-annual progress reports), and in-person and virtual site visits, including interviews and observations. The evaluation team collected data from grantees and their partners annually (in four “waves”) in years two through five of the grant awards (after the completion of the 10-month planning period) and collected data from a total of 405 interviews and 1,720 collaboration surveys.
Overall, the evaluation team used a mixed methods approach to analyze the data. The team used a quantitative approach to analyze the collaboration survey data and created individual results reports for each grantee after each wave of data collection. The team used a qualitative approach to analyze site visit data, summarizing key findings for each research question and identifying other emergent findings.
Citation
Layzer, C., Blocklin, M., Schachtner, R., & Cook, R. (2023). Child Welfare Community Collaborations Cross-Site Process Evaluation Design and Methods. OPRE Report 2023-334, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Glossary
- ACF:
- Administration for Children and Families
- CB:
- Children’s Bureau
- CWCC:
- Child Welfare Community Collaborations
- OPRE:
- Office of Planning, Research, and Evaluation