Strengthening Two-Generation Initiatives That Support Child Development and Improve Family Economic Security: Insights from the Next Steps for Rigorous Research on Two-Generation Approaches Project

ACF sponsored the NS2G project to further advance understanding of contemporary two-generation initiatives that support child development and family economic security. NS2G had three objectives:

1. To conduct formative research to better understand initiative implementation, strengthen promising initiatives, and prepare for evaluations of effectiveness
2. To build the capacity of initiatives and researchers to conduct rigorous and meaningful evaluations of two-generation initiatives
3. To address measurement issues to promote learning across evaluations and a better understanding of relevant processes and outcomes of two-generation initiatives

Activities to address these objectives included partnering with four sites on formative evaluations, facilitating a learning community of 10 two-generation initiatives (including the four initiatives participating in formative evaluations), and developing a measure of mutually reinforcing two-generation partnerships. This report describes the experiences and activities of two-generation initiatives participating in NS2G and shares findings, lessons, and common themes from their participation in the formative evaluations and learning community. A separate brief, “The Two-Generation Mutual Reinforcement Measurement Tool: Development and Pilot Study Findings” (Conroy et al. 2023) describes the process for developing the measure, findings from the pilot testing, and next steps for continuing to test and refine the measure. 

The initiatives participating in the formative evaluations aimed to improve the intentional alignment and coordination of their services. They developed and tested improvement strategies to streamline processes, such as intake and referrals, and to build staff knowledge and capacity, such as by developing training and examining staff members’ use of time. The formative evaluation work also promoted communication with partners. Two initiatives’ improvement strategies entailed exploring opportunities to strengthen communication with partners, including data sharing.

The initiatives generated insights about how formative evaluation helped strengthen their initiatives:

• Creating a two-generation logic model was a useful foundation for program improvement.
• Carefully documenting the opportunity for change and the improvement strategy helped initiative leaders communicate clearly with staff.
• Two-generation initiatives lengthened iterative improvement cycles to learn more about their improvement strategies or adjusted their planned approach as needed.
• The rapid cycle learning approach was motivating.
• The strategies initiatives tested intentionally integrated supports for staff.
• Regular, structured communication helped service providers coordinate efforts.

The initiatives participating in the learning community reported that they developed logic models and data systems, and they began work to identify and develop program improvement strategies. Four reported working on their two-generation logic models. All reported strengthening their two-generation data systems, with two reporting they adopted a new data system that supported family-level analyses of services and outcomes for primary caregivers and their children. Three reported examining data to assess alignment with the logic model and considering ways to improve the alignment. Initiative staff who participated in the learning community indicated:

• They appreciated the opportunity to come together in a like-minded community.
• They reported that their initiatives’ use of formative evaluation tools and activities was limited.
• The experiences of formative evaluation sites supported learning among the enrichment sites.

Overall, participants in the learning community expressed high levels of motivation and interest in the topics discussed. Future support would ideally combine opportunities for peer learning with more intensive support to help participating initiatives make more progress.

NS2G included three primary activities to continue building the evidence base on two-generation service delivery: (1) formative evaluations and technical assistance (TA) with four initiatives; (2) a learning community designed to build the evaluation capacity of those four initiatives and six others; and (3) development of a measure of two-generation initiative functioning, which involved three of the four initiatives that participated in formative evaluations.

For the formative evaluations, the NS2G project used the Learn, Innovate, Improve (LI²) framework to guide the initiatives. Grounded in implementation science, LI² is a framework for program improvement that helps practitioners unpack program challenges, develop evidence-informed solutions, and use analytic methods to gather data to assess the success of a solution (Derr 2022). The formative evaluation work provided initiative staff with an opportunity (and a requirement) to step back from their day-to-day responsibilities, examine how initiatives’ services were coordinated to lead to intended outcomes, and explore challenges and opportunities in strengthening service delivery. During the formative evaluations, a small group of initiative staff—typically two to three people—met for an hour each month with NS2G TA providers.

As noted above, NS2G included a learning community of 10 two-generation initiatives, which included the four initiatives participating in the formative evaluations and six others -- called “enrichment sites” -- that participated in only the learning community. Participating initiatives met virtually five times between October 2021 and January 2023, with meetings structured to achieve these goals. To assess initiatives’ progress during the learning community, the NS2G team reviewed notes from the facilitated discussions and surveyed participants after each meeting and before the fourth meeting, in October 2022. Lessons are drawn from enrichment site staff members’ reflections in the final virtual meeting in January 2023 and informed by meeting notes and survey responses.

The activities and experiences of NS2G initiatives suggest additional program and research development is needed to continue building evaluation capacity. Initiatives in the formative evaluations reported obstacles to strengthening services and improving readiness for summative evaluation, including entering and using program data and ongoing staff capacity limitations and turnover. At the end of NS2G TA support, most initiatives still needed to build internal evaluation and program improvement capacity or establish external partnerships to help with this work. Many of the initiatives participating in the NS2G learning community engaged in key developmental activities, such as documenting and refining a logic model and implementing a two-generation data system that could link caregiver and child records. As these systems are implemented and initiatives begin to use data to better understand how their services are working, initiatives indicated they would continue to identify additional ways to improve and expand. Drawing on findings from NS2G, the report suggests opportunities for future research into program development:

• Continue formative evaluation work with two-generation initiatives, including defining and exploring the core components and principles that comprise two-generation services and documenting categories of models of two-generation service delivery.
• Continue measure development to build understanding of key two-generation functions and processes.

Where possible, these activities should take a participatory approach. Practitioners and families should be involved from the outset, shaping research questions, sharing experiences and perceptions of two-generation services, and identifying important outcomes and family processes to explore.

Citation

Baumgartner, S., C. Ross, E. Sama-Miller, N. Fung, K. Conroy, D. Aharpour, A. Bauer, and A. Carrillo-Perez (2023). “Strengthening Two-Generation Initiatives That Support Child Development and Improve Family Economic Security: Insights from the Next Steps for Rigorous Research on Two-Generation Approaches Project. OPRE Report #2023-207. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

References

Chase-Lansdale, P.L., and J. Brooks-Gunn. “Two-Generation Programs in the Twenty-First Century.” Future of Children, vol. 24, no. 1, 2014, pp. 13—39.

Conroy, K., S. Brunskill, and A. Carrillo-Perez. “The Two-Generation Mutual Reinforcement Measure and Tool: Development and Pilot Study Findings.” OPRE Report #2023-149. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2023.

Derr, M. “Learn, Innovate, Improve: A Practice Guide for Enhancing Programs and Improving Lives.” Washington, DC: Mathematica, 2022.

Holzwart, R., H. Wagner, and M. Worden. “Exploring Core Components Research in Social Services Settings.” OPRE Report #2021-52. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2021

Hsueh, J., and M.E. Farrell. “Enhanced Early Head Start with Employment Services: 42-Month Impacts from the Kansas and Missouri Sites of the Enhanced Services for the Hard-to-Employ Demonstration and Research Project.” OPRE Report #2012-05. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2012.

Rossi, P.H., M.W. Lipsey, and H.E. Freeman. Evaluation: A Systematic Approach. Thousand Oaks, CA: Sage Publications, 2003.

Sama-Miller, E., C. Ross, T.E. Sommer, S. Baumgartner, L. Roberts, and P.L. Chase-Lansdale. “Exploration of Integrated Approaches to Supporting Child Development and Improving Family Economic Security.” OPRE Report #2017-84. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2017.

Smith, C., T.J. Devaney, T. Akiva, and S.A. Sugar. “Quality and Accountability in the Out-of-School Time Sector.” New Directions for Youth Development, vol. 2009, no. 121, 2009, pp. 109—127.

St. Pierre, R.G., J.I. Layzer, B.D. Goodson, and L.S. Bernstein. “National Impact Evaluation of the Comprehensive Child Development Program: Final Report.” Cambridge, MA: Abt Associates Inc., 1997.