Skip to main content
Skip to main content

Replication of Recovery and Reunification Interventions for Families (R3-Impact): Impact Evaluation Design Plan

This is the impact evaluation design of the R3-Impact Study, a multi-site longitudinal evaluation of approaches to support parents involved with child welfare due to substance use.

Report- Impact Evaluation Design Plan updated

File details: PDF / 27.46 MB

79 Pages

Download

Research Questions

  1. Does PMP improve the well-being of caregivers whose children are at risk of foster care placement due to parental substance use, including (1) mental and emotional health, (2) substance use, and (3) engagement with the child welfare case, compared to currently available services?

  2. Does START reduce the likelihood of a foster care placement among families who participate in START compared to similar families who were not offered START?

Introduction

This report details the impact evaluation design of the R3-Impact Study, a multi-site longitudinal evaluation of approaches to support parents involved with child welfare due to substance use. The study focuses on the effectiveness of “recovery coaching” services for parents and families. Recovery coaching services are provided by parents with lived experience in substance use disorder and child welfare, who are trained to work one-on-one with parents to motivate and connect them to substance use disorder treatment and other services to support the recovery process and help navigate child welfare and other systems. The report describes two programs being evaluated, the aims and questions guiding the evaluation, and the methods used to answer the research questions. A separate report (Betesh et al., 2024) describes the implementation evaluation design.

The programs being evaluated are:

            Parent Mentor Program (PMP)

            Sobriety Treatment and Recovery Teams (START)

Each program incorporates recovery coaching into its overall approach in unique ways. The R3-Impact Study includes separate impact evaluations of PMP and START. The study will use an experimental design for PMP and a quasi-experimental design for START.

The PMP and START evaluations will examine program effects on child welfare outcomes related to family preservation, permanency, and child maltreatment across multiple sites. For PMP, the evaluation will also examine the effects on parent mental and emotional health, substance use, and engagement with the child welfare case.

Purpose

Over the past 20 years, parental substance use has become an increasingly prevalent reason for children to enter foster care. In 2021, 42% of children placed in foster care had parental alcohol (6%) or other drug use (36%) as the identified circumstance associated with the child’s removal—a 23 percentage point increase from the year 2000 (U.S. Department of Health and Human Services, 2022; National Center on Substance Abuse and Child Welfare, 2019). In response to the increase in such cases, local, state, and federal agencies responsible for the welfare of children and families have begun investing in recovery coaching. Recovery coaching is one promising approach to support parents’ recovery and children’s safety, help prevent foster care entry, and shorten time to permanency for children who do enter foster care. 

Although recovery coaches have been part of the array of substance use recovery support services for some time, use of the practice in the child welfare setting is relatively new. A limited body of research suggests recovery coaching has the potential to benefit families engaged in the child welfare system.

The Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services (HHS) seeks to fill gaps in knowledge and practice by providing the public with information from a rigorous examination of recovery coaching programs designed for such families. The effort originates in the 2018 Substance Use—Disorder Prevention that Promotes Opioid Recovery and Treatment for Patients and Communities (SUPPORT) Act (Pub. Law 115-271), wherein the U.S. Congress directed ACF to replicate and rigorously evaluate a recovery coaching intervention with the goal of understanding its effectiveness and informing future potential efforts to scale effective interventions. 

Key Findings and Highlights

The impact evaluation design for the R3-Impact study will provide rigorous estimates of the effectiveness of two interventions that use recovery coaching with parents involved with child welfare and whose children are at risk of foster care placement due to parental substance use. The evaluation will determine the effectiveness in improving parent well-being and recovery and reducing entry into foster care and subsequent maltreatment. The impact evaluation will estimate intervention impacts using an experimental design for PMP, and a quasi-experimental design for START.

Methods

The R3-Impact study uses experimental and quasi-experimental methods in multiple sites with a five-year follow up period. For the experimental evaluation of PMP, parents who consent to participate in the study and complete a baseline survey will be randomly assigned to either: a group offered PMP in addition to the currently available services, or to a control group offered only the currently available services. The difference in outcomes between randomly assigned groups represents the impact of PMP. 

The quasi-experimental evaluation of START will estimate its effects by combining site-specific impact estimates using meta-analytic methods. These methods combine results from separate statistical analyses to produce a single set of overall results. 

Glossary

Family preservation: The goal of keeping families safe and together and avoiding foster care placement when possible, and the services associated with this goal.

Permanency: Achieving a safe, stable home and a legal, permanent family relationship for a child in foster care.

Child maltreatment: Child abuse and neglect, which includes physical abuse, neglect, emotional abuse, and sexual abuse/exploitation. 

Recovery: A process of change through which individuals improve their health and wellness, live a self-directed life, and strive to reach their full potential. It is an individual process that can take many forms or paths.

Recovery capital: The breadth and depth of internal and external resources that are available to help initiate and sustain a person’s recovery from substance use disorder. 

Recovery coaches: Parents with lived experience in substance use disorder and as a parent in the child welfare system, who are trained to work one-on-one with other parents to motivate, provide emotional support, model that change is possible, and connect them to treatment and other services to support the recovery process. They build recovery capital and help parents navigate child welfare and other systems.  

Recovery coaching: Services provided by parents with lived experience in substance use disorder and child welfare