""
Healthy Marriage and Relationship Education Program Evaluation Toolkit
Introduction
The Administration for Children and Families (ACF) provides grants to fund healthy marriage and relationship education (HMRE) programs to strengthen and improve the quality of relationships by offering a range of services from relationship education for youth to programming and services adult couples and individuals. ACF also provides funds for grant recipients to evaluate their programs. ACF is committed to ensuring these program evaluations provide information that is useful and meaningful for HMRE programs, ACF, and the field at large. To accomplish this, ACF provides standards and guidance to help HMRE program evaluators develop and implement rigorous evaluations that produce quality evidence.
Purpose
Mathematica developed this toolkit to help HMRE evaluators understand key program evaluation concepts, common evaluation challenges, and strategies to prevent or overcome these challenges. The toolkit consists of eight briefs. The briefs are standalone documents that can be read in any order. It is important to note that briefs provide general information and tips and are not intended to be comprehensive guide to program evaluation or the topic. The briefs use examples that are tailored to the populations served by HMRE programs—namely, youth, adult individuals, and adult couples.
Key findings and highlights
The Healthy Marriage and Relationship Education Program Evaluation Toolkit link above contains all of the materials in one resource. The individual briefs are available for download below. The briefs provide information on the following:
Enhancing the informed consent process and enrolling participants in a program evaluation. (PDF) This brief describes the importance of obtaining informed consent from participants and contains five tips that HMRE program staff and evaluators can use to enhance procedures for obtaining informed consent and enrolling participants in evaluations.
Improving survey response rates. (PDF) This brief describes the importance of maximizing the number of participants that respond to follow-up surveys (called response rates) and includes six tips that HMRE evaluators can use to improve their response rates.
Monitoring program dosage. (PDF) This brief introduces the concept of dosage (for example, hours of curriculum received) and its relevance to evaluations. The brief contains four tips that HMRE program staff and evaluators can use to collect information about dosage and address program retention challenges that might affect dosage.
Understanding and mitigating attrition. (PDF) This brief describes: (1) what attrition is and why it can be a problem; (2) how to calculate and assess it; and (3) steps evaluators can take to mitigate it.
Using mixed methods. (PDF) This brief explains the rationale for mixed methods research in HMRE evaluations, followed by a description of three common mixed methods designs. The brief closes with questions HMRE evaluators can consider when choosing a mixed methods approach.
Cleaning and preparing data for analysis. (PDF) This brief describes how errors can occur in the data HMRE evaluators collect and provides four tips for how to avoid them through data cleaning and preparation.
Creating equivalent research groups. (PDF) This brief contains information on: (1) various research designs that attempt to create groups that are equal in terms of their baseline characteristics (also called baseline equivalence); (2) how to regularly monitor equivalence throughout the evaluation; and (3) based on attrition or design, how to establish baseline equivalence for your final analytic sample before conducting analysis.
Powering an evaluation to detect effects. (PDF) This brief describes the rationale behind calculating statistical power before the evaluation begins, then recommends five steps to help evaluators conduct a power analysis and calculate effect sizes they can use throughout an evaluation.
Methods
Topics in this toolkit are intended to apply to all HMRE local evaluators conducting descriptive or impact evaluations of their funded programs. To develop the briefs, Mathematica consulted relevant evaluation literature and used lessons learned from working directly with evaluators to provide technical assistance.
Citation
Friend, Daniel, Rebecca Piatt, Avery Hennigar, Armando Yañez, Brandon Hollie, Sonia Alves, and Angela Valdovinos D’Angelo. (2024). “Healthy Marriage and Relationship Education Program Evaluation Toolkit.” OPRE Report #2024-138. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.