Introduction
Research Questions
- What is the impact of each program on indicators of progress in career pathways–relevant training, such as persistence in education and the achievement of certificates and degrees?
- What are the impacts of each program on employment and earnings?
- What are the impacts of each program on individual and family well-being?
- For programs with large study samples, to what extent do impacts vary across important subgroups within the sample?
- For programs that generated positive earnings impacts over the six-year period, do the benefits exceed the costs from the standpoint of important stakeholders?
The Pathways for Advancing Careers and Education (PACE) project is a study of nine programs that use a “career pathways” framework for increasing education, employment, and self-sufficiency among low-income individuals and families. Funded by the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services, PACE includes three rounds of analysis—at 18 months, three years, and six years after random assignment. The first round of reports covered program implementation and impacts at 18 months after random assignment. The second round of reports is currently being finalized and covers impacts and, in some sites, costs and benefits at three years. All reports are or will be published on the ACF Office of Planning, Research, and Evaluation (OPRE) website.
This Analysis Plan describes plans for the third round of reports, covering a six-year follow-up period. It is the third supplement to the Evaluation Design Report (Abt Associates 2014), which provided general plans for the PACE project. Two prior supplements (Abt Associates 2015; Judkins, Fein, and Buron 2018) described plans for the earlier reports.
Purpose
The purpose of this report is to describe a plan for answering the primary research questions using six-year outcomes. By specifying these details in advance, this document serves as a public commitment to the planned analysis.
Key Findings and Highlights
- This analysis plan identifies one or two confirmatory outcomes for each PACE program. These outcomes represent the main indicators of each program’s success in reaching its goals after six years. A common confirmatory outcome across the nine programs is average quarterly earnings over the 23rd and 24th quarters after random assignment. For four of the nine PACE programs, a second important confirmatory outcome is receiving an academic credential requiring a year or more of college.
- The report also identifies outcomes to be treated as secondary and exploratory in the analysis. Secondary outcomes represent a limited number of important outcomes targeted in the local program’s logic model. Exploratory outcomes include additional outcomes of interest that may or may not be identified in the logic model.
- The report describes the principal data sources for the six-year impact study. These data sources include a Six-Year Follow-Up Survey (conducted for four of the programs), the National Student Clearinghouse, and the National Directory of New Hires.
- The report describes analyses aimed at discerning the COVID-19 pandemic’s mediating influence on selected impact estimates.
Methods
Impact analyses use survey and administrative data to measure impacts as differences in mean outcomes between the randomly assigned treatment and control groups at each PACE site. The analyses incorporate technical adjustments for baseline imbalances, survey nonresponse, and item nonresponse. As in the earlier rounds of analysis, each PACE site will be analyzed separately with no pooling of results. We will provide separate reports for the four sites with six-year follow-up survey data. Analyses for administrative data only sites will appear in a single report covering results for all nine programs.
Citation
Fein, David, David Judkins, and Larry Buron. PACE Six-Year Follow-Up Analysis Plan. 2021. OPRE Report 2021-29. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Glossary
- ACF:
- Administration for Children and Families
- PACE:
- Pathways for Advancing Careers and Education
- TANF:
- Temporary Assistance for Needy Families