Introduction
Research Questions
- The six-year analyses explored whether educational impacts on longer-term credentials, first observed in the three-year follow-up, translated into earnings impacts, and whether three-year earnings impacts for one program persisted through six years.
This report summarizes six-year impact findings from the Pathways for Advancing Careers and Education (PACE) project and the first round of the Health Profession Opportunity Grants Program (HPOG 1.0 Impact Study). These two large-scale projects evaluated education and training programs for Temporary Assistance for Needy Families (TANF) recipients and other adults with low incomes. The nine PACE programs and 42 HPOG programs represent a range of strategies within the career pathways framework that aim to enroll participants on an initial pathway step and, through a range of academic and non-academic supports, help them earn a credential and advance to a subsequent step and credential. Over time, participants will earn credentials associated with jobs that pay well and are in high demand. PACE and HPOG 1.0 are among the earliest evaluations to use a career pathways framework to evaluate program effectiveness.
The nine PACE-specific evaluations (four of which were funded wholly or in part by HPOG) and the HPOG 1.0 Impact Study (which pooled across 42 HPOG-funded programs, including the four in PACE) used experimental designs to assess impacts on postsecondary training, earnings and employment, and other life outcomes. At the three-year follow-up, most programs had increased treatment group members’ educational progress, mostly toward short-term credentials such as Certified Nursing Assistant (CNA) certificates. Five programs put a relatively large emphasis on earning longer-term credentials associated with higher-paying jobs, such as Licensed Vocational Nurse. Pre-specified confirmatory analyses in this report indicate that after six years, two of those programs increased receipt of longer-term credentials. For all but one program, educational impacts did not translate into earnings impacts.
Purpose
This report presents a concise summary of the six-year impact findings on educational progress and labor market outcomes from across the PACE project’s nine program-level evaluations and the HPOG 1.0 Impact Study.
Key Findings and Highlights
- Two programs had confirmatory impacts on longer-term credential receipt.
Of the five evaluations that specified a confirmatory educational outcome at the six-year follow up, two found increases in longer-term credential receipt: Carreras en Salud (3.8 percentage points); and Valley Initiative for Development and Advancement (12.0 percentage points).
- One program, Year Up, increased earnings at the six-year mark.
Year Up’s impact on earnings, observed at three years, continued through six years. The program had a $1,895 impact on quarterly earnings six years after random assignment (Q23-Q24), among the largest impacts reported to date from randomized evaluations of training programs for adults with low incomes. None of the other PACE programs, nor HPOG 1.0 programs, had an impact on earnings at six years.
- The lack of earnings impacts for most programs may stem from multiple factors: that many programs had small impacts on credential receipt, the types of credentials earned are typically associated with low wages, and programs have limited connections to employers.
For a program to have an overall impact on earnings, it must have a sufficiently large impact on the number or type of credentials earned, those credentials must have sufficient labor market value to result in meaningful wage increases, and credential holders must find employment. If the program falls short in any area—that is, if treatment group members do not complete training at a sufficiently higher rate than the control group, if the impacts are primarily for low-level credentials with low economic returns, or completers do not find employment—then the program’s impact on overall earnings will be small. Most programs in the PACE and HPOG 1.0 evaluations fell short in one or more of these areas.
Methods
Both PACE and HPOG used an experimental design in which program applicants were assigned at random to a treatment group that could access the program or to a control group that could not access the program but could access other programs in the community. Such a design ensures that any estimated differences between the treatment and control groups (i.e., impacts) can be attributed to program access rather than to unmeasured differences between eligible study sample members with access (the treatment group) versus without access (the control group). Each of the nine PACE programs was evaluated separately. The HPOG 1.0 Impact Study pooled 42 programs across 23 HPOG 1.0 grantees.
The impact estimates for the PACE evaluations are based on samples ranging from about 500 to about 2,500 study participants randomly assigned between November 2011 and December 2014. Impact estimates for HPOG are based on a sample of about 13,800 study participants randomly assigned between March 2013 and November 2014. Four of the nine grantee programs evaluated in the PACE project were included among the 42 programs evaluated in the HPOG 1.0 Impact Study. Therefore, though PACE reported program-specific findings for these four programs, they also contributed to the pooled HPOG 1.0 findings. For the six-year analyses, PACE and HPOG 1.0 evaluations used quarterly wage data from the federal National Directory of New Hires, data from the National Student Clearinghouse, and, for four PACE programs and a subset of HPOG participants, data from a participant follow-up survey of study members conducted at about the six-year follow-up point.
Citation
Juras, Randall, Karen Gardiner, Laura Peck, and Larry Buron. (2022). Summary and Insights from the Long-Term Follow-Up of Ten PACE and HPOG 1.0 Job Training Evaluations: Six-Year Cross-Site Report. OPRE Report 2022-239 Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.