Innovative Approaches – Keeping Youth Engaged Between Follow-up Data Collection Periods

By: Michael Scuello, Donna Wilkens, Lisa Tillery, and William W. Green

When pre-adolescent and adolescent youth engage in risky behaviors such as fighting in school, getting in trouble at home, and abusing drugs and alcohol, their chances of engaging in sexual activity increase (Forehand, Gound, Kokchick, Armistead, Long & Miller, 2005 ). In 2016, Future Foundation, an Atlanta-based youth development agency, formed a partnership with Fulton County Schools , the local school system, and Metis Associates , an educational research firm. The partners sought to implement and rigorously assess the efficacy of the Second Family Model in preventing teen pregnancy among South Atlanta’s most high-risk middle school students.

With support from a five-year Personal Responsibility Education Innovative Strategies (PREIS) grant awarded by the Family & Youth Services Bureau (FYSB), an office of the Administration for Children & Families within the U.S. Department of Health and Human Services, Future Foundation and its partners set out to test innovative strategies in providing abstinence and contraceptive education with the incorporation of adulthood preparation programming. The Second Family Model is an afterschool program model that includes five interconnected core components:

1. Year-round academic support, including homework assistance and direct instruction
2. Student screening and referrals delivered by licensed social workers
3. Social-emotional learning and development, using the evidence-based Lion’s Quest Skills for Action curriculum
4. Adolescent sexual health education, using the evidence-based Advocates for Youth Respect, Rights, and Responsibility curriculum
5. Parent education workshops focused on improving family functioning

The Second Family Model PREIS research study, now in its fifth and final year, uses a randomized control trial to determine the PREIS program’s impact on youth outcomes related to sexual behavior, adolescent health education, educational success, and adulthood preparation. The primary data collection tool used to assess these outcomes is the Youth Outcome Survey (YOS), which Metis Associates and Future Foundation developed based on existing validated instruments. Specifically, the YOS measures:

• Social-emotional learning
• Sexual behavior and knowledge
• Interpersonal strength
• Affective strength
• Family involvement
• School functioning

Over three academic years, 816 students were assigned to treatment (N=413) and control (N=403) conditions. Three cohorts of treatment students received the Second Family Model core services for one full year. In contrast, control students in all three cohorts did not receive any programming related to the model. For both groups, however, student progress on outcomes was measured with the Youth Outcome Surveys at four intervals: at baseline (e.g., pre-program at the time of randomization), immediately after programming ends, and in three-month and six-month follow-up periods. Overall, treatment students tended to be easier to track for data collection because they regularly received programming and were familiar with Future Foundation staff. Control students posed a far greater challenge because of the extended periods between YOS data collection activities. After the first year of the study, it became readily apparent that some proactive measures would be necessary to ensure youth engagement throughout the data collection periods for both the treatment and control group students.

In the second year of the study, Future Foundation staff planned monthly activities with treatment and control students to ensure engagement. These monthly activities were supported by both the PREIS grant and additional donations from Future Foundation donors and partners. To facilitate these engagement strategies, they hired a full-time Community Engagement Manager (CEM) in May 2018 to serve as a liaison between Future Foundation, study youth, parents of participating students, and target school administrators and staff. The CEM successfully maintained contact and established relationships with treatment and control group youth; supported outreach, recruitment, and enrollment for all students; and assisted with administering the YOS with students. Moreover, the CEM, in collaboration with other PREIS program staff, developed and carried out a comprehensive plan for using innovative approaches to promoting youth engagement in the PREIS research study. Supported by PREIS as well as other private grants, the student engagement strategies include:

• Monthly Holiday-Related Events: “10 Days of Christmas” gifts and raffles in December 2018 and “14 days of Sweetness” in February 2019 for high attending treatment students; Holiday Gift Cups for control group youth in December 2019; an Atlanta Hawks (NBA) Holiday Party trip for control group students in March 2019; April Full of Fools/Code Breaker Challenge in April 2019; and holiday gift bags filled with goodies, treats, and sweets for treatment and control group students in December 2020
• Youth-Focused Field Trips: Cascade Family Skate in October 2018 for control group youth, the Mercedes Benz Stadium Tour in November 2018 for treatment and control groups, the UniverSoul Circus trip for treatment and control group students in February 2019, Sky Zone in September 2019 for control group youth, and the Alliance of American Football League trip for treatment and control group students in January 2019
• Attendance Incentives: A Scavenger Hunt and Riddles Competition in September 2018, Halloween Escape Room in October 2018, TEAM Challenges in November 2018, Design and Create a Board Game in January 2019, March Madness competition for highest average attendance for the month of March 2019, and the End of the Year Celebration/Rock the Test event in May 2019
• Data Collection Events: An End of the Year Survey Party for Cohort 2 treatment and control group students in May 2019 and a Candytopia Event for all incoming Cohort 3 students (before randomization) in August 2019
• Summer Engagement Activities: Avengers: Infinity War movie screening for the treatment and control group in May 2018, Six Flags for treatment students in June 2018, Atlanta United for treatment and control groups in June 2018, Malibu Speed Zone for treatment and control group students in June 2018, Atlanta Braves game for treatment and control group students in June 2018, Nitro Zone and Wild Life Animal Safari for the treatment group in June 2019, Verizon Summer Camp and Atlanta Hawks (NBA) Camp for control students in June and July 2019, and Rock Ranch for treatment students in July 2019
• Tokens of Appreciation: Chick-fil-A gift cards and breakfast for control group students in September 2019, Dunkin Donuts for treatment and control group youth in November 2019, and Valentine’s Day Cupcakes for the treatment and controls in February 2019
• Future Foundation Store: Treatment students earned daily Future Foundation Bucks for positive attendance, participation, and behavior that they utilized at the end of each week to “purchase” items from the student store (January thru May)
• Entertainment-In-A-Box: Theme-based rewards and incentives in a box for treatment students who participated in virtual programming and make-up sessions in June and July 2020

PREIS program staff also regularly engaged students with phone calls, texts, and emails and provided rewards (to treatment students) for good attendance. Students also received regular tokens of appreciation — like holiday treats, gift cards, and social-media shout-outs — to show appreciation for their time.

Simultaneously, with support from Metis Associates, Future Foundation developed an effective student attendance and dosage tracking system that the PREIS implementation team used to monitor individual-level student participation and deliver targeted outreach when necessary. These two partners met weekly to review attendance and dosage data to develop data-driven recommendations for ensuring the most effective outreach strategies to secure the highest possible response rates on the different Youth Outcome Survey administrations.

With the shift to online service delivery in March 2020, due to the COVID-19 pandemic, Future Foundation altered its strategies to sustain youth and family engagement. In addition to home-based meal delivery, this includes remote retention strategies for both treatment and control group youth, including weekly check-in questions and activities (e.g., virtual jigsaw puzzle, hot or not discussions, emoji board and photo contest, and virtual Netflix movie party). To further bolster engagement among students, monthly virtual retention activities have included a game night, field day, paint party, art project day, and STEM activities.

Irrespective of the modality by which engagement strategies have been implemented, the effect on student engagement has been undeniable. Future Foundation’s response rates on post- and follow-up Youth Outcome Surveys have vastly improved since the first cohort of students revealed the necessity for further attention. For example, in Cohort 1, response rates increased from 74 percent at immediate post-test to 91 percent at 3-month follow-up and 87 percent at 6-month follow-up. Likewise, immediate post-test response rates increased by at least ten percentage points from the 74 percent observed for Cohort 1 to 87 percent and 84 percent, respectively, for Cohorts 2 and 3.