Options and Opportunities to Address and Mitigate the Existing and Potential Risks, As Well As Promote Benefits, Associated with AI And Other Advanced Analytic Methods

HHS has been working to understand and promote the appropriate use of AI to advance its health and human services mission. This report synthesizes results from three internal-facing, pre-decisional analyses conducted by STPI: (1) a landscape analysis of 10 AI, machine learning (ML), and advanced analytics ethics documents published by Federal departments and agencies and released between 2018 and 2020; (2) a landscape analysis of AI activities across the Federal Government; and (3) a cross-case analysis of nine case studies with an analytical focus on understanding the existing and potential barriers, facilitators, risks, and benefits when using AI in health and human services.

Benefits of AI identified through the health and human services-related case studies and literature reviews conducted by STPI as part of case study development fit into three categories: 1) Improving mission-oriented processes and services; 2) Enabling larger scale analyses; and 3) The potential for increased personalization and targeting.

Seven categories of challenges were identified from the case studies and literature reviewed: 1) User confidence and trust; 2) Model performance; 3) Maintaining privacy; 4) Bias; 5) Data and dataset quality; 6) Transparency and explainability; and 7) Capacity

In addition, STPI identified several gaps in prior federal AI guidance, many of which have already been addressed in HHS documents, like the HHS Trustworthy AI Playbook  (PDF).

STPI’s analyses suggest both a set of general considerations for HHS and a set of more targeted considerations that could be incorporated into future HHS guidance regarding responsible AI.

STPI’s analyses also identified five broader considerations for increasing HHS’s capacity to catalyze responsible AI use: 1) Promoting organizational readiness; 2) Fostering HHS-wide AI infrastructure; 3) Promoting best practices and good governance approaches across HHS and across the Federal Government; 4) Building contracting platforms specific to responsible AI; and 5) Supporting social science research regarding responsible AI use in health and human services contexts.

This is a synthesized summary report of multiple analyses, using a mixture of landscape analyses and case studies.

There are two noteworthy limitations of this document:

1. AI is a rapidly evolving field and governance of AI activities by Federal agencies is changing quickly as well. The analyses are based primarily on documents published in 2020 and 2021. New approaches and considerations may have already emerged that could not be captured in this work.
2. The analyses of barriers, facilitators, risks, and benefits build upon a specific set of nine case studies related to particular health and services domains. There may be important challenges and opportunities associated with AI potentially relevant to health and human services domains that might have been identified were STPI to have used different starting points for our analyses, incorporated a broader set of domains, or conducted more case studies.

Zuckerman, Brian L., James M. Karabin, Rachel A. Parker, William E. J. Doane, and Sharon R. Williams (2022). Options and Opportunities to Address and Mitigate the Existing and Potential Risks, as well as Promote Benefits, Associated with AI and Other Advanced Analytic Methods, OPRE Report #2022-253, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.