Introduction
During the 2023 OPRE Methods Meeting, titled “Addressing Unit Missingness in Social Policy Survey Research,” Dr. Raphael Nishimura of the University of Michigan gave a presentation titled “Nonresponse Reduction and Adjustment Techniques.” In this session, Dr. Nishimura described weighting, imputation, and other analytic strategies researchers can use to minimize the impact of unit missingness—the failure to obtain survey information from an intended respondent—on survey data.
In his presentation, Dr. Nishimura described several strategies available to researchers interested in addressing unit missingness after data collection. Weighting approaches can be used to adjust for unequal selection probabilities, unknown eligibility, and nonsampling errors. Researchers can use calibration strategies such as poststratification, raking, and generalized regression to improve the efficiency of weighted estimates. Dr. Nishimura also discussed how imputation methods can be used to address item missingness, and leverage information about respondents and nonrespondents to predict responses. Imputation methods include mean value imputation, hot-deck imputation, regression imputation, and sequential regression imputation.
Purpose
This document was developed from questions posed by 2023 Methods Meeting attendees to Dr. Nishimura and serves as a reference for researchers interested in learning more about nonresponse reduction and adjustment approaches.
Questions addressed in this resource cover three topics: selecting a strategy, weighting approaches and considerations, and imputation and other techniques.
Key Findings and Highlights
Nonresponse weighting adjustment and data imputation serve different purposes in addressing missing data in surveys. Researchers typically use nonresponse weighting adjustment when dealing with unit nonresponse. Data imputation is suited for handling item nonresponse.
Some substantial nonresponse bias may exist—due to large differences between respondents and nonrespondents in survey outcomes—even if the response rate is high, or the item missing rate is low.
Nonresponse adjustment is advisable even if nonresponse is low, unless further nonresponse bias analyses (such as comparing respondents and nonrespondents on important auxiliary variables) present evidence that differences between respondents and nonrespondents may be small.
Several analytic strategies exist for addressing nonresponse and several factors, such as how much data are missing and what auxiliary information is available, can inform researchers’ decisions about which to employ.
Citation
Nishimura, R. (2024). Digging Deeper into Nonresponse Reduction and Adjustment Techniques: Frequently Asked Questions (OPRE Report 2024-165). Prepared by Westat Insight. U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation.
Glossary
- Auxiliary variables:
- information gathered from participants outside of the survey (e.g., screening or administrative data) that is used to identify differences between respondents and nonrespondents.
- Calibration adjustment:
- a weighting adjustment in which the sample distribution is matched to the population distribution on some auxiliary variables.
- Data imputation:
- an intervention researchers use to fill in missing survey responses with plausible values based on the patterns found in the rest of the data.
- Item nonresponse:
- the failure to obtain a response to a specific survey question.
- Missing at random (MAR):
- assumes that conditional on a set of auxiliary variables, missingness is unrelated to survey outcomes.
- Missing completely at random (MCAR):
- assumes that missingness is completely unrelated to survey outcomes.
- Missing not at random (MNAR):
- assumes that missingness is directly related to survey outcomes.
- Multiple imputation by chained equations:
- an imputation technique in which researchers impute missing values from multiple variables by performing a series of regression models.
- Nonresponse bias:
- the difference between the expected value of the estimate with the observed response rate and the value assuming a 100% response rate.
- Nonresponse class-based weighting adjustment:
- approach that groups respondents and nonrespondents into classes according to observed auxiliary variables to balance respondents and nonrespondents within each class.
- Nonresponse weighting adjustments:
- aim to adjust the survey estimates by attributing larger weight to respondents who are underrepresented due to nonresponse.
- Response rate:
- the proportion of estimated eligible sample members who complete the survey out of the total selected eligible sample.
- Survey attrition:
- when respondents of a longitudinal survey answer at baseline but do not participate in follow-up waves.
- Unit nonresponse:
- the failure to obtain any survey information from a sampled person.
- Weighting:
- a statistical intervention researchers use after data collection to adjust for nonresponse.