Inference from probability and nonprobability samples

Abstract

In the absence of time and monetary constraints, the ideal way to measure characteristics of a finite population (e.g., residents of the United States) would be to measure every member of the population, in other words, conduct a census. However, this goal is rarely achievable, and researchers must instead rely on measurements from a subset of the population, called a sample. Broadly speaking, there are two methods for obtaining a sample from a population: probability sampling and nonprobability sampling. In a probability sample, each unit in the population has a known, positive (non-zero) probability of being selected into the sample, and randomness, controlled by the designer of the survey, is involved in the selection of which units actually get included in one particular sample. In contrast, in nonprobability sampling the probability that a unit is observed is not in the control of the survey designer; for example, when a sample is based on volunteers. In this chapter, we review the fundamentals of probability sampling, including key design features (e.g., stratification) and traditional methods for inference. We then describe nonprobability samples, which unlike probability sampling cannot neatly fit into a single framework to describe either their design or their resulting inference. We provide examples of nonprobability samples that have worked and that have failed and describe the main problems with nonprobability samples, such as selection bias and nonresponse, comparing with properties of probability samples throughout. The two main approaches to inference for nonprobability samples are described (quasi-randomization, superpopulation modeling), and diagnostics for selection bias are briefly described.