A scientific survey must have a representative sample.
Are the results of an online poll or a call-in vote scientific? What if millions of people participated? Does that make it more “legitimate”?
Many people are understandably suspicious of poll results. Unless you know that a survey is done scientifically, there is cause to doubt the results. Sample selection and type of questions are two cues.
When conducting a survey, how a researcher selects participants is just as important as how many participate. Scientific surveys can include every member of the group to be studied, but this approach is usually impractical and/or expensive. Instead, researchers often draw conclusions about a target group using information gathered from a small representative sample of that group. Representative samples must be selected carefully and without bias. For example, samples made up of self-selected responders, such as people who participate in a survey or poll by calling an 800 number, are almost certainly biased samples. In a scientific survey, researchers choose samples through some random process that is usually mentioned in the survey background materials.
The term "random" has a different meaning in statistics than in ordinary language. In everyday terms, a random event is one that is unpredictable, lacks purpose and/or has no discernable pattern. In statistical terms, a random event is one that occurs with a certain, measurable chance or probability of happening. For example, under the simplest circumstances, where each member of a population has one chance of being sampled, the probability of getting selected for a survey can be calculated just by knowing a population size and desired sample size. One would have a 10 percent chance of being selected for a 100-person sample out of a total population of 1000. But, researchers use several methods for randomly selecting samples. These include stratified, cluster and systematic sampling. Stratified and cluster sampling require prior knowledge about the survey population but can produce more representative samples than simpler “blind” sampling methods. Researchers often use stratified sampling to capture the diversity of large populations with distinctive, homogeneous subgroups—such as the U.S. population. In all cases, feasibility and cost influence the sampling technique chosen by researchers.
Nonrandom samples include those that select members of the population based on their proximity, availability or through referrals by friends. There are sometimes scientific reasons to conduct non-random surveys, but they are often unscientific and should not be used to generalize statistically to larger populations.
The phrasing of questions in surveys is also important. One can be reasonably sure a survey is not scientific if questions are biased—that is, designed to elicit the answers that the survey’s sponsors want to get. “Is the search for more oil worth the environmental damage?” is a question that may elicit a negative response. Biased questions may also be phrased more subtly than that, with the understanding that people tend to give positive rather than negative responses. (One way to counteract this tendency is to ask a second question, reversing the possible responses.) Respondents also tend to choose the first option from a list and to answer as they think they “should” rather than telling the unvarnished truth. Scientific surveys are designed to minimize and account for these known tendencies.
Randomly selected samples and objective questions are two principal elements in a scientific survey. By paying attention to these factors, survey takers and consumers can learn to recognize a survey as scientific—or not.