Methodology of the Bay State Poll
The Bay State Poll is quarterly survey of public opinion in the state of Massachusetts that randomly interviews approximately 800 respondents from across the state. All adult residents of Massachusetts with working telephones (approximately 98% of the state's population) are eligible to participate in the poll. The poll uses a random-digit dial probability sampling design to select households for participation in each of its quarterly surveys.
The sample of telephone numbers used in conducting the Bay State Poll is constructed so that each county in Massachusetts is represented in proportion to census estimates of the number of households in that county. This "stratified" design ensures that each of the state's 14 counties are properly represented in the final sample. For example, if the census indicates that 11 percent of the households in Massachusetts are located in Essex county, then we draw 11 percent of our sample from Essex county telephone numbers. These telephone numbers can be identified based on their area code (the first three digits) and exchange (the next three digits). The final sample of telephone numbers for a county is selected by randomly generating the last four digits of the telephone number. This random selection process results in an equal probability of selection for both listed and unlisted telephone numbers.
Respondents within the designated households are randomly sampled using the “last-birthday” method. When a household is reached interviewers ask to speak with the person over 18 years of age at home who has had the most recent birthday. This ensures a random selection of eligible respondents within the households selected.
After the sample is selected and the correct respondents interviewed, sample weights are calculated to correct for the fact that some respondents are harder to reach than others. For example, we know that in general younger respondents are harder to reach because they move more frequently than older ones. As a result a poll may include only 10% representation from 18-30 year old category when in fact we know from census figures that this age group makes up 15% of the state's population.
In this example, to make sure that an analysis of our poll results is truly representative of the state's population, each respondent in the 18-30 year age range would be counted as the equivalent of 1.5 people by the computer (10% x 1.5 = 15%). Older respondents, who by definition are over represented in this hypothetical sample are correspondingly down-weighted by the appropriate factor. Commonly, the Bay State Poll creates weights for groups based on age and educational attainment, thereby permitting findings from a sample to be generalized to the larger target population of all Massachusetts adults.
As with all scientifically conducted surveys, the Bay State Poll is subject to sampling error -- the expected probable difference between interviewing everyone in a population, and a scientifically selected sample of that population. The size of the sampling error is based on the sample size -- in general, the larger the sample, the smaller the error. By way of illustration, if the sample contains 800 respondents, the associated sampling error is plus or minus 3.5 percentage points at the 95% confidence level. This means that if 50 percent of the sample were found to give the Governor a positive job rating, one could be 95 percent confident that the true value among all Massachusetts adults in the population would be somewhere between 46.5 (50 - 3.5) and 53.5 percent (50 + 3.5), with 50 percent being the most likely figure.
Sampling error does not take into account other sources of variation inherent in public opinion studies that arise from sample coverage, differential non-response, question wording effects, or context effects. Sample weighting, as described above, helps alleviate some of the error introduced by differential non-response and/or coverage. Careful questionnaire design may also reduce differential non-response, and help alleviate question wording and context effects. The Bay State Poll makes available its questionnaires on-line for those who wish to see the questionnaire wording and sequencing. In addition response rates, weight calculations, and survey dispositions for particular polls can be found the Data and Publications section of our website www.merrimack.edu/polling.