However, their accuracy has been open to question
Nicole Satherley, Andrew Sporle and Lara Greaves
Auckland and Wellington, August 26, 2023
The closer we come to election day on October 14, the more media focus we will see on Political Polls. Poll results are often used to project the makeup of Parliament, despite them being snapshots of the Polling period rather than reliable predictions.
Indeed, the accuracy of past Political Polls in New Zealand has been open to question. And the way results are framed can sometimes cause more confusion than provide useful context.
No Poll is perfect. But understanding the quality of a Poll and the results it produces requires knowing something about how the Poll was designed and carried out.
A Guide to Opinion Polls
We recently completed a guide to understanding public opinion Polling in New Zealand that describes the important features of Polls to look out for. These factors determine their quality and should be considered when making conclusions about the results.
Technical details (sample size, margin of error and so on) about a given Poll are usually available. So what information is important to consider?
Here are ten things to think about when evaluating a Political Poll.
Sample Size: Contrary to common misconceptions, good quality results about the New Zealand population can be obtained from Polls with as few as 500 to 1000 participants if the Poll is designed and conducted well. Bigger samples lead to less random variation in the results – that is, the differences between the results from a sample of the population compared with the whole population. But bigger samples are more expensive to collect, and they don’t make up for poor sampling design or Polling process.
Target Population: It should be clear which group of people the results are about. Results about sub-groups (for example, women in a certain age group) should be treated more cautiously, as these are associated with smaller samples and therefore greater error.
Sampling Method: Sampling design is crucial. It determines how well the Poll sample matches the target population (such as people intending to vote). Polls should be conducted with an element of choosing people at random (random sampling), as this achieves the best representation of the population. Polls that allow for self-selection and that do not control who can participate, such as straw Polls on media and social media sites, will end up over-representing some groups and under-representing others. This leads to biased, inaccurate results.
Sample Weighting: When characteristics about the population and the sample are known, such as the percentage of women, age or region, “weighting” increases the contribution of responses from groups under-represented in the sample to better match the population of interest. This is achieved by making responses from under-represented respondents count more towards the results of the total Poll.
Weighting cannot be used, however, to correct unknown differences between the Poll sample and the total population. The distribution of population characteristics, like gender and age, is known through the census and can be adjusted in the sample with weighting.
But we don’t have known population characteristics for other things that may affect the results (such as level of interest in politics). Good sampling design, including elements of random sampling, is the best way to ensure these important but unknown characteristics in the Poll sample are similar to the whole population.
Poll Commissioner and Agency:
Knowing who paid for the Poll is useful, as there may be vested interests at play. Results could be released selectively (for example, just those favourable to the commissioning organisation). Or there may be a hidden agenda, such as timing a Poll around particular events. Equally, we can be more confident in Poll results when the Polling agency has a strong track record of good practice, particularly if they follow national and international codes of best practice.
Poll Timing: Knowing when the Poll was conducted, and what was happening at the time, is important. Poll results describe public opinion at the time the Poll was conducted. They aren’t a prediction of the election outcome.
Margin of Error: Margins of error are a natural consequence of taking a sample. The margin depends on both the size of the Poll sample or sub-sample and the proportion of the sample selecting a given option. The margin of error is largest for a proportion of 50% and smaller at more extreme values, such as 5% and 95%.
This makes knowing the margin of error for smaller results very important. Minor parties, for example, may be close to the 5% threshold for entering parliament. Knowing the margin of error therefore provides a more accurate picture of where they stand relative to this important threshold.
Considering the margin of error is also vital for assessing changes in Poll results over time, and differences within Polls.
But the margin of error does not account for other sources of error in Poll results, including those due to poor sampling methods, poorly worded questions or poor survey process.
The total error in a Political Poll consists of these other sources of error as well as the sampling error measured through the margin of error. Unless a Poll is perfectly conducted (which is highly unlikely), the total survey error will always be larger than the margin of error alone would suggest.
Precise Question Wording: Responses to a Poll question can vary markedly depending on how it is asked. So, pay attention to what specifically was asked in the Poll, and whether question phrasing could influence the results.
Percentage of ‘Don’t Knows’: Large percentages of “don’t know” responses can indicate questions on topics that Poll respondents aren’t well informed on, or that are difficult to understand. For example, the percentage of “don’t know” responses to preferred prime minister questions can be as high as 33%.
The Electoral Context: The composition of Parliament is determined by both general and Māori electorate results. Pay attention to the Māori electorates, where Polls are often harder to conduct. Māori electorate results are important, as candidates can win the seat and bring other MPs (proportionate to their overall party vote) in on their “coattails”.
Finally, watch the trends: Making sense of Polls can be challenging. Readers are best placed to interpret the results alongside other Polls, past and present. Keeping margins of error in mind, this helps determine the overall trend of public opinion.
Nicole Satherley is an Honorary Academic in Psychology and Andrew Sporle is an Honorary Associate Professor at the University of Auckland; Lara Greaves is an Associate Professor at the Victoria University of Wellington. The above article has been published under the Creative Commons Licence.
