As opinion research professionals, we’re very aware that the practice of polling is not always flawless; even a perfectly designed poll will be wrong from time to time. As with any information source, it’s always best to be an informed reader of the polling process; blind faith in polling and outright rejection of its results will lead you to inaccurate conclusions.
The conventional wisdom says that the polls in 2016 were wildly erroneous. Just as many pollsters were moving from behind the curtains to the center stage of political punditry, Donald Trump’s victory led many to proclaim that polls were wrong, biased, just guesswork, or otherwise worthy of suspicion.
The problem with this narrative is that it’s just not true; while it surprised most of the conventional wisdom, President Trump’s victory was not as shocking as the polling indicated. While many treated Secretary Clinton’s victory as a foregone conclusion, it was not supported by the data.
In 2016 the polls performed as well as they usually have by historical standards. The same can be said about their performance in the 2018 midterm elections, a time when it can be especially difficult to measure turnout and voter enthusiasm.
Here are a few keys to help you navigate the polling deluge to come in 2020 and how it can apply to your business, whether you’re conducting research or building a business plan:
Basics of Survey Research
The basic purpose of a political poll (or any survey research) is to gauge the opinion of the overall audience by questioning a random, representative sample. Think of a survey like a pot of soup. You do not need to eat the entire pot of soup to understand its flavor. A spoonful will give you some idea, but a cup—or better yet a bowl—can more accurately “represent” its flavor.
There are four pieces to a successful poll, whether it’s for an election or meeting your business needs:
1. Obtain an Accurate Sample
To field an accurate survey, a pollster must first obtain a representative sample of their audience. The survey must ensure that respondents have an equal (random) chance of participating in the survey. If the survey method excludes certain audiences, it impacts the poll’s accuracy.
For election polling, the preferred instrument is a “likely voter file.” While voters have a secret ballot, whether they voted is public knowledge. In elections the best prediction of future performance is past behavior, so those who vote regularly are the most likely to vote in the future.
Private companies maintain likely voter file lists, which are purchased by campaigns and party committees for purposes such as their own polling, targeted advertising, and Get Out the Vote (GOTV) operations. Some public pollsters will use a file of registered voters as a cost saving measure.
What you should know: When conducting research, make sure you’re talking to the right people. Identify your target audience and verify that they’re who you should be talking to.
2. Field the Survey
For generations, the industry standard in political polling has been landline telephone surveys. Landlines offered the benefit of being inexpensive and a common mode of communication for voters regardless of their age, region, race, or gender.
However, the number of voters who are “cell phone only” has skyrocketed. Voters without a landline are younger, less likely to own a home, and more likely to have lower-incomes. To obtain a representative sample, calling cell phones is a must—however, this method is costly. Due to FCC regulations, call centers are not allowed to call cell phones using automated calling and instead must manually dial each number.
An additional obstacle is declining response rates. An entire generation has grown up on caller ID and call screening, and many will not answer an unknown number. While this poses challenges for obtaining an accurate sample, the biggest impact is the added cost of dialing additional numbers in order to field a survey.
As a result, more surveys are conducted online. This method is less expensive, but more difficult to obtain an accurate sample. The main obstacle is that some Americans lack Internet access, and they are demographically different than other Americans: they are older, less affluent, more likely to live in rural areas, and more likely to be African-American or Hispanic.
What you should know: Choose a research method that is efficient and accessible for your audience. This can be over the phone, online, in-person, or through the mail. Be flexible and cognizant of your audience and how they communicate.
3. Achieve Correct Sample Size
Assuming the pollster has obtained a representative sample of voters, the survey will still only be as accurate as the sample size. There are diminishing returns for surveying larger audiences. However, larger survey samples allow for more precise targeting among subgroups.
What you should know: Make sure you’re talking to enough people to represent all points of view. Don’t put blinders on your findings by neglecting important voices.
4. Weight Final Data
Even after following best practices and obtaining an accurate survey sample, the data may not match the demographics of the survey audience. Therefore, the data must be weighted.
As an example, the typical composition of likely voters in a general election is 52% women, 48% men. If the survey results come back with a sample that is split 50/50, the following weights would be applied to a sample of 400 respondents; women will have their responses “count” 1.04 times, while men will have theirs count 0.96 times.
|Group||Actual Sample||Expected Sample||Weight Applied|
Accurate, unbiased weighting is one of the most critical parts of survey research. There’s evidence that, in 2016, pollsters who did not weight on college education underestimated Donald Trump’s performance. So, much like you would do with any article, blog post, or periodical, be sure to fact check survey results—as there may be more at play, hidden behind the scenes.
What you should know: Good research is about staying methodical and making sure you haven’t left any stone unturned. Weighting a survey is just like putting the finishing touches on a business plan; you must take a comprehensive view and make sure you haven’t neglected any potential customers.