As election draws nearer, unreliable election surveys are taking social media by storm. Here’s how you can identify which ones to trust. There are four things that anyone sharing a survey result should tell you, and if they don’t, think twice before you trust their results.
Number 1: Sampling Method – is the sample nationally representative? In Sri Lanka, this might be the most important thing to know about a survey, because there are a lot of shortcuts being taken in selecting the sample. The most reliable surveys are those that randomly select the respondents so that everyone in the country of interest to the survey has an equal chance of being selected. For example, this is not feasible in an online survey, because that will bias the selection to certain types of people that can be reached online.
Number 2: The probability of error and sample size. Unlike a census, a survey always has a probability of error that can be mathematically estimated. A trustworthy survey will always report two numbers on this. (i) margin of error and (ii) confidence level. A recent survey by Verité Research had a margin of error of ±-3%, at a 95% confidence level. That means, if 30% of the respondents answered ‘yes’ to a question, we can be 95% certain that between 27-33% (±-3%) of the population would answer the same way. There are elections surveys that tolerate a ±-5% margin of error. But check that it is at a confidence level that is not below 95%.
So what about the sample size? For a Sri Lankan survey to get a ±-3% error margin at a 95% confidence level on a randomly selected nationally representative sample, the minimum sample size should be a little over a 1,000 people. That’s why you should check if the sample size is reported – it helps to be sure if the margin of error and confidence level make sense. But remember, all that depends crucially on the first critical factor – was the sampling done in a way that everyone of interest to the survey had an equal chance of being selected.
Number 3: Framing of the question. Framing is about how the question and the response options are presented. The question matters because it can change how people answer. For example: the survey can ask “who will you vote for?” or ask “who can best solve the problems of the country?”. The answers given to these two questions could be very different.
How response options are presented matters because a survey can simply list the response options, or list the response options with some positive or negative description, potentially swaying the respondents. For example: [Option 1] - person X, who has experience, or [Option 2] - person Y, who has been accused of corruption.
You also need to know when the survey was implemented, as the answer can depend on the date the question is asked. For example, an answer to an election related question can differ depending on whether there was a big corruption scandal or some positive economic news reported around that time.
Number 4: Responsible organization. Some polls circulate without being attributed to any known organization. Others have dubious organization names that have no public recognition, or no known persons behind them taking responsibility for the results presented. These surveys cannot be trusted.
To check if there is a responsible organization behind the survey, don’t just check if there is a logo of a reputed organization. It’s easy to insert a logo without permission. A reputed organization will always publish the survey results on its website or social media handles. With AI technology easily able to change images to reflect any result, it is not advisable to share images of claimed survey results – only share links to survey results published on the pages of trusted sources.
Here are the four questions again.
- Is it a randomly selected, nationally representative sample, giving everyone an equal chance of being selected?
- Is the error margin less than ±5% and the confidence level more than 950%?
- How were the questions and answers framed, and when were they asked?
- Is it published in the pages of the responsible organization standing behind the results?
If any of the above questions are not addressed and verified, – then you should think twice before believing the results of a survey. Don’t forward it to others till you know it can be trusted.
