Calgary's civic election polls were a 'comprehensive failure,' report says

A new report into confusing and conflicting polls ahead of last year's civic election in Calgary says there's plenty of blame to go around but singles out one polling firm and one news organization, in particular.

The report says polling done by Mainstreet Research for Postmedia the company that owns the Calgary Herald and Calgary Sun was especially off the mark and detrimental to public trust in the polling industry.

"Their polls were seriously, methodologically flawed,"said Paul Adams, a co-authorof the report along with Christopher Adams and David Zussman.

• Mainstreet admits 'big polling failures' after predicting Nenshi would lose Calgary election

But the failures weren't limited to just these companies, according to report, which found broader problems with other polls and other news reporting, as well.

It all added up to a "comprehensive failure from the perspective of Calgary voters," Adams said.

The municipal election campaign was initially expected to be a relatively low-key affair but was then turned on its head last fall, the report says, when the Calgary Herald and Calgary Sun first published the results of aMainstreetpoll that surprised virtually everyone who follows municipal politics in the city.

The poll suggested incumbent Mayor Naheed Nenshi was trailing upstart challenger Bill Smith, who had little public profile at the time, by nine percentage points. Two subsequent Mainstreet polls put Smith even further ahead: by 17 and 13 points, respectively.

The report's authors two of whom have a background in pollingthemselves, and the third in government andpublic policy found the polls "significantly affected the course of the campaign."

"They threw Nenshi's campaign on the defensive, gave impetus to Smith's campaign, and possibly doomed the prospects of another candidate, AndreChabot, who Mainstreet's poll suggested was not a close contender," the report reads.

Nenshiwent on to win the election by eight percentage points.

Mainstreetpresident QuitoMaggilater admittedto "big, big polling failures," despite frequently and aggressively defending his polls from a variety of criticisms during the campaign.

The report contrasted the humility expressed after the election with public comments from Mainstreetofficials during the campaign.

"In the media and on social media, they expressed a kind of bloated self-confidence that was not justified by what they knew to be problems in their methodology," the report reads.

Where the pollswent wrong

Maggi, who voluntarily participated in the external investigation,said Postmedia requested a conference call with him after the release of the first poll during the campaignto discuss thepublic criticism it was receiving.

He said he then decided to change the methodology for the second poll by increasing the sample size and ensuring there was a minimum number of people surveyed from each of the city's 14 electoral districts.

But the report found that only made things worse.

That's becauseinstead of using random-digit dialling to reach people by phone, Mainstreet used a "directory" that it had procured from a variety of sources. The firm then relied on postal code data from that directory in order to determine which electoral district a survey respondent lived in.

Any respondents who had no postal code associated with their phone numbers in the directory were excluded, and that often meant people who only had cellphoneswere cut out of the results.

"They ended up with an over-representation of landlines with this new methodology," said Christopher Adams.

"And so, with very low cellphones in that sample, they had a distorted youth/adult sample at the end."

The report also found methodological flaws with other polls released during the campaign, including asurvey commissioned by a pro-transit group called LRTOn The Green. That poll suggested Nenshiwas up by 15 points up over Smith but the report found it asked leading questions.

Another survey commissioned by a group of academics and conducted by Forum Research was the most methodologically sound, according to the report. That poll put Nenshi up by 17 points over Smith but it was not intended as a strict horse-race poll and wasconducted over a period of weeks, rather than days, making it less useful as a "snapshot in time" of voter intentions.

Overall, the report acknowledges that polling has become significantly more difficult compared to the recent past, when landlines were ubiquitous and random-digit-dialling was standard.It recommends a higher degree of transparency from pollsters when it comes to their increasingly "novel" methods of reaching respondents and calculating results.

The report also recommends news organizations develop a greater commitment to high-quality reporting on polls, while detailing how many failed to adequately inform voters during last year's election campaign.

Where the media went wrong

The city's newsrooms had shrunk significantly since the previous municipal election, the report found, leaving fewer and less experienced reporters to cover the 2017 campaign, resulting in coverage that was less sophisticated.

Some journalists tried to be balanced in their coverage but"were constrained by an uncertain grasp of the methodological issues," the report says, a problem that was further exacerbated by a lack of transparency from pollsters.

"It should be said that Postmedia was in a better position than other news organizations to evaluate Mainstreet Research's public claims of absolute confidence in its work," the report says.

"As reported to us by Maggi, Postmedia executives requested a conference call to address the criticisms of the first poll. In the course of that conversation, Mainstreet indicated it would be tweaking its methodology ...Despite Postmedia's awareness that Mainstreet Research was concerned enough to make changes to its polling, the Calgary Herald and the Calgary Sun continued to print quotations from Mainstreet executives that proclaimed a pugnacious self-confidence about their methodology."

The newspaper company did not participate in the external investigation, telling the report's authors that "legal action citing Postmedia in a separate matter will prevent us from being able to take part in this review."

LorneMotley, editor-in-chief of both the Calgary Sun and Calgary Herald, declined to comment on the release of the report, citing the same, unspecified legal matter.

Maggi said Mainstreet's contract with Postmediaexpired in February and has not been renewed.

Mainstreet welcomes release of report

For his part, Maggiwelcomed the release of the external report, saying it came to many of the same conclusions as Mainstreet, itself.

"We really wanted this report to published," he said. "The report is extremely detailed and looks at the errors in all the polls."

In line with the report's recommendations, he saidMainstreethas already made changes to the way it reports its polls, in order to bemore transparent about its methods and results.

The company had previously concluded a pair of its own investigations, one looking into itspolling methodologyand the other into the wayit communicatedwith the public during the election campaign.

Those internal investigations were separate from the results of the external investigation that were released today, which wasinitially commissioned by theMarketing Research and Intelligence Association (MRIA).

The MRIA is a voluntary, self-regulatory body that represents the market and survey research industry in Canada, but it has run into financial difficulties recently. Last week, it announced plans to disband.

• Canada's polling industry association is shutting down

The MRIA had previously received the report but had yet to release it. As a result of the news of the MRIA's demise, the authors opted to release the report, themselves.

Maggialso said he was pleased the report found no evidence Mainstreet was "in cahoots" with any politician or news organization to deliberately skew poll results, as had been suggested by some members of Nenshi'scampaign team.

"You couldn't pay me to tank a poll," he said. "There's not enough money in the world."

You can read the full report below.

• MORE CALGARY NEWS|Swerve magazine shuts its doors after 14 years of featuring ordinary Calgarians and their extraordinary lives
• MORE CALGARY NEWS|Students allege inhumane treatment of lab animals at U of C's psychology department

Read more articles byCBC Calgary, like us onFacebookfor updates and subscribe to ourCBC Calgary newsletterfor the day's news at a glance