Polls

Pollsters: Don't trust us to winnow GOP field

Networks' use of polls for debate entry comes at a time of increasing doubts about their accuracy.

By Steven Shepard

Pollsters surveyed by POLITICO have a unanimous warning for the Republican National Committee and the TV networks who are using public-opinion surveys to exclude presidential candidates from debates: Don’t trust polls to detect often-tiny grades of opinion in a giant field.

Indeed, the unprecedented reliance on polls to winnow the Republican field is coming at a time when many pollsters feel they’re blinder than ever to trends in public thinking — and that using polls to keep out candidates who are otherwise well qualified could seriously alter the race.

“Polls are being used to do a job that they’re really not intended for — and they’re not as qualified for as they used to be,” said Cliff Zukin, a professor at Rutgers University and past president of the American Association for Public Opinion Research.

Zukin warned that polling has become increasingly unreliable because Americans are harder to reach — nearly half of adults are unreachable on a landline phone — and roughly nine in 10 are unwilling to participate in surveys even when pollsters manage to contact them. The declining response rates, Zukin says, create a situation in which true public feelings are more difficult than ever to discern.

But when CNBC announced on Wednesday its criteria for the Oct. 28 GOP debate, it marked the third time out of three debates that the sponsoring news organization is using the candidates’ standing in public polling as the main determinant for eligibility.

That's fraught with risk, pollsters say. Jeb Bush, for example, is at 7 percent in this week’s NBC News/Wall Street Journal poll, but the statistical margin of error means he’s really between 1 and 14 percent — a huge range that underscores why some pollsters continue to be uneasy about the prospect of averaging together surveys and then only admitting candidates above a certain threshold.

"These numbers all have a margin of error around them," said Jocelyn Kiley, an associate director at the Pew Research Center. "We try very hard, as do most of our colleagues in the field, to make clear when there are significant differences and when there aren’t."

So when pollsters themselves announce a margin of error, it should be taken seriously, she said.

The types of national surveys being considered by TV networks typically have as few as 200 respondents for the Republican primary — there were 230 in the NBC News/Wall Street Journal poll that pegged Bush at 7 percent. A single respondent can make a big difference when networks are averaging as few as four polls together and a tenth of a percent could make the difference between appearing onstage with the front-runners — and standing next to George Pataki in the undercard.

“It’s like asking a scale that can only tell pounds to measure ounces,” Zukin said. “They’re just not that finely calibrated. ... I think polls can do a good job talking about tiers of candidates in name recognition. That’s all that polls can do. But they can’t tell the difference between Bobby Jindal, who’s not in the debate, and Chris Christie, who is."

CNBC’s criteria are slightly different than the calculations Fox News Channel and CNN used to divide the candidates into two fields — the prime-time main debate and an earlier, undercard event. First, CNBC will only use polls conducted for major broadcast and cable news organizations, ignoring other polls conducted using the same methodology by academic institutions that were used by Fox and CNN.

Second, rather than take the top 10 candidates for the main-stage debate, CNBC is instead setting a floor, only allowing those candidates who average at least 2.5 percent in qualifying polls released between the second debate and Oct. 21, a week before the third debate.

But with 15 candidates still competing for votes, some who have high name recognition and have commanded strong followings in the past will fall short of 2.5 percent. Kentucky Sen. Rand Paul is most in danger: He’s at 2.75 percent, just above the 2.5 percent threshold.

That may seem like a small percentage, but Carly Fiorina was at 1 percent in polls before the Fox News debate and, after earning a place on the main stage at the CNN debate, rocketed to 11.5 percent in the current average.

"Using early national polls — especially the ones with very small sample sizes and questionable methodologies — to impair or hold back candidates separated by 1 or 2 percent is just irresponsible," said Brad Todd, a senior adviser to a super PAC backing Jindal, who is at only 0.5 percent.

For now, the drama centers around Paul, for whom the difference between making the debate and falling short is only 0.25 percent — essentially, a matter of two respondents in all the polls put together.

Some pollsters said the science behind CNBC’s intention to use polling averages to create manageable debate dynamics is just as problematic as it was before the first two debates — but they are less troubled this time around because the candidates have had the opportunity to make their case to voters.

“They’re still estimates, they’re still ranges, decimal points are still decimal points,” said Lee Miringoff, director of the Marist College Institute for Public Opinion. “The same statistical problems apply, but we’re at a different point in the campaign.”

Even Miringoff, who wouldn’t allow Marist to be a part of the polling average before the Fox debate because of his objections, has little sympathy for Paul, even if he falls short by just hundredths of a percentage point.

“Let’s say he falls below” the 2.5 percent threshold, Miringoff said. “It’s not like he hasn’t been on the stage.”

But others see it as a simple matter of fairness — if networks are using flawed science to judge who is excluded or not, they should stop doing so.