SOS polls duel on sampling, turnout model

You couldn’t have gotten two more different results from polls a week apart in Louisiana’s fall secretary of state race. Why these differ and what this means fascinates.

As noted last week, a poll for The Hayride website by Remington Research produced the following results:

Kyle Ardoin: 13%
Renee Fontenot Free: 10%
Heather Cloud: 8%
Julie Stokes: 8%
A.G. Crowe: 7%
Gwen Collins-Greenup: 6%
Rick Edmonds: 3%
undecided: 45%

And this week, a poll by JMC Analytics for the state Rep. Rick Edmonds campaign gave these results:

Fontenot Free: 22%
Stokes: 11%
Edmonds: 11%
Collins-Greenup: 4%
Ardoin: 3%
Cloud: 2%
Crowe: 1%
Other candidate: 1%
undecided: 46%

About the only things they even come close to agreeing upon is the undecided proportion. Looking at it another way, excluding that and the JMC “other,” the average absolute error (absolute average differences divided by number of observations) between the two is over 5 percent, which is stunning for a contest with this many people in the single digits.

That they collected data a week apart can’t explain this – there’s was no major news between Sep. 12-20 that could cause such dramatic reshufflings in this low-profile contest. Nor does an explanation by the creator of the temporally later of the two, JMC’s principal John Couvillion, wash.

Couvillion tried to explain why the incumbent Ardoin polled so low by arguing revelation of alleged bid-rigging by his office. But that was old news prior to the Remington poll, having emerged a month earlier. And the few campaign communications that have gone out since then haven’t had any Ardoin opponent extensively hammer home the issue.

Observed differences, then, came from one of two or both sources: either the manner of polling differed, or one or both were victimized by “unhappy randomization,” or hitting on the one in twenty chance (according to the polls’ metrics) that one or both drew an unrepresentative sample of the public. Reviewing the details of each can determine which seems to present the most representative snapshot.

One major difference is sample size. The Remington one had over twice as many respondents, 1,615, than did the JMC one. All other things equal, larger samples more likely present more representative results.

Remington’s also speaks to contacting likely voters, while JMC’s says it contacted likely voting households through an automated system, which Remington also used. Remington asked more than 20 questions and JMC only 3. This is a crucial point. Automated collection runs the risk of picking up nonvoters and even unregistered individuals. Further, it could skew the demographics to make the sample less representative.

Pollsters will try to compensate for this by asking demographic questions and adjusting the call balance throughout. That is, if an inordinate amount of completed calls come from, say, black households, the dialing protocol can adjust to oversample on white households until the desired balance occurs. So, the more demographic questions asked, the more one can mitigate the chances of an unrepresentative sample.

As it turns out, JMC asked a single demographic question – sex, the least robust control of all. Remington asked at least one additional – ideology identification, another control not all that robust – but perhaps several others as part of the survey protocol. If so, that gives it a distinct advantage in securing maximal representativeness.

(NOTE: after this original publication, Remington confirmed it asked questions about sex, race, and party identification, while geographic information was inferred from the respondent information used.)

Regardless of confirmation strategies, the demographics didn’t end up too far apart. JMC overweighed a bit on black households and Lafayette residents while Remington went higher on Baton Rouge and New Orleans residents and Democrats. Much of this likely has to do with differing models of turnout, although some could be an artifact of sampling which, because of its smaller sample, would have had a greater likelihood with JMC.

Some of these differences might explain the differing distributions; for example, Ardoin’s better showing on Remington’s and better performances by the Democrats with JMC’s. Then again, four percent more black households seems a stretch to explain why JMC had Democrats picking up over a quarter of the vote while Remington had them at less than a sixth, and that the only black Democrat in the race, Collins-Greenup, got six percent in Remington’s but only four percent in JMC’s.

One other issue might have inflated Ardoin’s totals with Remington: that survey apparently didn’t rotate names in the question, instead asking all Republicans first in alphabetical order, then the two Democrats in the same fashion. Some respondents not knowing anything about a race but wanting to give an answer for whatever reason will attach to the first name they hear, in Remington’s case apparently always Ardoin.

The short JMC poll did ask another question that overlap to a degree with Remington’s: whether to reelect Democrat Gov. John Bel Edwards. Both delivered the same proportion to reelect, 43 percent, but Remington had 46 percent for “someone new” while JMC had 37 percent to say he didn’t deserve “to be reelected.”

Undoubtedly the question wordings – “If the 2019 Election for Governor of Louisiana were held today, would you definitely vote to reelect John Bel Edwards or would you give someone new a chance?” and “Do you believe that Governor John Bel Edwards deserves to be re-elected?” – had something to do with 9 percent more undecided than negative. It’s also possible that the slightly different sampling frame, with more blacks in JMC’s, might also have reduced the negative response.

So, should Edmonds be reassured, Ardoin worried, etc.? Given its much larger sample and apparent control questions, Remington’s results have the edge. Discounting the week’s difference as immaterial and weighing JMC’s at a third of both, you get something like this (rounded):

Fontenot Free: 14%
Ardoin: 10%
Stokes: 9%
Cloud: 6%
Edmonds: 6%
Collins-Greenup: 5%
Crowe: 5%
undecided: 45%

Which comes out with a more closely-packed field that seems more realistic in a low-information, low-interest environment.