Polling is really hard, NYC Mayoral Primary Edition

with these sample sizes they should call it the medium apple

Generally my thoughts on New York are that it’s not my problem. This week, it is a little my problem, in that folks are having a round of Oh No Our Primary Polling Was Bad and that’s like catnip to me personally.

It’s well documented in a Politico piece here and a NYT collection of polls here. In case you (somehow?) weren’t seeing it, polls mostly said that Andrew Cuomo was going to win the first round by a solid margin (something in the mid-teens, generally) and then win the final round by some similar margin. This is, well, not what happened! Whoops. Why were the polls so off? I have some theories.

[Edit: Ruth Igielnik at NYT wrote basically this post already, but I am much more wordy and inside-baseball, so I think this is a useful edition nonetheless]

Polling Primaries Is Hard And Sucks

Election polling is, generally, really hard. You’re guessing at the composition of an electorate that doesn’t exist yet, which makes even designing your sample quite difficult, and then you also have to hope that nothing dramatic happens between now and election day to make your poll irrelevant. Polls are snapshots, not predictions, but people are still going to judge you on getting the “right” answer about an event that could be months or years in the future.

One small saving grace is that two party general elections tend to turn out very similarly to whatever the previous election was. It’s kind of odd- even with largish churn in the electorate, election results tend to be pretty stable. “Pretty stable” here means moving from 54% to 48% and not from 54% to like, 80% or 32%. Results are highly correlated with previous results. This is great, if you’re trying to do polling, because you have a vague sense of the range of “normal” results, and you can look back at previous elections to figure out what the electorate will be like.

Primary elections are a nightmare. The eligibility requirements are more complicated (only registered voters of one party, independents, etc), and the makeup of the electorate is a mystery. In general elections, you can at least figure that people will care more about a presidential than a midterm, and will care more about something likely to be close than a total blowout. With primaries, “is this going to be a big deal to normal people” is....vibes. Even if you get high vs low turnout correct, figuring out what types of voters will show up is similarly tough. Normally, primaries are for extremely committed and high-information partisans. But that’s not always true, and it’s hard to predict when it’ll stop being true.

In generals, “how many Democrats and Republicans are there in this geography and in this voting population” is a knowable-ish question. (I am eliding a WHOLE WORLD of tricky analysis here but that’s fine). For a primary, this data doesn’t exist. You don’t have party registration or ideology models to help you. You’re stuck trying to sample a reasonable electorate with minimal information and no obvious tells about how an individual is likely to vote. It’s rough.

Polling RCV Is Hard [if the RCV bit matters, which it maybe didn’t]

Look. It is hard enough to be asking people which candidate they’re going to vote for months in advance, when they don’t have partisanship to provide them with a fast heuristic. Even some voters in generals (our beloved confusing swing voter) don’t make up their minds until the last minute. In primaries, they have to choose between candidates who are all far more likely to be close to them on issues, and you should assume they’re going to be much more wobbly about that choice.

Suddenly with RCV you are asking people to *order multiple candidates*? In advance? And then you have to use that ordering to produce your final result? And it matters if they ranked Mark Miscellaneous 2nd or 3rd? This is more accuracy than can be reasonably expected of a survey respondent. Note that I am not saying voters can’t *produce* that ranking- we know they can, since the research on RCV shows that most people do rank multiple candidates. It’s the stability of that ranking, especially the middle to lower end, that I worry about. If you ask me right now what my fourth-favorite ice cream flavor is, that answer will 1000% change if you ask again in a month.

Now, in NYC specifically, it doesn’t seem like the RCV of it all actually mattered. Winning on a first ballot is the most clear cut option, and this is probably not a big reason that the polls were wrong. It’s just a complicating factor for this type of race in particular.

What Went Right?

There just weren’t all that many polls of the race, in the scheme of things. Of the ones archived on NYT, exactly one had Mamdani up in the first round, this PPP poll (toplines here).

The toplines posted are fairly short. It looks to be an SMS poll, and there’s no questions listed about past vote history. Since they mention young people who didn’t vote in 2021 as a key group in their post, either they did targeting off the voter file or asked questions not included in the published toplines. I’m guessing the former, since one of the advantages of an SMS poll is you can directly assemble your list of who is probably in the electorate, take a representative slice, and then contact those people (or well, try to, response rates are still lousy).

They mention specifically that

Usually when polling a primary election pollsters start out with a list of voters who have participated in similar elections in the past.

It was clear in this election though that Mamdani was building a movement that was going to bring a lot of people into the process that had never voted in a city election before. So we made a conscious decision not to require people we polled to have voted in 2021. If they said they were going to vote on our screening question that was good enough.

This is interesting! Normally, you would build out a likely voter audience before running your poll. This is not quite the same thing as “starting with a list of voters who have participated in similar elections in the past”, but that is an element. Defining a likely voter population usually involves pulling people who voted in the past (usually in one of several previous elections) or recently registered to vote. You could also do a likely voter *screen*, in which you ask “are you going to vote in this election?” and drop people who say no. That second thing, the screen, seems to be what PPP didn’t do. The Politico write up and NYT tracker calls it a “likely voter” poll, but there’s a ton of wiggle room in defining that audience, and they may have made the decision to expand it as much as possible. Notably, likely voter audiences *should* include young people who recently registered (potentially because they just became eligible) all the time. The choice would be more to include less-young people (like, 30, rather than 18) who registered a few years ago but hadn’t previously voted despite being eligible.

Non-Transferable Lessons

Taking them at their word, the thing PPP did right was estimating that the electorate would have a higher proportion of young people. What does this tell you if you want to do good primary polling in the New York of 2029? Stone cold nothing. Being able to translate vibes, media coverage, and political instincts into an accurate estimate of the electorate is a valuable skill, don’t get me wrong. What it doesn’t do is produce strong specific guidelines about How To Do Polling that can be generally applied.

I am almost certainly generalizing too hard from a single poll in a single race, but this is a fascinating example of folks being honest about what it’s like running election polling in a modern environment. “We made a guess and it was right” is what most pollsters who are lauded for being the only one to get it right should be saying. The Politico article about the poll’s release makes no mention of the choice to poll a younger electorate (although I haven’t been able to find any formal press release, perhaps that was elsewhere), which is a bit tragic, because that would have been a hell of a called shot.

Unfortunately for us all, the lessons from this race seem to be less about how to be a good pollster and more about how to be a good forecaster. I’ll look forward to more retrospectives and analysis from other pollsters, which may yet include bigger conclusions about structural fixes. I’m not super optimistic- there’s only so much you can do in a >1k person poll in a primary- but I hold out hope that we’ll stumble into a way to fix our current polling morass that *isn’t* simply guessing right.