It’s extremely easy to write a polling question that gives misleading results. Half the time you may not even realize you’re doing it. One of the biggest challenges of doing survey research is that respondents are not perfect little bags of information you can tap into at will, they’re people who inherently respond to cues given in the survey itself. And those cues can extremely mess up the results you get! This makes polling hard, and measurement harder.
Here’s an example, from the Institute for Middle East Understanding Policy Project. [And note, this is NOT the only poll to ever have this issue, it’s just the most recent that I saw, so I’m using it as an example] This guy went around social media a lot! It’s a surprising topline, and was broadly framed as “Gaza is the top issue for Biden-not-Harris voters”.
When Biden 2020 voters cast a ballot for someone besides Harris in 2024 were asked “Which one of the following issues was MOST important in deciding your vote?” they selected:
29% - Ending Israel’s violence in Gaza
24% - The economy
12% - Medicare and Social Security
11% - Immigration and border security
10% - Healthcare
9% - Abortion policy
5% - Don’t know
My guess at what happened with this survey question is that they took an existing question- something like this guy from YouGov- that asked voters what the most important issue to their vote was, trimmed it down a bit to only the categories the client cared about, added the Israel plank, and went ahead with it. This is a pretty ordinary way to construct a question (grabbing language from a question you know to be reliable and altering it is a great way to start), but it can have some odd effects if you aren’t careful about editing.
Here’s the YouGov question. Note that this was asked of all adults, not just Biden voters who didn’t vote for Harris, so the proportions are going to be different.
Which of the following issues has been the most important for you in deciding how to vote in this year's presidential election?
Abortion 12%
Crime 4%
Education 2%
Environment and energy 4%
Foreign policy 3%
Health care 7%
Immigration 13%
Inflation 13%
Social issues 6%
The economy 21%
None of these are important to my presidential vote 2%
I am not voting in this presidential election 5%
Not sure 6 %
Notably, in the original YouGov question, all the planks are roughly similar in length and language intensity. If you’re reading through and selecting between “immigration”, “inflation”, etc, you don’t have any good cues as to what the people running the poll care about. My one note is that “environment and energy” is framed to be softer than either a “climate” or “energy” plank individually, which is a little odd. But I get needing to compress categories together for length.
However, in the IMEU poll, “Ending Israel’s violence in Gaza” is much more loaded, intense language than the rest of the planks. Imagine you’re taking this poll with no idea what it’s about- you’re essentially being asked to choose between pretty neutral statements like “healthcare”, etc, and “ending violence”. That’s a strong cue as to which is the “correct” answer! As much as we can try to argue that polls aren’t pop-quizzes, there are no correct answers, etc, the experience of taking a poll for a politically unaware respondent is extremely reminiscent of a pop quiz. The industry has a variety of terms for types of bias introduced by respondents who would love nothing more than to get through this poll and get back to their day, including things like selecting “yes” the whole way through, answering randomly, etc. The broad category of bias induced by doing just enough to get through the poll is called “satisficing”, but this specific phenomenon might be better called “demand effects”.
The incentives of poll respondents are complicated and vary based on the type and method of poll. If you’re answering a phone poll, you might want the interviewer to think you’re smart and informed, you might want them to think you’re a good person, or you might want to get through the poll as fast as possible. If you’re taking an online poll for a monetary reward (or points towards a future reward, which is more common), you want to be allowed to finish the poll and not get screened out, which means you want to present as whatever group they’re looking to poll, and you also probably want to seem smart and informed and so on. Incentives from both types of respondents could easily lead to selecting what you believe to be the “right” answer in the eyes of the pollster, whether or not you agree with it.
For the IMEU poll, the language difference on the “Israel” plank probably cued respondents that was the answer they wanted, and increased that share of responses. This is a really common problem with polls run for and by groups working in particular issue areas. If you spend all your time working on an issue, you probably have specific language you use to discuss it. If you include this language unaltered in a poll, it’s going to stick out compared to the neutral language used for planks you don’t care about. I’ve had this conversation with clients before, and it’s rough to ask them to water down their habitually used language or let you reframe a question. That language was probably developed for a reason, it’s how they talk about things, and a “neutral” frame can feel hostile or confusing. This is one of the jobs of a pollster, to intervene and talk clients out of mistakes that can render their results misleading. It’s also a hard conversation, and relies on the pollster having sufficient leverage to say no, which you don’t always have.
There’s a couple lessons to be learned here. One is, if you’re a consumer of polling, you really really need to take a look at the original question wording and context in which it was placed. I like to ask myself “if I was a respondent, could I guess the views of the person who wrote this question?”. If the answer is yes, it’s very likely respondents will pick up on that, and demand effects will skew towards that answer. I’m assuming here that if you’re reading this post you’re probably already up on things like sample size (and checking if the pollster is real), but that matters too.
Second, if you’re an advocacy organization trying to do polling, PLEASE let your pollster write the questions how they want. Don’t copypaste your language into the poll. You’re paying them for expertise, and part of that expertise is knowing how to make questions read neutrally to respondents. If you really really want to poll the language you habitually use, you can do that, but treat it as a high water mark or a response to your messaging, not as a straight up support measurement.
A final lesson is that polling is difficult and complicated, and a single question is never going to be the end of the story. Pollsters can go back and forth debating the merits of a question framing, and different frames may give you different results. Sometimes it’s not clear which of those results is most “correct”! This is not even touching the difficulties of weighting, sample selection, etc. One number isn’t a complete answer. Sometimes folks will point to odd polling results coming from questions with really loaded language as “bad” or “wrong”, and this isn’t strictly true. The poll is giving you results to the question you asked. The problem is when the questions don’t align with the information you’re seeking, or don’t align with how you’re framing that result.
Addendum
The question of what counts as “neutral” language is a really thorny one, that I’ve mostly dodged here. What matters for the purpose of a polling question is not so much if the language is “neutral” in an abstract sense, but if it indicates a “correct” option to the respondent. In theory, you could improve a question that was like:
What is the best fruit?
Pears
Apples
The magnificent grape
Bananas
Other
By converting it to
What is the best fruit?
Delicious pears
Amazing apples
Magnificent grapes
Wonderful bananas
Other
Increasing the intensity of all the planks so that they match erases the bias towards grapes. Probably. In practice, doing this with real world political issues would be *really hard*. Intense language around issues is highly partisan and variable, and I don’t think you’d be able to make it so respondents couldn’t easily guess the bias (and partisanship) of the pollster. Plus I suspect respondents would bounce right out of a survey where every question involved all high-stakes language. So, for the purposes of a polling question, “neutral” language means language that gives respondents the least indicators about the person conducting the survey, and language that matches with the way other issues are presented in the poll. If this sounds vague and tricky to implement, it is! Isn’t polling great?
I keep thinking, "Beware the single-cause explanation." The poll reminds me of those strange lists of symptoms and numbers one gets in the doctor's office--a kind of a quiz thing that gets you into the ballpark but onto no clear base. There is an ongoing wishful-thinking narrative among leftish activists who want everyone to believe that Harris loss because she did not break with U.S. policy on Gaza. Surely she lost votes because of that; we saw it in Michigan, though not all Arab-Muslim abstentions or votes for Trump were purely because of Gaza (in the Muslim majority enclave-town in Detroit of Hamtramck, other social issues like anti-LGBTQ attitudes played a role). But this one IMEU poll keeps getting trotted out as conclusive and definitive. Am wondering if any more analysis has been done on this poll and this claim that you can recommend. Thanks for your work.